Network model compression method, device, storage medium and computer equipment

Through the combination of neural architecture search and L1 regularized loss function, the redundancy problem of deep neural network model structure is solved, efficient compression of the model is achieved, and suitable for resource-constrained computing platforms.

CN110490323BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910770193.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-08-20
Publication Date
2024-11-26
Estimated Expiration
2039-10-07

AI Technical Summary

Technical Problem

The huge amount of parameters of the deep neural network model leads to redundancy in the model structure, which in turn increases internal parameter consumption, limiting its application on resource-constrained computing platforms.

Method used

The neural architecture search method is used to search for the target network structure from the search space, and the replacement network model is evaluated through the L1 regularization loss function until the preset requirements are met and the network model is compressed.

Benefits of technology

It effectively reduces the number of parameters and complexity of the network model, ensures the accuracy of the model, improves structural redundancy problems, and reduces the calculation cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110490323B_ABST
    Figure CN110490323B_ABST
Patent Text Reader

Abstract

The present application relates to a network model compression method, device, storage medium and computer equipment, which obtain the network model to be compressed; determine the current structure to be processed in the network model to be compressed; search for the target network structure from the search space, and use the target network structure to replace the current structure to be processed; use the L1 regularization loss function to evaluate the model effect of the replaced network model; when the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all the processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model. Compressing the network model through neural architecture search can ensure the accuracy of the network model. At the same time, L1 regularization processing can effectively reduce the number of parameters of the network model and reduce the complexity of the network model, thereby effectively improving the problem of structural redundancy of the network model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a network model compression method, device, storage medium and computer equipment. Background Art

[0002] With the advancement of science and technology, deep learning technology based on neural network models has developed rapidly. This technology has achieved breakthrough results in many application fields including image recognition, target detection, semantic segmentation, speech recognition and natural language processing.

[0003] However, the number of parameters in deep neural network models is huge, which makes the model structure redundant and leads to large consumption of internal parameters. Summary of the invention

[0004] Based on this, it is necessary to provide a network model compression method, device, storage medium and computer equipment that can effectively improve the network model structure redundancy problem in order to address the technical problems existing in the prior art.

[0005] A network model compression method, comprising:

[0006] Obtain the network model to be compressed;

[0007] Determining a current structure to be processed in the network model to be compressed;

[0008] Searching for a target network structure from the search space, and using the target network structure to replace the current structure to be processed, to obtain a replaced network model;

[0009] Using an L1 regularization loss function to evaluate the model effect of the replaced network model, obtaining an evaluation result;

[0010] When the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all the processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0011] A network model compression device, comprising:

[0012] A model acquisition module, used to acquire the network model to be compressed;

[0013] A structure determination module, used to determine the current structure to be processed in the network model to be compressed;

[0014] A structure replacement module, used to search for a target network structure from the search space, and use the target network structure to replace the current structure to be processed, so as to obtain a replaced network model;

[0015] The effect evaluation module is used to use the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result; when the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0016] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Obtain the network model to be compressed;

[0018] Determining a current structure to be processed in the network model to be compressed;

[0019] Searching for a target network structure from the search space, and using the target network structure to replace the current structure to be processed, to obtain a replaced network model;

[0020] Using an L1 regularization loss function to evaluate the model effect of the replaced network model, obtaining an evaluation result;

[0021] When the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all the processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0023] Obtain the network model to be compressed;

[0024] Determining a current structure to be processed in the network model to be compressed;

[0025] Searching for a target network structure from the search space, and using the target network structure to replace the current structure to be processed, to obtain a replaced network model;

[0026] Using an L1 regularization loss function to evaluate the model effect of the replaced network model, obtaining an evaluation result;

[0027] When the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all the processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0028] The above-mentioned network model compression method, device, storage medium and computer equipment obtain the network model to be compressed; determine the current structure to be processed in the network model to be compressed; search for the target network structure from the search space, and use the target network structure to replace the current structure to be processed to obtain the replaced network model; use the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result; when the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model. Compressing the network model through neural architecture search can ensure the accuracy of the network model. At the same time, L1 regularization processing can effectively reduce the number of parameters of the network model and reduce the complexity of the network model, thereby effectively improving the problem of structural redundancy of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A diagram showing an application environment of a network model compression method in an embodiment;

[0030] Figure 2 A schematic diagram of a flow chart of a network model compression method in one embodiment;

[0031] Figure 3 A schematic diagram of a sequence for determining a current structure to be processed in one embodiment;

[0032] Figure 4 is a schematic diagram of network layers and nodes in one embodiment;

[0033] Figure 5 A schematic diagram of a sequence for determining a current structure to be processed in another embodiment;

[0034] Figure 6 A schematic diagram of a processing flow in which a target network structure includes a random deactivation unit in an embodiment;

[0035] Figure 7 A schematic diagram of a process of using an L1 regularization loss function to evaluate the model effect of a replaced network model in an embodiment;

[0036] Figure 8 The figure is a flow chart of evaluating the model effect of a trained network model by using an L1 regularization loss function in one embodiment;

[0037] Fig. 9 is a structural block diagram of a network model compression device in one embodiment;

[0038] Fig.10 FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] Figure 1 FIG. 1 is an application environment diagram of a network model compression method in an embodiment. The network model compression method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 10 and the server 20 communicate through the network. The network model compression method of an embodiment of the present application can be run on the server 20, the terminal 10 sends the network model to be compressed to the server 20, and the server 20 obtains the network model to be compressed; determines the current structure to be processed in the network model to be compressed; searches for the target network structure from the search space, and uses the target network structure to replace the current structure to be processed to obtain the replaced network model; uses the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result; when the evaluation result meets the preset requirements, returns to the step of determining the current structure to be processed in the network model to be compressed, until all the structures to be processed in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model. Among them, the terminal 10 can be a desktop terminal or a mobile terminal, such as a desktop computer, a tablet computer, a laptop computer, a smart phone, etc. The server 20 can be an independent physical server, a physical server cluster or a virtual server.

[0041] In another embodiment, the network model compression method of the present application can be run on the terminal 10, and the terminal 10 obtains the network model to be compressed; determines the current structure to be processed in the network model to be compressed; searches for the target network structure from the search space, and uses the target network structure to replace the current structure to be processed to obtain the replaced network model; uses the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result; when the evaluation result meets the preset requirements, returns to the step of determining the current structure to be processed in the network model to be compressed, until all the structures to be processed in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0042] like Figure 2 As shown, in one embodiment, a network model compression method is provided. This embodiment mainly applies the method to the above Figure 1 The terminal 10 (or server 20) in FIG. Figure 2 , the network model compression method specifically includes the following steps:

[0043] S110, obtaining a network model to be compressed.

[0044] Among them, the network model to be compressed can be a network model with a complex and redundant structure due to a huge number of parameters, or a network model with compression requirements. Generally speaking, the size of the network is proportional to the effect, that is, the more network parameters a model has, the larger the model size is, and the higher the accuracy of the model is. For example, some deep learning models with better effects may have millions or even hundreds of millions of parameters. However, the huge number of parameters will also lead to high computing costs, thus limiting the application of network models in resource-constrained computing platforms such as Field-Programmable Gate Array (FPGA), Reduced Instruction Set Computer (RISC), and Advanced RISC Machine (ARM).

[0045] Based on this, the network model needs to be compressed to reduce the complexity of its model structure. Specifically, the compression of the network model refers to the process of removing redundant parameters and channels in the network model. For example, the component unit of the network model is the network layer. The network layer can be divided into convolutional layer, activation layer, pooling layer, mapping layer and batch normalization layer based on different functions. The compression of the neural network model can refer to changing, replacing or deleting the parameters of the network layer. Taking the convolutional layer as an example, model compression can refer to compressing the size and number of convolution kernels, where the size of the convolution kernel includes: number of channels, height and width.

[0046] S120, determining a current structure to be processed in the network model to be compressed.

[0047] The current structure to be processed refers to the network layer in the network model that needs to be compressed. After obtaining the network model to be compressed, the current structure to be processed can be determined according to preset rules. The preset rules may include preset processing order rules, such as determining the network layer structure that needs to be compressed in the network model in order from back to front as the current structure to be processed.

[0048] Specifically, a single-layer structure in the network model may be determined as the current structure to be processed, for example, a single-layer convolution layer, a single-layer pooling layer, etc. Since a single-layer network structure is processed, the accuracy of model compression is guaranteed. In addition, considering that the number of network layers in the network model is large, a multi-layer structure in the network model may be determined as the current structure to be processed, such as a multi-layer structure composed of multiple convolution layers, a multi-layer structure composed of multiple convolution layers and multiple pooling layers, etc., so as to ensure the efficiency of model compression.

[0049] S130, searching for a target network structure from the search space, and using the target network structure to replace the current structure to be processed, to obtain a replaced network model.

[0050] In the present application, when compressing the network model, Neural Architecture Search (NAS) is mainly used to design the architecture of the network model, and the original network layer of the network model (i.e., the current structure to be processed) is replaced by an optimized and simplified layer (i.e., the target network structure). The current structure to be processed and the search for the target network structure can be determined specifically by a controller in the neural architecture search, and the controller searches for the target network structure from the search space by using a search strategy.

[0051] Among them, the search strategy defines how the controller can quickly and accurately find a more suitable target network structure. The search strategy can be any one of random search, Bayesian optimization, transfer learning, reinforcement learning, evolutionary algorithm, genetic algorithm, greedy algorithm and gradient-based algorithm. The search space refers to a collection of multiple network units, including basic neural network units, such as convolution kernels, activation functions, pooling units, RNN (Recurrent Neural Network) units, etc. Among them, the convolution kernel refers to the weighted average of pixels in a small area in the input image when the input image is given during image processing, where the weight is defined by a function, which is called the convolution kernel. The activation function refers to the function running on the neuron, which is responsible for mapping the input of the neuron to the output. The pooling unit is mainly used for parameter dimensionality reduction, compressing the number of data and parameters, reducing overfitting, and improving the fault tolerance of the model. The RNN unit refers to a neural network unit used to process sequence data. In addition, the network unit can also include other units, such as random dropout units. Random dropout is a method for optimizing neural networks with deep structures. By randomly returning some weights or outputs of parameters to zero, the interdependence between nodes is reduced, thereby achieving regularization of the neural network and reducing its structural complexity.

[0052] The target network structure refers to a structure composed of network units in the search space, for example, a structure composed of multiple convolution kernels, a structure composed of multiple pooling units, etc. After determining the current structure to be processed, the controller of the neural architecture search can search for the target network structure from the search space according to the search strategy, and replace the current structure to be processed with the target network structure to obtain a replaced network model.

[0053] S140, using the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result.

[0054] L1 regularization loss function refers to the use of L1 regularization in the loss function, wherein the loss function (LossFunction) is a function that maps the value of a random event or its related random variable to a non-negative real number to represent the "risk" or "loss" of the random event. In this application, it can be used to measure the model effect of the model. The loss function can specifically select mean square error, cross entropy, Hinge Loss, etc. L1 regularization refers to the process of bringing the weight of the parameters in the network model closer to 0. In the network model, if there is a relationship between the various parameters, the complexity of the network model will be increased, and the ability to interpret the network model will not be improved. Therefore, parameter selection is required to make the network model easy to interpret. For example, when the cause of triggering event A needs to be analyzed through the network model, assuming that there are 1000 possible influencing factors, the analysis is more difficult. At this time, if the network model is trained so that the weight of a part of the influencing factors is 0, then when analyzing according to the remaining influencing factors, the difficulty of analysis can be greatly reduced. Therefore, by using L1 regularization processing, the network model parameters can be automatically selected. The weights of the network model are mainly concentrated on the parameters with high importance. For unimportant parameters, the weights will quickly tend to 0, making the network model sparse and achieving the purpose of network model compression.

[0055] Specifically, let L1 regularized loss function be L, the loss function without regularization be L0, the total number of network layers of the network model be H, and the weight vector of each layer be W k ,k=1,2,...,H. Assuming that the number of layers of the network model and the number of parameters of each layer are regularized, the expression of the L1 regularized loss function is:

[0056]

[0057] Among them, λ and μ k ,k=1,2,...,H is the regularization coefficient, which needs to be solved in the model effect evaluation.

[0058] After replacing the current structure to be processed with the target network structure and obtaining the replaced network model, the controller of the neural architecture search uses the L1 regularization loss function to evaluate the model effect of the replaced network model. The model effect evaluation process includes L1 regularization of the parameters of the network model, so that while obtaining the model effect evaluation results, the parameters of the network model can be reduced and the complexity of the model structure can be reduced.

[0059] S150, when the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all the structures to be processed in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0060] After obtaining the evaluation results, the controller of the neural architecture search judges the evaluation results to determine whether the evaluation results of the replaced model meet the preset requirements. If not, the controller can re-search for a new target network structure and repeat the above-mentioned network model structure replacement and network model effect evaluation process until a target network structure that meets the requirements is found. In addition, the process of the neural architecture search controller performing target network structure search, network model structure replacement, and network model effect evaluation can be considered as a controller iteration (Controller Epoch) process. During the controller iteration process, it also includes: when the number of controller iterations reaches a preset number (such as 2000 times), if a target network structure that meets the requirements has not been searched, the original structure of the current structure to be processed can be kept unchanged, that is, the current structure to be processed is not subjected to structural replacement processing.

[0061] It should be noted that the compression process of the network model in the present application is an iterative processing process. In the iterative processing process, a single processing process includes structural replacement of all current structures to be processed in the network model, and model effect evaluation of the corresponding replaced network model. When the iterative processing process of the network model meets the preset iteration completion condition, it can be confirmed that the network model compression is completed, and the preset iteration completion condition includes at least one of the preset number of iterations and the preset iteration duration.

[0062] This embodiment provides a network model compression method, which compresses the network model through neural architecture search to ensure the accuracy of the network model. At the same time, L1 regularization processing can effectively reduce the number of parameters of the network model and reduce the complexity of the network model, thereby effectively improving the problem of structural redundancy of the network model.

[0063] In one embodiment, the network model to be compressed is a deep neural network (DNN). A neural network generally includes an input layer, a hidden layer, and an output layer. A deep neural network refers to a neural network containing multiple hidden layers. Since the number of hidden layers is large, the number of parameters of the deep neural network is also large. Therefore, the deep neural network can be compressed to reduce the number of its parameters and reduce the complexity of the network structure.

[0064] When compressing a deep neural network, determining a current structure to be processed in the network model to be compressed includes: determining a single-layer structure or a multi-layer structure in the network model to be compressed as the current structure to be processed in the reverse order of the output layer, the hidden layer, and the input layer.

[0065] Specifically, Figure 3 As shown in the figure, it is a schematic diagram of the sequence of the controller determining the current structure to be processed. The deep neural network includes an input layer A, a hidden layer B and an output layer C. The hidden layer B includes layers B1 to B n The more complex the structure of the network model is, the larger the value of n is. The direction of the arrow in the figure is the direction of determining the current structure to be processed. The processor first determines the output layer C of the deep neural network as the current structure to be processed; then, when determining the current structure to be processed from multiple hidden layers B, it can be determined according to the hidden layer B connected to the output layer. n The current structure to be processed is determined in the order of the hidden layer B1 connected to the input layer; finally, the output layer A of the deep neural network is determined as the current structure to be processed. Since there is a certain error in the output of the neural network, processing in the reverse order of the network structure, that is, optimizing the network structure and updating the weights from the output layer, can reduce the error and improve the model accuracy.

[0066] When the controller determines that a single-layer structure or a multi-layer structure is the current structure to be processed, it can be determined based on the compression progress. For example, in the early stage of model compression, a multi-layer structure with a large number of layers can be selected as the current structure to be processed, thereby speeding up the compression progress; in the middle stage of model compression, the number of layers of the multi-layer structure can be gradually reduced; in the late stage of model compression, a single-layer structure can be selected as the previous structure to be processed, thereby ensuring compression accuracy. In addition, the controller can also be determined based on the weights of each layer structure. For example, for a network layer with a larger weight, the corresponding single-layer structure can be determined as the current structure to be processed; for a network layer with a smaller weight, the corresponding multi-layer structure can be determined as the current structure to be processed after the corresponding network layer is composed into a multi-layer structure. In addition, the processor can also determine the specific number of layers of the current structure to be processed based on the search strategy.

[0067] In one embodiment, the neural architecture search may adopt an Efficient Neural Architecture Search (ENAS) model. ENAS defines the concept of a node. The function of a node is similar to that of a network layer in a network model. The difference is that network layers are interdependent, that is, a subsequent network layer is attached to a previous network layer, while a node can arbitrarily replace a previous input. Specifically, Figure 4As shown in the figure, efficient neural architecture search needs to learn and select the connection relationship between nodes. Different connections will produce different network structures. Selecting the optimal structure from different network structures is equivalent to "designing" a new network model.

[0068] refer to Figure 4 , the first network model, which is composed of network layers connected in a straight line, is different from the second network model, which is composed of nodes cross-connected. The two are different computational graphs (Graph), and the internal event (Checkpoint) of the first network model that exports the model weights cannot be imported into the second network model, but intuitively the position of the nodes has not changed. If the input and output tensors (Tensor) and shapes (Shape) remain unchanged, the number of weights of these nodes is the same, that is, the weights of each network layer on the left can be copied to the nodes on the right. Based on the above principle, ENAS can achieve the purpose of weight sharing. Specifically, a fixed number of nodes is first defined, and then a set of parameters is used to control the preceding nodes connected to each node. This set of parameters is the parameters finally selected to represent the fixed network structure. This set of parameters can be tuned and selected through optimization algorithms, such as Bayesian optimization and DQN (Deep Q-Learning).

[0069] In one embodiment, according to the reverse order of the output layer, the hidden layer, and the input layer, a single-layer structure or a multi-layer structure in the network model to be compressed is determined as the current structure to be processed, including: according to the reverse order of the output layer, the hidden layer, and the input layer, based on the evaluation result of the replaced network model after the last replacement, the single-layer structure or the multi-layer structure in the network model to be compressed is determined as the current structure to be processed.

[0070] Specifically, Figure 5 As shown, when the controller determines the current structure to be processed, it can also determine it in combination with the model effect evaluation result of the network model after the last replacement. Figure 5 The dashed box in the middle represents the current structure to be processed. For example, the B i Layer and B i-1 The structure of the layers is relatively close. i After the structure of the layer is replaced, if the controller determines that the model effect evaluation result of the replaced network model is good, it can continue to replace the B i-1 The layer is replaced structurally, thereby further improving the model effect. When processing the new network layer, the controller replacement efficiency and model effect can be improved by combining the historical replacement record information.

[0071] In one embodiment, the target network structure includes one or more basic neural network units and random inactivation units. When the controller searches for the target network structure in the search space, it can select different types of units to combine to obtain the target network structure, for example, a target network structure composed of pooling units and convolution kernels; it can also select multiple units of the same type to combine to obtain the target network structure, for example, a target network structure composed of multiple convolution kernels; in addition, the parameters of the same type of units can also be different specifications, for example, a target network structure composed of multiple convolution kernels of different sizes and different moving steps, a target network structure composed of pooling units of different pooling sizes, pooling steps, and pooling types, etc.

[0072] In one embodiment, when the target network structure includes a randomly deactivated unit, the L1 regularized loss function is used to evaluate the model effect of the replaced network model to obtain an evaluation result, including: randomly setting one or more parameters connected to the target network structure in the replaced network model to an invalid state, and then using the L1 regularized loss function to evaluate the model effect of the replaced network model to obtain an evaluation result.

[0073] The principle of random dropout is to randomly set some parameters to an invalid state during each iteration, so that only some parameters in the network model are valid in each iteration. Therefore, as the number of iterations increases, the sensitivity between model parameters will decrease because different parameters may become invalid at any time.

[0074] like Figure 6 As shown in B i-1 Layer and B i The target network structure includes random dropout units as an example. In the network model before replacement, B i-1 The layer includes parameters M1, M2, M3, etc. i The layer includes parameters N1, N2, N3, N4, etc., and replaces B with the target network structure containing random inactivation units. i After the layer, the parameters M1, M2, and M3 are connected to the random dropout unit through a channel. At this time, when evaluating the model effect, the internal processing of the random dropout unit randomly sets one or more parameters to an invalid state. For example, parameter M2 is set to an invalid state, that is, the random dropout unit only uses parameters M1 and M3. It should be noted that the setting of the parameter invalid state is only performed inside the random dropout unit, that is, the random dropout unit does not affect B. i-1 The actual state of the layer's parameters is affected.

[0075] In one embodiment, Figure 7As shown, the L1 regularization loss function is used to evaluate the model effect of the replaced network model to obtain an evaluation result, including steps S142 to S146.

[0076] S142, obtaining sample data, where the sample data includes training sample data and test sample data;

[0077] S144, training the replaced network model using the training sample data to obtain a trained network model;

[0078] S146, based on the test sample data, the model effect of the trained network model is evaluated by using the L1 regularization loss function to obtain an evaluation result.

[0079] Alternatively, if Figure 8 As shown, the model effect of the trained network model is evaluated by using the L1 regularization loss function to obtain an evaluation result, including steps S1462 to S1464.

[0080] S1462, performing L1 regularization processing on each network layer of the trained network model by using an L1 regularization loss function, wherein the number of parameters with a weight of 0 in the trained network model after the processing is greater than the number of parameters with a weight of 0 in the trained network model before the processing;

[0081] S1464, performing a model effect evaluation on the processed trained network model to obtain an evaluation result.

[0082] Specifically, after obtaining the overall sample data uploaded by the user, the overall sample data is broken up and segmented to obtain training sample data and test sample data. In addition, the divided training sample data and test sample data can also be directly obtained. Subsequently, the replaced network model is trained using the training sample data, and the training process specifically includes alternating updates of parameters and parameter weights, etc., until the replaced network model meets the convergence conditions. After the training is completed, the model effect is evaluated using the test sample data and the L1 regularization loss function to obtain an evaluation result, so that it can be determined based on the evaluation result whether the current structure replacement operation has a better effect than the original structure model.

[0083] In one embodiment, the evaluation result includes an accuracy evaluation result and a complexity evaluation result.

[0084] Model accuracy can be used to measure the performance of the model in achieving its functions. For example, when the model is applied to image recognition, model accuracy can specifically refer to the image recognition rate. When the model is applied to image classification, model accuracy can specifically refer to the accuracy of image classification, etc. The accuracy evaluation result can be obtained by using the test sample data as the input of the trained network model and calculating the loss value between the output of the trained network model and the expected result through the loss function.

[0085] Model complexity can be divided into time complexity and space complexity, among which time complexity determines the training / prediction time of the model. If the time complexity is too high, it will take a lot of time to train and predict the model, and it will be impossible to quickly verify ideas and improve models, nor to make fast predictions. Space complexity determines the number of parameters of the model. Due to the limitations of the Curse of Dimensionality, the more parameters a model has, the more data is required to train the model, and the data sets in actual situations are usually not too large, which will make the model training more prone to overfitting. The time complexity can be specifically reflected by the number of operations of the model, and the space complexity can be specifically reflected by the volume of the model itself.

[0086] In one embodiment, the target network structure is used to replace the current structure to be processed to obtain a replaced network model, including: when there is a historical target network structure that is consistent with the structure of the target network structure, the weight of the historical target network structure is set as the initial weight of the target network structure, and the target network structure is used to replace the current structure to be processed to obtain the replaced network model.

[0087] Specifically, in the compression process of the network model, based on the principle of parameter weight sharing, when performing structural replacement of the network model, the historical information of structural replacement can be combined to improve the processing efficiency. For example, in the i-th processing process, the target network structure S is used to replace the current structure to be processed, and the model effect evaluation result of the replaced model is good. In this processing process, the weight of the target network structure S is determined to be α S . Then, in the jth processing, if the target network structure S is used again for structural replacement, α can be directly replaced S As the initial weight of the target network structure S in the jth processing process. Generally speaking, the initial weight of the new network structure is often 0 or a random value. After using 0 or a random value as the initial weight, it takes a long time to retrain the model. S As the initial weight in the j-th processing process, it is equivalent to training on the previous training results, which can effectively reduce the model training time and improve the training efficiency.

[0088] In the jth processing process, when the target network structure S is used to replace the new current structure to be processed, when evaluating the model effect of the new replaced network model, the initial weight α can be directly used to evaluate the model effect of the new replaced network model. S , evaluate the model effect of the new replaced network model.

[0089] It should be understood under reasonable conditions that although the various steps in the flowcharts involved in the foregoing embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in each flowchart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0090] In one embodiment, Fig. 9 As shown, a network model compression device is provided, which includes: a model acquisition module 110, a structure determination module 120, a structure replacement module 130 and an effect evaluation module 140.

[0091] The model acquisition module 110 is used to obtain the network model to be compressed;

[0092] The structure determination module 120 is used to determine the current structure to be processed in the network model to be compressed;

[0093] The structure replacement module 130 is used to search for a target network structure from the search space, and use the target network structure to replace the current structure to be processed to obtain a replaced network model;

[0094] The effect evaluation module 140 is used to use the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result; when the evaluation result meets the preset requirements, it returns to the step of determining the current structure to be processed in the network model to be compressed, until all the structures to be processed in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0095] The present application provides a network model compression device, which compresses the network model through neural architecture search to ensure the accuracy of the network model. At the same time, through L1 regularization processing, the number of parameters of the network model can be effectively reduced, and the complexity of the network model can be reduced, thereby effectively improving the problem of structural redundancy of the network model.

[0096] In one embodiment, the structure determination module 120 is used to determine the single-layer structure or multi-layer structure in the network model to be compressed as the current structure to be processed in the reverse order of output layer, hidden layer, and input layer.

[0097] In one embodiment, the structure determination module 120 is used to determine the single-layer structure or multi-layer structure in the network model to be compressed as the current structure to be processed based on the evaluation result of the replaced network model after the last replacement in the reverse order of the output layer, the hidden layer, and the input layer.

[0098] In one embodiment, the effect evaluation module 140 is used to: when the target network structure includes a randomly deactivated unit, randomly set one or more parameters connected to the target network structure in the replaced network model to an invalid state, and then use the L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result.

[0099] In one embodiment, the effect evaluation module 140 is used to: obtain sample data, the sample data includes training sample data and test sample data; use the training sample data to train the replaced network model to obtain the trained network model; according to the test sample data, use the L1 regularization loss function to evaluate the model effect of the trained network model to obtain an evaluation result.

[0100] In one embodiment, the structure replacement module 130 is used to: when there is a historical target network structure that is consistent with the structure of the target network structure, after setting the weight of the historical target network structure as the initial weight of the target network structure, use the target network structure to replace the current structure to be processed to obtain a replaced network model.

[0101] Fig.10 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 10 (or server 20) in FIG. Fig.10 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the network model compression method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can execute the network model compression method. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0102] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In one embodiment, the network model compression device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.10 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the network model compression device, such as: Fig. 9 The model acquisition module, structure determination module, structure replacement module and effect evaluation module shown in the figure. The computer program composed of each program module enables the processor to execute the steps in the network model compression method of each embodiment of the present application described in this specification.

[0104] For example, Fig.10 The computer device shown can be Fig. 9 The model acquisition module in the network model compression device shown acquires the network model to be compressed. The computer device can determine the current structure to be processed in the network model to be compressed through the structure determination module. The computer device can search for the target network structure from the search space through the structure replacement module, and use the target network structure to replace the current structure to be processed to obtain the replaced network model; the computer device can use the L1 regularization loss function to evaluate the model effect of the replaced network model through the effect evaluation module to obtain an evaluation result; when the evaluation result meets the preset requirements, return to the step of determining the current structure to be processed in the network model to be compressed, until all processing structures in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model.

[0105] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the network model compression method described above. The steps of the network model compression method here may be the steps of the network model compression method in each of the above embodiments.

[0106] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the network model compression method described above. The steps of the network model compression method here can be the steps of the network model compression method in each of the above embodiments.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0108] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A network model compression method, It is characterized in that include: Obtaining a network model to be compressed; the network model to be compressed is a network model applied to the field of image processing technology, and has a complex and redundant structure due to a huge number of parameters; the network model to be compressed is a deep neural network; the deep neural network includes an input layer, a hidden layer, and an output layer connected in sequence; According to the reverse order of the output layer, the hidden layer, and the input layer, a portion of the network layers in the network model to be compressed is determined as the current structure to be processed; Searching for a target network structure from the search space, and using the target network structure to replace the current structure to be processed, to obtain a replaced network model; Using an L1 regularization loss function to evaluate the model effect of the replaced network model, obtaining an evaluation result; When the evaluation result meets the preset requirements, return to the step of determining a part of the network layers in the network model to be compressed as the current structure to be processed in the reverse order of the output layer, the hidden layer, and the input layer, until all the structures to be processed in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model; The determining that a portion of network layers in the network model to be compressed is a current structure to be processed includes: In the early stage of model compression, a multi-layer structure with a large number of layers is selected as the current structure to be processed; in the middle stage of model compression, the number of layers of the multi-layer structure is gradually reduced; in the late stage of model compression, a single-layer structure is selected as the current structure to be processed; or For the network layer with a larger weight, the corresponding single-layer structure is determined as the current structure to be processed; for the network layer with a smaller weight, after the corresponding network layers are organized into a multi-layer structure, the corresponding multi-layer structure is determined as the current structure to be processed.

2. The method according to claim 1, It is characterized in that According to the reverse order of the output layer, the hidden layer, and the input layer, determining a portion of the network layers in the network model to be compressed as the current structure to be processed includes: According to the reverse order of the output layer, the hidden layer, and the input layer, based on the evaluation result of the replaced network model after the last replacement, the single-layer structure or the multi-layer structure in the network model to be compressed is determined as the current structure to be processed.

3. The method according to claim 1, It is characterized in that The target network structure includes one or more basic neural network units and random inactivation units.

4. The method according to claim 3, It is characterized in that When the target network structure includes a random dropout unit, the L1 regularization loss function is used to evaluate the model effect of the replaced network model to obtain an evaluation result, including: After randomly setting one or more parameters connected to the target network structure in the replaced network model to an invalid state, an L1 regularization loss function is used to evaluate the model effect of the replaced network model to obtain an evaluation result.

5. The method according to claim 1, It is characterized in that The L1 regularized loss function is used to evaluate the model effect of the replaced network model to obtain evaluation results, including: Acquire sample data, wherein the sample data includes training sample data and test sample data; Using the training sample data to train the replaced network model to obtain a trained network model; According to the test sample data, the model effect of the trained network model is evaluated by using the L1 regularization loss function to obtain an evaluation result.

6. The method according to claim 1 or 5, It is characterized in that The evaluation results include accuracy evaluation results and complexity evaluation results.

7. The method according to claim 5, It is characterized in that According to the test sample data, the model effect of the trained network model is evaluated by using the L1 regularization loss function to obtain an evaluation result, including: Performing L1 regularization processing on each network layer of the trained network model by using an L1 regularization loss function, wherein the number of parameters with a weight of 0 in the trained network model after the processing is greater than the number of parameters with a weight of 0 in the trained network model before the processing; The processed trained network model is evaluated for model effect to obtain an evaluation result.

8. The method according to claim 1, It is characterized in that Using the target network structure to replace the current structure to be processed to obtain a replaced network model includes: When there is a historical target network structure that is consistent with the structure of the target network structure, after setting the weight of the historical target network structure as the initial weight of the target network structure, the target network structure is used to replace the current structure to be processed to obtain a replaced network model.

9. A network model compression device, It is characterized in that include: A model acquisition module is used to acquire a network model to be compressed; the network model to be compressed is a network model applied to the field of image processing technology, and has a complex and redundant structure due to a huge number of parameters; the network model to be compressed is a deep neural network; the deep neural network includes an input layer, a hidden layer and an output layer connected in sequence; A structure determination module, used to determine a portion of network layers in the network model to be compressed as a current structure to be processed in the reverse order of output layer, hidden layer, and input layer; A structure replacement module, used to search for a target network structure from the search space, and use the target network structure to replace the current structure to be processed, so as to obtain a replaced network model; An effect evaluation module is used to use an L1 regularization loss function to evaluate the model effect of the replaced network model to obtain an evaluation result; when the evaluation result meets the preset requirements, return to the step of determining a part of the network layers in the network model to be compressed as the current structure to be processed in the reverse order of the output layer, the hidden layer, and the input layer, until all the structures to be processed in the network model to be compressed are processed, and the finally obtained replaced network model is used as the compressed network model; The structure determination module is specifically used to implement any one of the following two items: The first item: In the early stage of model compression, a multi-layer structure with a large number of layers is selected as the current structure to be processed; in the middle stage of model compression, the number of layers of the multi-layer structure is gradually reduced; in the late stage of model compression, a single-layer structure is selected as the current structure to be processed; The second item: for the network layer with a larger weight, the corresponding single-layer structure is determined as the current structure to be processed; for the network layer with a smaller weight, after the corresponding network layer is organized into a multi-layer structure, the corresponding multi-layer structure is determined as the current structure to be processed.

10. The device according to claim 9, It is characterized in that The structure determination module is used to: According to the reverse order of the output layer, the hidden layer, and the input layer, based on the evaluation result of the replaced network model after the last replacement, the single-layer structure or the multi-layer structure in the network model to be compressed is determined as the current structure to be processed.

11. The device according to claim 9, It is characterized in that The effect evaluation module is used to implement any one of the following: The first item: when the target network structure includes a random inactivation unit, after randomly setting one or more parameters connected to the target network structure in the replaced network model to an invalid state, the replaced network model is evaluated using an L1 regularization loss function to obtain an evaluation result; The second item: obtaining sample data, wherein the sample data includes training sample data and test sample data; using the training sample data to train the replaced network model to obtain a trained network model; based on the test sample data, evaluating the model effect of the trained network model through an L1 regularization loss function to obtain an evaluation result.

12. The device according to claim 11, It is characterized in that The effect evaluation module is specifically used for: Performing L1 regularization processing on each network layer of the trained network model by using an L1 regularization loss function, wherein the number of parameters with a weight of 0 in the trained network model after the processing is greater than the number of parameters with a weight of 0 in the trained network model before the processing; The processed trained network model is evaluated for model effect to obtain an evaluation result.

13. The device according to claim 9, It is characterized in that The structure replacement module is used to: when there is a historical target network structure that is consistent with the structure of the target network structure, after setting the weight of the historical target network structure as the initial weight of the target network structure, use the target network structure to replace the current structure to be processed to obtain a replaced network model.

14. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.