Robust model compression method for distributed external scene image recognition

Abstract

The invention discloses a robust model compression method for out-of-distribution scene image recognition, which is characterized in that on one hand, model parameters are finely adjusted by using training data and the characterization capability of in-distribution samples is inherited, on the other hand, an over-fitting avoiding method is combined, the generalization capability of out-of-distribution samples is ensured by using sparse optimization screening parameters, the technology is easy to implement, and the universality is relatively high. The method mainly comprises the steps that Dropout is added in front of a feature processing part and a final classification layer in a pre-pruning stage, an L1 regular term is introduced to adjust an original model structure, sparsity is enhanced, and meanwhile overfitting caused by follow-up pruning is avoided; a Dropout layer is shielded in the pruning stage, and a Taylor expansion is used for evaluating the influence of all parameters on model loss before and after zero setting to serve as importance indexes; based on the evaluation result and the compression ratio, removing the parameter with the lowest importance degree; and finally recovering model performance through fine tuning. Parameter evaluation and fine adjustment are carried out at the same time, and evaluation is continuously carried out according to current parameters so as to adapt to importance changes caused by parameter adjustment.

Description

Technical Field

[0001] The invention relates to a robust model compression method for out-of-distribution scene image recognition, and belongs to the technical field of machine learning and image classification application. Background Art

[0002] Image recognition models based on deep learning have made significant progress in many application tasks, such as image classification, object detection, image segmentation, etc. These achievements are attributed to the widely available training data on the one hand, and the model architecture that can absorb a large number of parameters on the other hand. However, a large number of model parameters leads to a significant increase in computing and storage costs. How to reduce costs, store efficiently, and quickly complete model inference is a challenging problem.

[0003] To address these challenges, neural network compression is a mainstream technology for image recognition models, focusing on efficient model storage and reasoning. Its representative technology is the model pruning algorithm, which aims to use the sparsity inherent in neural networks to delete "less important" weight parameters, thereby reducing the number of parameters and computing load. Specifically, the model pruning algorithm accelerates model reasoning and deployment by minimizing hardware storage and computing requirements, while trying to maintain the accuracy of the model's recognition of images.

[0004] However, a large number of experiments have shown that as the compression rate (pruning rate) increases, although the existing pruning algorithms can maintain the accuracy of the image recognition model on independent and identically distributed data, their performance is often poor when it comes to out-of-distribution samples in open scenarios, and they are difficult to migrate and lack generalization, which seriously affects the versatility of the model in downstream scenarios. Summary of the invention

[0005] Purpose of the invention: After analyzing the reasons for the defects of the prior art, it was found that a key factor is the limited observation data, which results in the model pruning process only retaining the feature recognition ability of the training image data, and it is difficult to take into account the feature recognition ability of the test samples, resulting in performance degradation and overfitting, and it is difficult to robustly generalize for tasks outside the training data. In view of the problem that the image recognition model in the prior art is greatly limited in the application scenarios after pruning, a two-stage model pruning framework TSP is proposed, which aims to optimize the model structure and enhance sparsity through pre-pruning, retain the generalization ability of the original model, and thus be suitable for other downstream tasks.

[0006] Technical solution: A robust model compression method for out-of-distribution scene image recognition, including:

[0007] 1) obtaining an original image recognition model and a data set to be pruned, wherein the training method of the original image recognition model includes a known loss calculation formula, and the data set may be a conventional data set or a multi-domain data set;

[0008] 2) Adjust the structure of the original image recognition model, add Dropout before the feature processing and final classification layer of the original image recognition model, and introduce L1 as the fine-tuning target loss in the pre-pruning stage;

[0009] 3) The Dropout layer is masked and the Taylor expansion is used to evaluate the influence of all parameters on the loss of the image recognition model before and after they are set to zero as an importance indicator;

[0010] 4) Based on the evaluation results, gradually select the parameters with the lowest importance until the pruning rate requirements are met;

[0011] 5) The final image recognition model is obtained by fine-tuning the recovery performance.

[0012] The image recognition model may be other recognition models.

[0013] The training method and necessary training details of the original image recognition model are known so as to be used to restore the image recognition model performance later.

[0014] The training dataset can be a regular dataset such as ImageNet, or a multi-domain dataset such as PACS.

[0015] The Dropout layer needs to be placed after the batch normalization layer and before the linear classification layer to ensure the correct parameter normalization relationship.

[0016] The Taylor expansion is used to evaluate the influence of all parameters on the change of model loss before and after being set to zero as an importance indicator, and the steps are as follows:

[0017] 31) The classification loss plus the Coral regularization term is used as the loss in the evaluation basis;

[0018] 32) Use Taylor expansion to approximate the change in loss before and after the parameters are set to zero;

[0019] 33) Filter out the parameter with the smallest change, i.e. the “least important parameter”, as the parameter to be pruned in the current step.

[0020] A robust model compression device for out-of-distribution scene image recognition, the device comprising:

[0021] An acquisition unit, used for acquiring an original image recognition model and a data set to be pruned, wherein the training method of the original image recognition model including the loss calculation formula is known, and the data set may be a conventional data set or a multi-domain data set;

[0022] The first reconstruction unit is used to transform the original image recognition model structure and add a Dropout layer to adjust the relationship between different parameters;

[0023] The first fine-tuning unit is used to fine-tune the image recognition model. The L1 norm is introduced on the basis of the original training loss. On the one hand, as a typical overfitting mitigation method, it works together with the Dropout layer to encourage the image recognition model to learn common category features, thereby enhancing reliability and generalization ability; on the other hand, the sparse optimization of the L1 norm facilitates subsequent parameter screening and pruning;

[0024] The first pruning unit is used to perform pruning compression according to a given pruning rate. First, the influence of each parameter on the loss before and after being set to zero is evaluated as an importance indicator, and the parameters with the smallest influence are gradually eliminated until the given pruning rate requirement is reached;

[0025] The second fine-tuning unit fine-tunes the pruned image recognition model to restore performance. For the domain generalization task and the corresponding dataset, the fine-tuning loss is set to the original training loss plus the Coral constraint term to balance the loss gradients of different domains. The generalization ability of the final model is close to the original model.

[0026] The implementation process of the device is the same as the method and will not be described in detail.

[0027] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the robust model compression method for out-of-distribution scene image recognition as described above are implemented.

[0028] A computer-readable storage medium stores a computer program for executing the robust model compression method for out-of-distribution scene image recognition as described above.

[0029] Beneficial effects: In order to solve the problem of reduced generalization ability of out-of-distribution samples after pruning of image recognition models, the present invention provides a robust model compression method and device for out-of-distribution scene image recognition, which consists of two stages: a fine-tuning (pre-pruning) stage and a pruning stage including parameter evaluation. Compared with the prior art, the present invention focuses on the overfitting caused by changes in the model structure when relying solely on observed data. A dropout layer is introduced in the fine-tuning stage to reduce the mutual dependence between parameters, and sparse optimization is used to promote the subsequent pruning process, which effectively improves the classification ability and versatility of samples of other styles or forms outside the training data after the image recognition model is compressed. In addition, the present invention supports the original model training method, is suitable for initial models and image recognition tasks of different architectures, and has certain robustness and versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0031] Figure 2 It is an overall framework diagram of an embodiment of the present invention;

[0032] Figure 3 is a flowchart of modifying and fine-tuning the model structure in step (2) in an embodiment of the present invention;

[0033] Figure 4 is a flow chart of the pruning phase in steps (3), (4), and (5) in an embodiment of the present invention;

[0034] Figure 5 It is a block diagram of the composition of a robust model compression device for distributed out-of-scene image recognition implemented in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0036] like Figure 1 As shown in FIG. 1 , the robust model compression method for out-of-distribution scene image recognition mainly includes:

[0037] (1) Obtaining the original image recognition model to be pruned and the data set corresponding to the image recognition model, wherein the training method of the original image recognition model including the loss calculation formula is known, so as to be subsequently used to restore the performance of the image recognition model, and the data set may be a conventional data set or a multi-domain data set. In the embodiment of the present invention, taking the multi-domain data set as an example, it contains a number of image samples of different styles, and the performance of the pruned model on the style images appearing in the training data needs to be guaranteed.

[0038] In the embodiment of the present invention, the process of acquiring the target image data set may be performed according to an existing acquisition method, for example, an interface dedicated to the target image data is set to acquire the target data set.

[0039] (2) Adjust and optimize the model structure. Adding the Dropout layer before the batch normalization layer will cause an offset between the variances of training and testing, that is, the variances with and without Dropout statistics are inconsistent. In order to avoid this problem, Dropout is added after the batch normalization layer of the original image recognition model. Taking the Resnet50 network as an example, in the prior art, Dropout is usually only used before the classification layer. In the example of the present invention, Dropout with different discarding ratios is introduced before both the convolution layer and the classification layer to decouple the parameters of each neuron and facilitate subsequent pruning.

[0040] The loss calculation formula for the subsequent fine-tuning stage is:

[0041]

[0042] in, is the original model training loss, the second term is the sum of the absolute values ​​of the model parameters, i.e., the L1 norm, and λ is the regularization term coefficient. Relatively complex models have larger L1 norm values. The addition of L1 minimizes the training error while keeping the model as simple as possible, which to some extent alleviates the problem of excessive complexity of the remaining parameters after pruning and the decline in generalization ability due to fitting the individual characteristics of the samples. The unique sparsity of the L1 regularization term can be regarded as "pre-pruning". Even if the parameters that have been lowered are subsequently set to zero, the impact on the classification surface of the sample space will be less destructive.

[0043] (3) Parameter importance evaluation. The calculation formula is as follows:

[0044]

[0045] Among them l I Indicates that when the model parameter is ω, it is in the training set D tr The loss on the parameter to be evaluated is i The change in model loss before and after zeroing is the parameter importance evaluation result I(ω i ). For multi-domain datasets, Add the Coral term to the original model training loss to align the second-order statistics between different domains to find domain-invariant relationships and screen parameters that are more beneficial to out-of-distribution samples:

[0046]

[0047] in is the Coral regularization term, d represents the output dimension of the fully connected layer of the model, and C S , C R is the feature covariance matrix of different sample domains.

[0048] To avoid the large amount of calculations caused by multiple predictions, the evaluation formula can be expanded using Taylor's formula, retaining the first-order derivative, i.e., the gradient g i , we get an approximate evaluation:

[0049] I(ω i )=(g i ·ω i ) 2

[0050] According to the evaluation results, find the "least important" parameter, which is the parameter to be pruned.

[0051] (4) Prune to the specified compression ratio. This process is carried out simultaneously with fine-tuning the recovery performance. After several iterations, the evaluation results of all parameters are obtained through the formula in step (3), and the n parameters with the smallest impact are set to zero. Then, fine-tuning iterations are continued until the total number of pruned parameters N reaches the pruning rate requirement:

[0052] P=N / M

[0053] Where M is the total number of parameters of the original image recognition model.

[0054] (5) The final image recognition model is obtained through fine-tuning. After pruning is completed, fine-tuning is continued for several epochs to restore the performance. Finally, the original model f is input and the original dataset D is obtained. tr When the pruning ratio is required to be P, the algorithm of the present invention The resulting image recognition model:

[0055]

[0056] The present invention focuses on the overfitting problem caused by focusing on the original training set during the model pruning process, which greatly limits the versatility of the model. The present invention introduces a pre-pruning stage to guide the enhancement of model sparsity while retaining the ability to recognize common features in the data set, thereby alleviating overfitting; and the process of filtering parameters based on the original data set ensures the model performance within the distribution. Compared with existing pruning and compression algorithms, the image recognition model obtained by the present invention is more applicable in relatively complex downstream tasks and application scenarios.

[0057] like Figure 5 As shown, the present invention is used for a robust model compression device for out-of-distribution scene image recognition, specifically comprising:

[0058] An acquisition unit, used for acquiring an original image recognition model and a data set to be pruned, wherein the training method of the original image recognition model including the loss calculation formula is known, and the data set may be a conventional data set or a multi-domain data set;

[0059] The first reconstruction unit is used to transform the original model structure and add a Dropout layer to adjust the relationship between different parameters;

[0060] The first fine-tuning unit is used to fine-tune the model. The L1 norm is introduced on the basis of the original training loss. On the one hand, as a typical overfitting mitigation method, it works together with the Dropout layer to encourage the model to learn common category features, thereby enhancing reliability and generalization ability. On the other hand, the sparse optimization of the L1 norm facilitates subsequent parameter screening and pruning.

[0061] The first pruning unit is used to perform pruning compression according to a given pruning rate. First, the influence of each parameter on the loss before and after being set to zero is evaluated as an importance indicator, and the parameters with the smallest influence are gradually eliminated until the given pruning rate requirement is reached;

[0062] The second fine-tuning unit fine-tunes the pruned model to restore performance. For the domain generalization task and the corresponding dataset, the fine-tuning loss is set to the original training loss plus the Coral constraint term to balance the loss gradients of different domains. The generalization ability of the final model is close to the original model.

[0063] Further, such as Figure 5 As shown, the first pruning unit includes:

[0064] Acquisition module, used to evaluate and determine the parameters to be pruned;

[0065] Building blocks for updating model parameters in an iterative process.

[0066] This example is used to solve the problem of reduced generalization ability and robustness of data outside the distribution of the training set after pruning the deep learning-based image recognition model, and achieves the reduction of inference cost while largely guaranteeing the performance in applicable scenarios consistent with the original model. The method and device of the present invention are mainly based on the following observations: Existing studies have shown that the neural network model itself has a certain sparsity, which is an important basis for the feasibility of network pruning; however, the existing pruning methods mainly focus on evaluating the performance of the pruned model on in-distribution data, while ignoring the generalization on out-of-distribution (OOD) data. The present invention focuses on the overfitting problem caused by the destruction and restoration of the classification surface during the pruning process, and enhances the sparsity of parameters and the robustness of subsequent pruning by introducing pre-pruning, and then uses the new importance evaluation score to optimize the parameter screening process to ensure that the final model is suitable for downstream tasks with different data distributions.

[0067] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0068] Obviously, those skilled in the art should understand that the various steps of the robust model compression technology for distributed out-of-scene image recognition or the various units of the robust model compression device for distributed out-of-scene image recognition of the above-mentioned embodiments of the present invention can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0069] This example verifies the effectiveness of the widely used out-of-distribution generalization datasets PACS and Office Home, which contain 7 and 65 samples respectively. The PACS dataset includes four different styles of pictures: art paintings, cartoons, photos, and stick figures, while the Office Home dataset includes four styles: art paintings, clipart, products, and photos. In the experiment, one style was selected as the final test environment, and the remaining three data were used to train the original model and used for pruning. The accuracy of the final model and the original model in the test environment under different pruning methods was compared. The comparison methods include the baseline pruning method (no pre-pruning stage, only the original training loss evaluation parameters are used), the DG pruning algorithm (adding the loss variance evaluation model parameters under different environments), the FPGM algorithm, and the SWAD algorithm (weighted averaging of parameters in the final fine-tuning stage). The experimental results show the average accuracy of different methods using the PACS and Office Home datasets and the Resnet50 model at a pruning rate of 50%.

[0070] PACS Experiment Results Office Home Experiment Results Baseline Pruning Methods 79.47 62.31 DG Pruning Algorithm 79.70 62.44 FPGM algorithm 63.90 50.73 SWAD algorithm 79.49 62.37 Method of the present invention 80.10 63.34

Claims

1. A robust model compression method for out-of-distribution scene image recognition, characterized in that: include: 1) obtaining an original image recognition model and a data set to be pruned, wherein the training method of the original image recognition model includes a known loss calculation formula; 2) Adjust the structure of the original image recognition model, add Dropout before the feature processing and final classification layer of the original image recognition model, introduce the L1 regularization term as the fine-tuning target loss in the pre-pruning stage, and use the original training data to fine-tune the modified image recognition model; 3) The Dropout layer is masked and the Taylor expansion is used to evaluate the influence of all parameters on the loss of the image recognition model before and after they are set to zero as an importance indicator; 4) Based on the evaluation results, gradually select the parameters with the lowest importance until the pruning rate requirements are met; 5) The final image recognition model is obtained by fine-tuning the recovery performance.

2. The robust model compression method for out-of-distribution scene image recognition according to claim 1, characterized in that: The training method of the original image recognition model is known.

3. The robust model compression method for distributed out-of-scene image recognition according to claim 1, characterized in that: The training dataset is the regular dataset ImageNet or the multi-domain dataset PACS.

4. The robust model compression method for out-of-distribution scene image recognition according to claim 1, characterized in that: The Dropout layer is placed after the batch normalization layer and before the linear classification layer. Dropout with different discarding ratios is introduced before the convolutional layer and the classification layer to decouple the parameters of each neuron.

5. The robust model compression method for out-of-distribution scene image recognition according to claim 1, characterized in that: The Taylor expansion is used to evaluate the influence of all parameters on the change of model loss before and after being set to zero as an importance indicator, and the steps are as follows: 31) The classification loss plus the Coral regularization term is used as the loss in the evaluation basis; 32) Use Taylor expansion to approximate the change in loss before and after the parameters are set to zero; 33) Select the parameter with the smallest change, i.e. the "least important parameter", as the parameter to be pruned in the current step.

6. The robust model compression method for out-of-distribution scene image recognition according to claim 5, characterized in that: In the step 3), the process of screening the least important parameters is carried out simultaneously with fine-tuning the recovery performance to obtain the importance index of the current parameters after adjustment; during the operation, an evaluation is performed every several iterations, and some parameters are removed until the pruning ratio requirement is met, and then fine-tuning is performed for several epochs.

7. A robust model compression device for out-of-distribution scene image recognition, characterized in that: The device comprises: An acquisition unit, used to acquire an original image recognition model and a data set to be pruned, wherein the training method of the original image recognition model includes a known loss calculation formula; The first reconstruction unit is used to transform the original image recognition model structure and add a Dropout layer to adjust the relationship between different parameters; The first fine-tuning unit is used to fine-tune the image recognition model and introduce the L1 norm based on the original training loss; The first pruning unit is used to perform pruning compression according to a given pruning rate. First, the influence of each parameter on the loss before and after being set to zero is evaluated as an importance indicator, and the parameters with the smallest influence are gradually eliminated until the given pruning rate requirement is reached; The second fine-tuning unit fine-tunes the pruned image recognition model to restore performance. For the domain generalization task and the corresponding dataset, the fine-tuning loss is set to the original training loss plus the Coral constraint term to balance the loss gradients of different domains.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the robust model compression method for out-of-distribution scene image recognition as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the robust model compression method for out-of-distribution scene image recognition as described in any one of claims 1-6.

Images