Double-stage model re-optimization method for deep convolution model

By adopting a two-stage model re-optimization method on edge devices, using multi-branch re-parameterized training module and knowledge distillation technology, the performance of the deep convolution model is improved, and the problem of insufficient model accuracy and general applicability on edge devices is solved. , realizes high-performance model deployment under cost and power consumption constraints.

CN119067195APending Publication Date: 2024-12-03SHANGHAI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411217885.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

On edge devices, the deep convolution model is insufficient in accuracy and general use, making it difficult to achieve the effect of high-performance models in the laboratory under conditions of limited cost and power consumption.

Method used

The two-stage model re-optimization method is adopted, and the original deep convolution model is transformed into a multi-branch re-parameterized training module, and the knowledge distillation framework is used for distillation training, converted into an intermediate model, and then restored to the original model to improve the model performance.

Benefits of technology

Without changing the model's calculation quantity, parameter quantity and inference speed, the performance of the deep convolution model is significantly improved. It is suitable for all convolution-based deep learning models, improving the capacity and feature expression capabilities of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067195A_ABST
    Figure CN119067195A_ABST
Patent Text Reader

Abstract

The invention discloses a double-stage model re-optimization method for a deep convolution model, and the method comprises the steps: transforming an original deep convolution model into an intermediate model which comprises a multi-branch re-parameterization training module in a first stage, and carrying out the distillation training of the intermediate model; in the second stage, a re-parameterization training module of the trained intermediate model is converted, so that the structure of the intermediate model is recovered to an original deep convolution model; performing actual deployment by using the optimized deep convolution model in the online stage; according to the method, the knowledge distillation framework based on the vector magnitude and the multi-branch re-parameterization training module are combined, and the performance of the deep convolution model is effectively improved under the condition that the calculation amount, the parameter amount, the model structure and the reasoning speed of an original model are not changed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural networks, specifically a two-stage model reoptimization method for deep convolutional models. Background Art

[0002] At present, deep learning algorithms have made in-depth development in various fields such as online transactions, intelligent security monitoring, and smart city construction, and have been widely used in identity recognition, voice interaction, and intelligent education. In order to improve the accuracy of the model in actual tasks, the number of parameters and computing resources required for methods based on deep neural networks are often relatively large. However, with the gradual productization of artificial intelligence technology and the development of intelligent applications towards miniaturization, more and more deep learning applications are required to be deployed on edge computing devices. In the artificial intelligence laboratory, the deep learning server is equipped with ample storage space and computing power to facilitate the efficient training and reasoning of large-scale deep models. In contrast, due to the strict control of cost and power consumption, the storage capacity and computing resources of edge device systems are often severely limited, resulting in the deployment of poor-performing deep models on them, which is difficult to achieve the performance of high-performance models in the laboratory. Summary of the invention

[0003] In order to address the problems in existing edge device application scenarios, such as insufficient model accuracy of deep convolutional models, insufficient versatility of original model selection, and failure to meet the requirements for improving the performance of original models before actual deployment, the present invention proposes a two-stage model reoptimization method for deep convolutional models. The method combines a vector-level knowledge distillation framework and a multi-branch re-parameterization training module to effectively improve the performance of deep convolutional models without changing the computational complexity, parameter quantity, model structure, and inference speed of the original model.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a two-stage model reoptimization method for a deep convolutional model. In the first stage, the original deep convolutional model is transformed into an intermediate model including a multi-branch reparameterized training module, and distillation training is performed on the intermediate model. In the second stage, the reparameterized training module of the trained intermediate model is transformed to restore the structure of the intermediate model to the original deep convolutional model. In the online stage, the optimized deep convolutional model is used for actual deployment.

[0006] The intermediate model is obtained by replacing the 3×3 convolution in the original deep convolutional model with a re-parameterized training module.

[0007] The distillation training refers to: using the intermediate model as a student model and another higher performance deep model as a teacher model in a knowledge distillation framework, and using the knowledge distillation framework to perform distillation training on the neural network.

[0008] The higher performance deep model is implemented using but not limited to the GroupFace network, the Wavlm model plus the X-Vector head network.

[0009] The re-parameterized training module is implemented by, but not limited to, a convolution module including multiple branches and multiple convolutions that is equivalent to a common convolution. Technical Effects

[0010] The present invention combines knowledge distillation technology and structural reparameterization technology, proposes the concept of an intermediate model, and uses a vector-level knowledge distillation framework to distill the intermediate model instead of directly distilling the original model. At the same time, the multi-branch reparameterization training module significantly improves the model capacity and feature expression ability during the training phase, and is equivalently converted to a single 3×3 convolution during the reasoning phase. Compared with the prior art, the present invention significantly improves the performance of the deep convolutional model without affecting the parameter quantity, network structure, reasoning speed, and computational complexity of the original model. It is simple to operate and has high versatility, and can be applied to all convolution-based deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of the present invention;

[0012] Figure 2 Schematic diagram of the re-parameterized training module;

[0013] In the figure: r represents the height and width of the feature map, c represents the number of channels of the feature map;

[0014] Figure 3 Schematic diagram of the reparameterization of convolution and BN layers. DETAILED DESCRIPTION

[0015] like Figure 1 As shown, this embodiment relates to a two-stage model optimization method for a deep convolutional model, which specifically includes:

[0016] Step 1: Re-parameterization training module replacement and distillation training, including:

[0017] 1.1 Replace all 3×3 convolutions in the original convolution model with Figure 2The multi-branch re-parameterized training module shown in the figure obtains the intermediate model. The re-parameterized training module includes: a 3×3 convolution branch, a 1×1 convolution branch, a residual connection branch, a mixed convolution branch and a function activation layer, wherein: the mixed convolution branch is based on an inverted bottleneck structure. First, a 1×1 convolution is used to increase the number of feature map channels to enrich the feature space. Then, a 3×3 convolution is used to extract spatial features on the feature map with amplified channels. Finally, a 1×1 convolution is used to reduce the number of channels again. The number of output feature map channels is the same as that of the input feature map channels. The residual connection branch is used to solve the gradient vanishing problem in deep networks, making the network easier to converge. The size of the output feature map of each branch is consistent with the size of the input feature map. The results of the four branches are added and fused and then passed through the function activation layer.

[0018] A batch normalization layer (BN) is provided after the convolutional layer in the four branches of the re-parameterized training module to standardize the feature map, which can accelerate the model convergence speed and prevent the model from overfitting.

[0019] A batch normalization layer is further added to the residual link branch.

[0020] 1.2 Use Figure 1 The knowledge distillation architecture shown performs distillation training on the intermediate model, including:

[0021] A) Use offline vector distillation on the overall distillation framework, and use the distillation loss to constrain the distance between the output feature vectors of the teacher model and the student model, and calculate the distillation loss between the output vector of the student model and the output vector of the teacher network model Among them: F t represents the output vector of the teacher model, F s represents the output vector of the student model, and m represents the number of samples.

[0022] The teacher model can be implemented by using a GroupFace network, a Wavlm model and an X-Vector first network.

[0023] B) Using the adversarial learning method, the adversarial loss L of the output feature vectors of the teacher model and the student model is calculated through a discriminator consisting of three fully connected layers. AS =[1-D(S(x))] 2 , where: D represents the discriminator and S represents the student model.

[0024] C) Using the classifier reuse method, in the final stage of training, the classifier of the student model is directly replaced with the teacher model and the parameters are fixed, so that the student model can learn the knowledge contained in the teacher model classifier, and calculate the loss function between the student model output and the data set label, so that the final output feature vector of the student model and the output feature vector of the teacher model will establish a close constraint relationship, so that the student model can better learn the knowledge of the teacher model, specifically: Among them: m represents the number of samples, s represents the modulus is a constant, and Θ represents the degree of the angle space.

[0025] D) Calculate the total output function L total =L dis +λ·L AS +β·L Arc , where: λ and β are hyperparameters used to adjust the ratio of the loss function, and the trained intermediate model is obtained through distillation training.

[0026] Step 2: Convert the intermediate model back to the original model, including:

[0027] 2.1 As Figure 3 As shown in the figure, the re-parameterized training module after training is converted into a 3×3 convolution through the model re-parameterization technology, specifically: the parameters of the serial 3×3 convolution and BN layer are merged into a single 3×3 convolution; the 1×1 convolution and BN layer are converted into 3×3 convolution; the parameters of the convolution kernel of the 1×1 convolution are filled with a circle of parameters with a value of 0 to convert it into a 3×3 convolution; the learnable scaling coefficients of the BN layer are rearranged, that is, it is regarded as a 1×1 convolution layer, and then a circle of parameters with a value of 0 are filled outside its own parameters. After the conversion, the three branches on the left side of the training module are transformed into a single 3×3 convolution, and the rightmost mixed convolution branch contains a series of 1×1 convolution, 3×3 convolution, and 1×1 convolution.

[0028] 2.2 The mixed convolution branches in 2.1 are fused, including:

[0029] i) The sequence of 1×1 convolution and 3×3 convolution is fused into a 3×3 convolution, that is, the linear weighted operation on the feature map in the 1×1 convolution is transformed into a linear weighted operation on the input channel dimension of the 3×3 convolution kernel, so that the 1×1 convolution and 3×3 convolution series sequence can be fused into a 3×3 convolution;

[0030] ii) Fuse the 3×3 convolution and 1×1 convolution sequence into a 3×3 convolution: Convert the linear combination of feature maps into a linear combination of the output channel dimensions of the 3×3 convolution kernel, and then fuse the sequence of 3×3 convolution and 1×1 convolution into a single 3×3 convolution. Finally, the sequence of 1×1 convolution, 3×3 convolution and 1×1 convolution can be converted into a 3×3 convolution.

[0031] 2.3 Combine steps 2.1 and 2.2 to transform the original re-parameterized training module into a single 3×3 convolution. Specifically, according to the additivity of convolution, the output feature map addition operation of multi-branch convolution can be transformed into the addition operation of multi-branch convolution kernels to transform the multi-branch structure into a single-branch structure. Then, the four parallel 3×3 convolutions can be fused into a single 3×3 convolution.

[0032] In terms of specific experiments, the pytorch framework was used on the experimental testing platform (GPU: RTX 3090), and two typical deep convolutional model application scenarios, face recognition and speaker recognition, were selected for experimental verification.

[0033] In the face recognition experiment, the GroupFace model is selected as the teacher model. The 3×3 convolutions in the original face recognition model are all replaced with multi-branch re-parameterized training modules to obtain an intermediate model. The intermediate model is distilled and trained using the distillation framework: During training, the input face image I∈R 3×112×112 After the network features are extracted, the teacher model and the student model output 512-dimensional feature vectors respectively: t ∈R 512×1×1 , I s ∈R 512×1×1 Then I s and I t Calculate the distillation loss function L dis , I s and I t Send it to the discriminator to calculate the adversarial loss function L AS , I t Calculate the arcface loss function L with the true label of the dataset Arc The three loss functions are weighted and added using hyperparameters λ and β to obtain the final loss function, and the parameters of the student model are updated. After the training is completed, the intermediate model is losslessly converted to the original model structure, that is, the re-parameterized training module in the intermediate model is converted to a single 3×3 convolution, thereby obtaining the original network structure without any performance loss.

[0034] This method is applied to multiple deep learning face recognition models and tested on three face verification datasets: LFW, CALFW, and CPLFW. It can be observed that after using the present invention, the accuracy of several face recognition models has been significantly improved, as shown in Table 1. The results in the table are accuracy ACC, and the higher the value, the stronger the model performance.

[0035] Table 1 Face recognition test data set accuracy test (ACC)

[0036] The results in Table 1 show that the present invention can effectively improve the performance of face recognition models based on deep convolutional networks.

[0037] In the speaker recognition experiment, the Wavlm model plus the X-Vector head was selected as the teacher model. The specific process is consistent with the face recognition experiment. The original speaker recognition model is first converted into an intermediate model, and then the intermediate model is distilled and trained. After the training is completed, the intermediate is losslessly converted back to the original model structure. The present invention is applied to two deep learning speaker recognition models and tested on three official tests of the VoxCeleb1 speaker recognition data set: Vox1-O, Vox1-E and Vox1-H. It can be observed that after using the present invention, the accuracy of the two speaker recognition models has been significantly improved, as shown in Table 2. The results in the table are equal error rates EER, and the lower the value, the stronger the model performance.

[0038] Table 2 Speaker recognition test data set accuracy test (EER)

[0039] The results in Table 2 show that the present invention can effectively improve the performance of the speaker recognition model based on the deep convolutional network.

[0040] The experimental results of two typical identity recognition tasks, face recognition and speaker recognition, prove that the present invention can effectively improve the performance of deep convolutional models without changing the number of model parameters, the amount of calculation, and the network results.

[0041] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A two-stage model reoptimization method for deep convolutional models, characterized in that: In the first stage, the original deep convolutional model is transformed into an intermediate model containing a multi-branch re-parameterized training module and distilled. In the second stage, the re-parameterized training module of the trained intermediate model is converted to restore the structure of the intermediate model to the original deep convolutional model. In the online stage, the optimized deep convolutional model is used for actual deployment.

2. The two-stage model reoptimization method for a deep convolutional model according to claim 1, characterized in that: include: Step 1: Re-parameterization training module replacement and distillation training, including: 1.1 Replace all 3×3 convolutions in the original depth convolution with re-parameterized training modules to obtain the intermediate model; 1.2 Use the knowledge distillation architecture to perform distillation training on the intermediate model, including: A) Use offline vector-level distillation on the overall distillation framework, and use the distillation loss to constrain the distance between the output feature vectors of the teacher model and the student model, and calculate the distillation loss L between the output vector of the student model and the output vector of the teacher model dis ; B) Using the adversarial learning method, the adversarial loss L of the output feature vectors of the teacher model and the student model is calculated through a discriminator consisting of three fully connected layers. AS ; C) Using the classifier reuse method, in the final stage of training, the classifier of the student model is directly replaced with the classifier of the teacher model and the parameters are fixed, so that the student model learns the knowledge contained in the teacher model classifier and calculates the loss function L between the student model output and the dataset label Arc ; The final output feature vector of the student model will establish a close constraint relationship with the output feature vector of the teacher model, so that the student model can better learn the knowledge of the teacher model; D) Calculate the total output function L total =L dis +λ·L AS +β·L Arc , where: λ and β are hyperparameters used to adjust the ratio of the loss function, and the intermediate model is trained after distillation training; Step 2: Convert the intermediate model back to the original model, including: 2.1 Convert the trained re-parameterized training module into a 3×3 convolution through model re-parameterization technology; 2.2 Fusion processing is performed on the mixed convolution branches in 2.1; 2.3 Combine steps 2.1 and 2.2 to convert the original re-parameterized training module into a single 3×3 convolution. Specifically, convert the output feature map addition operation of the multi-branch convolution into the addition operation of the multi-branch convolution kernel, convert the multi-branch structure into a single-branch structure, and fuse the four parallel 3×3 convolutions into a single 3×3 convolution.

3. The two-stage model reoptimization method for a deep convolutional model according to claim 2, characterized in that: The re-parameterized training module includes: a 3×3 convolution branch, a 1×1 convolution branch, a residual connection branch, a mixed convolution branch and a function activation layer, wherein: the mixed convolution branch is based on an inverted bottleneck structure, firstly adopts 1×1 convolution to increase the number of feature map channels and enrich the feature space, then adopts 3×3 convolution to extract spatial features on the feature map with amplified channel number, and finally reduces the number of channels again through 1×1 convolution, and the number of output feature map channels is the same as the number of input feature map channels; the residual connection branch is used to solve the gradient vanishing problem in the deep network, making the network easier to converge; the size of the output feature map of each branch is consistent with the size of the input feature map, and the results of the four branches are added and fused and then passed through the function activation layer.

4. The two-stage model reoptimization method for a deep convolutional model according to claim 3, characterized in that: A batch normalization layer (BN) is provided after the convolutional layer in the four branches of the re-parameterized training module to standardize the feature map, accelerate the model convergence speed and prevent the model from overfitting.

5. The two-stage model reoptimization method for a deep convolutional model according to claim 3 or 4, characterized in that: A batch normalization layer is further added to the residual link branch.

6. The two-stage model reoptimization method for a deep convolutional model according to claim 2, characterized in that: The model reparameterization technology refers to: fusing the parameters of the serialized 3×3 convolution and BN layer into a single 3×3 convolution; converting the 1×1 convolution and BN layer into a 3×3 convolution; filling a circle of parameters with a value of 0 outside the convolution kernel of the 1×1 convolution to convert it into a 3×3 convolution; rearranging the learnable scaling coefficients of the BN layer, that is, treating it as a 1×1 convolution layer, and then filling a circle of parameters with a value of 0 outside its own parameters. After the conversion, the three branches on the left side of the training module are transformed into a single 3×3 convolution, and the rightmost mixed convolution branch contains a series of 1×1 convolution, 3×3 convolution, and 1×1 convolution.

7. The two-stage model reoptimization method for a deep convolutional model according to claim 2, characterized in that: The fusion processing specifically includes: i) The sequence of 1×1 convolution and 3×3 convolution is fused into a 3×3 convolution, that is, the linear weighted operation on the feature map in the 1×1 convolution is transformed into a linear weighted operation on the input channel dimension of the 3×3 convolution kernel, so that the 1×1 convolution and 3×3 convolution series sequence can be fused into a 3×3 convolution; ii) Fuse the 3×3 convolution and 1×1 convolution sequence into a 3×3 convolution: Convert the linear combination of feature maps into a linear combination of the output channel dimensions of the 3×3 convolution kernel, and then fuse the sequence of 3×3 convolution and 1×1 convolution into a single 3×3 convolution. Finally, the sequence of 1×1 convolution, 3×3 convolution and 1×1 convolution can be converted into a 3×3 convolution.

8. The two-stage model reoptimization method for a deep convolutional model according to claim 2, characterized in that: The distillation loss Among them: F t represents the output vector of the teacher model, F s represents the output vector of the student model, and m represents the number of samples; The loss function between the student model output and the dataset label Among them: m represents the number of samples, s represents the modulus is a constant, and Θ represents the degree of the angle space.