Low-complexity audio scene classification method based on pruning distillation

Abstract

The invention discloses a low-complexity audio scene classification method based on pruning and distillation, and the method comprises the following steps: 1, constructing a Rep-Mobile model through a re-parameter technology, complementing the added convolution kernels of different shapes into an original 3 * 3 convolution kernel shape through a zero value, and carrying out the calculation of the original 3 * 3 convolution kernel shape; linear mapping in the BatchNorm layer is merged to offset values of convolution kernels through equivalent transformation, and different convolution kernels are linearly added and merged to a main branch; 2, a plurality of teacher models are fused with a distillation strategy, knowledge distillation is performed on the student model, and optimization of the student model is jointly guided through distillation loss and classification loss; and 3, compressing single-step pruning into multiple small steps by using an iterative pruning strategy, and reducing the interval of parameters. According to the method, the classification accuracy can be greatly improved under the condition that the parameter quantity and the calculation quantity are not increased; calculation resource saving is achieved, the model generalization ability is enhanced, the model size is remarkably reduced, and the calculation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to a low-complexity audio scene classification method based on pruning and distillation. Background Art

[0002] Audio scene classification (ASC) is a basic audio processing task that identifies and classifies audio signals into predefined environmental sound scenes, such as airports, parks, and city streets. Due to the limitations of application scenarios, ASC systems often face problems such as mismatches between recording devices, constraints caused by low model complexity, and scarcity of labeled data, especially when performing real-time processing and deployment on resource-constrained devices.

[0003] In order to build a low-complexity ASC system, many researchers have proposed many methods to reduce the amount of model computation, including using the powerful functions of convolutional neural networks (CNN) to develop ASC systems based on multi-layer stacked two-dimensional (2D) or one-dimensional (1D) convolution kernels. Although these works have made good progress in building low-complexity ASC models, most of these works only focus on architecture design, and rarely involve training strategies including distillation and pruning methods.

[0004] The audio self-supervised pre-training model is a method that uses a large amount of unlabeled audio data to learn speech features. The basic idea of ​​the audio self-supervised pre-training model is to define a self-supervised task and let the model predict or reconstruct certain parts based on the existing audio information. Common self-supervised tasks include audio reconstruction (let the model reconstruct the complete audio based on the input partial audio information), predict future audio frames (predict future audio frames based on past audio frames), mask prediction (mask some clips in the audio and let the model predict the masked part), etc. Then, the pre-trained model is used as a feature extractor or initialization parameter for downstream tasks for fine-tuning or end-to-end training. The audio self-supervised pre-training model has shown good performance on various downstream tasks. However, the self-supervised pre-training method has the problems of poor robustness and insufficient generalization ability.

[0005] Therefore, technicians in this field are committed to developing a low-complexity audio scene classification method based on pruning distillation. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problems to be solved by the present invention are: the amount of model calculation is too large, resulting in high application delay; the performance is not good when building a small model; the performance is seriously degraded after model pruning and compression, etc.

[0007] To achieve the above object, the present invention provides a low-complexity audio scene classification method based on pruning distillation, the classification method comprising the following steps:

[0008] Step 1: Use the re-parameterization technology to build the Rep-Mobile model, fill the added convolution kernels of different shapes with zero values ​​to form the original 3×3 convolution kernel shape, merge the linear mapping in the BatchNorm layer into the bias value of the convolution kernel through equivalent transformation, and linearly add and merge different convolution kernels into one main branch;

[0009] Step 2: Use multiple teacher models to fuse distillation strategies, perform knowledge distillation on the student model, and use distillation loss and classification loss to jointly guide the optimization of the student model;

[0010] Step 3: Use iterative pruning strategy to compress single-step pruning into multiple small steps and reduce the interval of parameters;

[0011] Furthermore, the Rep-Mobile model is built based on MobileNet, and the receptive field regularization technology is used to limit the range of the receptive field;

[0012] Furthermore, the convolution kernels of different shapes include convolution kernels of 1×1, 3×1 and 1×3 shapes;

[0013] Furthermore, the 1×1 convolution kernel is used to strengthen the mapping relationship;

[0014] Furthermore, the 3×1 convolution kernel extracts corresponding features from the time dimension;

[0015] Furthermore, the 1×3 convolution kernel extracts corresponding features from the feature dimension;

[0016] Furthermore, the multiple teacher models are multiple Transformer-based audio scene classification models, multiple models whose parameters are frozen;

[0017] Furthermore, after the multiple teacher models are fused, the student model is guided to perform knowledge distillation through distillation loss and classification loss;

[0018] Further, the distillation loss is the soft labels output by the multiple teacher models obtained by calculating the KLD divergence of the outputs of the multiple teacher models and the student model;

[0019] Furthermore, the classification loss is the cross entropy loss between the output of the student model and the true label.

[0020] The low-complexity audio scene classification method based on pruning and distillation of the present invention, by adding re-parameterization technology, constructs a Rep-Mobile model, which can greatly improve the classification accuracy without increasing the number of parameters and calculations; a low-complexity audio scene classification model is constructed by distillation and pruning technology, which realizes computing resource saving, enhanced model generalization ability, rapid deployment and response, high accuracy and recall rate, significantly reduced model size and improved computing efficiency.

[0021] When the classification method provided by the present invention is implemented in production, it has strong hardware compatibility, can reduce hardware costs, is easy to develop software and integrate with existing systems, and has broad prospects for industrial application. In the field of consumer electronics, it can be used in smart phones, smart wearable devices, etc. to improve user experience; in the field of smart homes, it can enable smart devices to better serve users and ensure family safety; in the field of smart security, it can assist in monitoring public places and border security; in the field of audio content management, it is helpful for multimedia database classification and audio copyright management, etc., and has huge commercial value and social significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the process framework of a low-complexity audio scene classification method based on pruning and distillation provided in an embodiment of the present invention;

[0023] Figure 2 It is a schematic diagram of a Rep-Mobile re-parameterization process of a low-complexity audio scene classification method based on pruning distillation provided in an embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of the multi-teacher model fusion knowledge distillation process of the low-complexity audio scene classification method based on pruning distillation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0026] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0027] The process overview of the low-complexity audio scene classification method based on pruning distillation of the present invention is as follows: Figure 1As shown in the figure, first, the Rep-Mobile model is constructed using the heavy parameter technology; second, the student model is distilled by fusing multiple teacher models with a distillation strategy; third, multiple small-step pruning is performed during pruning compression through an iterative pruning strategy.

[0028] More specifically, as shown in the figure, the Rep-Mobile reparameterization process is schematically shown. The overall framework of Rep-Mobile follows the construction of MobileNet, and the receptive field regularization technology is used in the design to limit the range of the receptive field. For each convolution module, there are only 3×3 convolution kernels originally. During training, multiple convolution kernels of different shapes are added to extract different representations, including 1×1 to strengthen the mapping relationship, and 3×1 and 1×3 extract corresponding features from the time dimension and feature dimension respectively. After the training is completed, through the reparameterization technology, the convolution kernels of different shapes can be padded into the shape of 3×3 convolution kernels through zero values, and then the linear mapping in the BatchNorm layer is merged into the bias value of the convolution kernel through equivalent transformation. Finally, the different convolution kernels can be merged into a main branch by linear addition, which will not introduce any computational complexity while improving the performance.

[0029] Evaluated on the DCASE2024 Task 1 dataset, as shown in Table 1, the above Rep-Mobile model is stronger than previous models such as BC-ResNet, CP-ResNet, CP-Mobile, and TF-SepNet with the same amount of computation and parameters, and its accuracy has been greatly improved.

[0030]

[0031] Table 1: Performance comparison of Rep-Mobile and other models

[0032] As an optional embodiment, different from the common method of using a single teacher model for knowledge distillation, the present invention proposes to train multiple teacher models and use the fusion of multiple teacher models to perform knowledge distillation on the student model. The whole process is as follows Figure 2 shown.

[0033] In the distillation process, first, multiple Transformer-based audio scene classification models are trained or fine-tuned on the training set, and the parameters are frozen to serve as the teacher model in the knowledge distillation. Subsequently, the KLD divergence between the student model output and the teacher model output is calculated, and the soft labels of the outputs of the multiple teacher models are used as guidance signals to guide the student model to perform gradient updates. At the same time, the cross entropy loss between the output of the student model and the true label is also calculated as another guidance signal. In general, the optimization of the student model is guided by the distillation loss and the classification loss.

[0034] The evaluation was conducted on the DCASE2024 Task 1 dataset. As shown in Table 1, knowledge distillation can effectively improve the performance of the student model, among which the performance of the Transformer-based model as the teacher model is most significantly improved after fusion.

[0035]

[0036] Table 2: Comparison of knowledge distillation results using teacher models with different structures

[0037] As an optional embodiment, in view of the situation that if the single-step compression ratio is too high during conventional pruning and compression, it will lead to serious loss of model performance, the present invention proposes an iterative pruning strategy, which compresses the single-step pruning into multiple small steps, effectively reducing the parameter interval.

[0038] The test was conducted on the DCASE2024 audio scene classification dataset. As shown in Table 3, the effectiveness of the iterative pruning strategy was verified. It can be seen that the iterative pruning strategy compresses the model from 96 to 32, and the performance is improved compared to single-step compression.

[0039]

[0040] Table 3: Performance comparison using iterative pruning strategies

[0041] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A low-complexity audio scene classification method based on pruning distillation, characterized in that: The classification method comprises the following steps: Step 1: Use the re-parameterization technology to build the Rep-Mobile model, fill the added convolution kernels of different shapes with zero values ​​to form the original 3×3 convolution kernel shape, merge the linear mapping in the BatchNorm layer into the bias value of the convolution kernel through equivalent transformation, and linearly add and merge different convolution kernels into one main branch; Step 2: Use multiple teacher models to fuse distillation strategies, perform knowledge distillation on the student model, and use distillation loss and classification loss to jointly guide the optimization of the student model; Step 3: Use an iterative pruning strategy to compress single-step pruning into multiple small steps and reduce the interval between parameters.

2. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 1, characterized in that: The Rep-Mobile model is built based on MobileNet, and the receptive field regularization technology is used to limit the range of the receptive field.

3. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 2, characterized in that: The convolution kernels of different shapes include convolution kernels of 1×1, 3×1 and 1×3 shapes.

4. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 3, characterized in that: The 1×1 convolution kernel is used to strengthen the mapping relationship.

5. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 3, characterized in that: The 3×1 convolution kernel extracts corresponding features from the time dimension.

6. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 3, characterized in that: The 1×3 convolution kernel extracts corresponding features from the feature dimension.

7. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 1, characterized in that: The multiple teacher models are multiple Transformer-based audio scene classification models, multiple models whose parameters are frozen.

8. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 7, characterized in that: After the multiple teacher models are fused, the student model is guided to perform knowledge distillation through distillation loss and classification loss.

9. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 8, characterized in that: The distillation loss is the soft labels output by the multiple teacher models obtained by calculating the KLD divergence of the outputs of the multiple teacher models and the student model.

10. The low-complexity audio scene classification method based on pruning distillation as claimed in claim 8, characterized in that: The classification loss is the cross entropy loss between the output of the student model and the true label.

Images