Audio classification method, model training method, system, equipment and medium

Abstract

The invention relates to an audio classification method, a model training method, a system, equipment and a medium. The training method comprises the following steps: acquiring an extended audio set and an audio tag set corresponding to the extended audio set; inputting the extended audio set into the trained original model, and extracting generated reference original category labels and original features; inputting the extended audio set into an extended model, and extracting a generated original category label and / or a newly added category label and an extended feature generated by the same network layer as the original model; and updating parameters of an expansion model based on the difference degree between the original category label and the reference original category label, the difference degree between the expansion feature and the original feature, and the difference degree between the original category label, the newly added category label and the corresponding audio label in the audio label set, and taking the trained expansion model as an audio classification model. According to the method, the problem of knowledge forgetting possibly occurring when the extension model learns a new category is solved, and meanwhile, the learning ability of the extension model to a new category is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal processing, and in particular to an audio classification method, a model training method, a system, a device and a medium. Background Art

[0002] Multi-label audio classification refers to predicting the category to which the audio belongs (such as human voice, dog barking, running water, etc.) given an audio signal. With the development of deep learning technology, multi-label audio classification tasks have been widely used. However, there are many categories of audio, and the same audio may belong to multiple categories. After training and deploying a multi-label audio classification model using data of limited categories, it may be necessary to introduce new audio categories through model fine-tuning and other methods in subsequent applications. However, after adding new audio categories, the model may forget the knowledge of the previously learned categories, and the classification accuracy of the model with the new categories for the old categories of audio is seriously reduced. This is because the introduction of new categories will affect the parameters of the trained model. This phenomenon will be more obvious when the difference between the introduced category and the original category data volume is large. Therefore, it is necessary to provide an audio classification method, a model training method, a system, a device and a medium. Summary of the invention

[0003] In view of the shortcomings of the prior art described above, an object of the present invention is to provide an audio classification method, a model training method, a system, a device and a medium, which improve the problem of knowledge forgetting that is prone to occur when performing incremental learning of audio categories in the prior art.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a training method for an audio classification model, comprising: obtaining an extended audio set and a corresponding audio tag set; inputting the extended audio set into a trained original model, extracting the generated reference original category labels and original features; inputting the extended audio set into an extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same corresponding structure as the original model; based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio tag set, updating the parameters of the extended model, and using the trained extended model as the audio classification model.

[0005] In one embodiment of the present invention, the step of inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and the extended features generated by the same network layer as the original model, includes: preprocessing the extended audio set to generate an extended audio feature set; inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution; and inputting the extended features into the classifier of the extended model to generate the original category labels and / or newly added category labels corresponding to the extended features.

[0006] In one embodiment of the present invention, the extended audio set is preprocessed to generate an extended audio feature set, including: performing fast Fourier transform on each audio in the extended audio set to generate a corresponding spectrogram; performing Mel filter group processing on each spectrogram to generate a corresponding Mel spectrogram; performing logarithmic compression on each Mel spectrogram; performing discrete cosine transform on each compressed Mel spectrogram to obtain the extended audio feature set constructed by the transformed Mel spectrograms.

[0007] In one embodiment of the present invention, the extended audio feature set is input into the feature extraction network of the extended model, and based on multi-scale depthwise separable convolution, the extended features of the extended audio feature set are generated and extracted, including: based on the depthwise separable convolution of the first scale and the depthwise separable convolution of the second scale, the first feature and the second feature of the extended audio feature set are correspondingly extracted; wherein the first feature and the second feature correspond to the depthwise separable convolution of the corresponding scales respectively; the first feature and the second feature are feature fused to obtain a comprehensive feature; the comprehensive feature is batch normalized to generate a normalized feature; the normalized feature is nonlinearly activated to generate and extract an extended feature.

[0008] In one embodiment of the present invention, the step of inputting the extended features into the classifier of the extended model to generate original category labels and / or newly added category labels corresponding to the extended features includes: performing feature mapping on the extended features to obtain confidence levels of various audio categories; and selecting categories with confidence levels greater than a preset threshold as original category labels and / or newly added category labels.

[0009] In one embodiment of the present invention, the parameters of the extended model are updated based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and the trained extended model is used as the audio classification model, including: calculating the output difference loss between the original category label and the reference original category label; calculating the feature difference loss between the extended feature and the original feature; calculating the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set; weighted summing the output difference loss, the feature difference loss and the autonomous learning cross entropy loss to obtain a comprehensive loss; updating the parameters of the extended model based on the comprehensive loss, and using the trained extended model as the audio classification model.

[0010] In one embodiment of the present invention, the calculation of the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set includes: calculating the original category cross entropy loss between the original category label and the corresponding audio label in the audio label set; calculating the newly added category cross entropy loss between the newly added category label and the corresponding audio label in the audio label set; and weighted summing the original category cross entropy loss and the newly added category cross entropy loss to obtain the autonomous learning cross entropy loss.

[0011] In one embodiment of the present invention, an audio classification method is also provided, which includes: obtaining audio to be classified; inputting the audio into any of the audio classification models described above, extracting extended features of the audio, and generating original category labels and / or newly added category labels of the audio based on the extended features.

[0012] In one embodiment of the present invention, a training system for a multi-label audio classification model is also provided, the system comprising: an audio acquisition module, used to acquire an extended audio set and a corresponding audio label set; an original label acquisition module, used to input the extended audio set into the trained original model, and extract the generated reference original category label and original feature; an extended label acquisition module, used to input the extended audio set into the extended model, and extract the generated original category label and / or newly added category label and the extended feature generated by the same network layer as the original model; wherein the feature extraction network of the extended model is the same as the corresponding structure of the original model; a model training module, used to update the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and use the trained extended model as the audio classification model.

[0013] In one embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements any of the audio classification model training methods or audio classification methods described above.

[0014] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the audio classification model training methods or audio classification methods described above.

[0015] As described above, the audio classification method, model training method, system, device and medium of the present invention have the following beneficial effects: by utilizing the consistency of the extended model and the original model in the feature extraction network structure, it is ensured that the extended model can maintain the original generalization ability while learning new categories. By using the incremental learning method, only the data of the newly added categories are used for incremental training of the extended model, which effectively improves the knowledge forgetting problem that may occur when the extended model learns new categories, and at the same time enhances its learning ability for the newly added categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic flow chart of a method for training an audio classification model provided by an embodiment of the present invention;

[0017] Figure 2 The overall flow chart of the training method of the audio classification model of the present invention;

[0018] Figure 3Shown is a structural block diagram of a training system for an audio classification model provided by an embodiment of the present invention;

[0019] Figure 4 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0023] The inventors found that after introducing new audio categories, the multi-label audio classification model is prone to the problem of knowledge forgetting. The audio classification described in the present invention refers to the classification of speech categories, and multi-label refers to one data corresponding to multiple different labels. The prior art generally solves this problem through a feature extraction method. The scheme mainly includes: first, using the audio data of the specified category to train a model including a feature extraction network and a classifier. After the model training is completed, the new audio category data to be introduced is obtained, the feature extraction network parameters of the previously trained model are fixed, and the model is trained using the new audio category output. After the training is completed, the classifier of the model can classify the new audio data. This scheme can retain the feature representation of the old category and reduce the problem of knowledge forgetting caused by the introduction of new category data. However, when introducing new category data in this way, since the feature extractor is frozen after the initial stage, the model cannot update the parameters of the feature extractor to adapt to the new category data. This may lead to insufficient feature representation of the new category, affecting the classification performance of the new category. Freezing the feature extractor means that the feature representation ability of the model is fixed and cannot be adjusted and optimized according to the new data. This may lead to insufficient feature representation when facing diverse and complex audio data. The feature representation of old categories may conflict with the feature representation of new categories. The frozen feature extractor cannot resolve this conflict, which may cause the classifier to lose its ability to distinguish between old and new categories. This situation will seriously affect the application of multi-label audio classification models on devices such as headphones and mobile phones.

[0024] In view of the above problems, the present invention provides a training method for an audio classification model. Through a class incremental learning scheme, when a new classification category needs to be added, only the new category data can be used to train the extended model, which can avoid the problem of forgetting the original category knowledge caused by learning new categories, and can also prevent the new extended model from being overly biased towards the new category due to the imbalance of new and old category data. Among them, class incremental learning is a learning method for gradually learning new categories. This method can quickly complete the new category learning and deployment of the existing original model, avoid the redesign and training of the new extended model, and enable the extended model after the original model is extended to efficiently learn new categories. The design of the extended model of the present invention is based on multi-scale deep separable convolution, which greatly reduces the number of network parameters and the faster calculation speed. This design not only enables the network to be easily accelerated using arm operators during deployment to achieve low-power real-time processing, but also significantly reduces the demand for deployment equipment computing power and memory, providing excellent conditions for efficient deployment in resource-constrained environments. This extended model has low computational complexity and small number of parameters, and can be deployed on resource-constrained edge devices such as mobile phones and headphones.

[0025] See also Figure 1 ,The training method of the audio classification model includes the following steps:

[0026] S1. Obtain an extended audio set and a corresponding audio tag set.

[0027] The extended audio set is a part of audio samples that are newly added to the original audio set, where the original audio set is used to train the original model, and these samples only cover several audio categories that were initially predefined. After the original model is trained, when it is necessary to recognize audio of a new category, the audio samples containing the newly added category are used as the extended audio set. Among them, the extended audio set only includes audio samples of categories that are newly added relative to the original audio set, and the extended audio set is composed of a mixture of single-label audio and multi-label audio. Among them, single-label audio represents audio samples that only belong to the newly added category, and multi-label audio represents audio samples that belong to both the newly added category and the original category. For each audio sample in the extended audio set, a corresponding label is generated according to its category to form an audio label set, which is used to guide the training of the extended model. Exemplarily, the original audio set includes three audio categories: "human voice", "dog barking" and "cat meowing". In addition to these three categories, the extended audio set also includes two new categories: "chicken crowing" and "background sound". For each single-label audio in the extended audio set, its label is only one of the new categories (such as "chicken crowing" or "background sound"). For each multi-label audio in the extended audio set, its label will be marked with at least one original category and at least one new category (such as labels such as "human voice", "background sound" and "chicken crowing").

[0028] S2. Input the extended audio set into the trained original model, and extract the generated reference original category labels and original features.

[0029] See also Figure 2, input the extended audio set into the trained original model, wherein the original model includes a feature extraction network and a classifier, the feature extraction network is used to extract high-level feature representations from the input audio data, and the classifier is used to classify based on the extracted features and assign the audio samples to corresponding categories (such as "human voice", "dog barking", "background sound", etc.). Since the original model is trained using the original audio set (such as "human voice", "dog barking", etc.), it can identify the relevant original categories in the extended audio set and has the audio classification function, but does not have the recognition ability for the newly added categories (such as "chicken crowing" or "background sound"). Input the extended audio set into the trained original model, and for each input extended audio sample, the original model generates at least one reference original category label. Among them, the reference original category label refers to the category label output by the original model after predicting the input audio sample based on the original category it has learned (such as "human voice", "dog barking", etc.), which represents the classification result of the original model on the extended audio sample for the original category. At the same time, the output features of a certain layer are extracted from the original model as the original features. Considering that the features output by the feature extraction network are the best feature representations of the input data, preferably, the output of the previous layer of the original model classifier (i.e., the feature extraction network) is used as the original feature. This original feature is used as an intermediate representation of the audio sample, indicating the feature representation capability of the original model on the input data.

[0030] S3. Input the extended audio set into the extended model, extract the generated original category labels and / or newly added category labels and the extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same corresponding structure as the original model.

[0031] The structure of the extended model is the same as that of the original model, and also includes a feature extraction network and a classifier. Among them, the feature extraction network of the extended model has the same structure as that of the original model, and uses the trained feature extraction network parameters of the original model through direct migration. The classifier of the extended model is generated based on the newly added categories and the original categories in the extended audio set, so that audio samples of the original categories and the newly added categories can be recognized and classified at the same time. Specifically, the output unit of the classifier of the extended model adds output units consistent with the number of audio categories that need to be newly learned on the basis of the classifier of the original model.

[0032] In one embodiment of the present invention, the step of inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and the extended features generated by the same network layer as the original model, includes:

[0033] Preprocessing the extended audio set to generate an extended audio feature set;

[0034] Inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution;

[0035] The extended features are input into the classifier of the extended model to generate original category labels and / or newly added category labels corresponding to the extended features.

[0036] Each extended audio sample in the extended audio set is preprocessed to obtain an extended audio feature set. The extended audio feature set is input into the feature extraction network of the extended model, and based on the multi-scale deeply separable convolution operation, the multi-level feature information of each extended audio sample is captured, thereby generating and extracting extended features. The extracted extended features are input into the classifier of the extended model. Since the classifier includes several cascaded fully connected layers, the fully connected layer performs a linear transformation on the input extended features and calculates the linear combination of the outputs. These outputs are input into the activation layer, and the features output by the fully connected layer are nonlinearly transformed through a nonlinear activation function. According to the probability distribution of the output, the original category label and / or the newly added category label of each sample are generated. It should be noted that since the feature extraction networks of the original model and the extended model are both composed of multi-scale deeply separable convolution blocks, both the original model and the extended model are lightweight models.

[0037] In one embodiment of the present invention, preprocessing the extended audio set to generate an extended audio feature set includes:

[0038] Performing fast Fourier transform on each audio in the extended audio set to generate a corresponding spectrum graph;

[0039] Perform Mel filter bank processing on each spectrum graph to generate the corresponding Mel spectrum graph;

[0040] Logarithmically compress each Mel-spectrogram;

[0041] The compressed Mel-spectrograms are respectively subjected to discrete cosine transformation to obtain the extended audio feature set constructed by the transformed Mel-spectrograms.

[0042] In order to convert the original audio data into a feature representation that is more suitable for deep learning model processing, a series of preprocessing operations need to be performed on the extended audio set before the extended audio set is input into the extended model / original model. Specifically, each audio sample in the extended audio set is first subjected to a fast Fourier transform, so as to transform the audio sample from a time domain signal to a frequency domain signal and generate a corresponding spectrogram. For the spectrogram of each audio sample: a Mel filter group is applied to process it and a corresponding Mel spectrogram is generated. The Mel spectrogram of each audio sample is logarithmically compressed to obtain a logarithmic Mel spectrogram of each audio sample to reduce the dynamic range. The logarithmic Mel spectrogram of each audio sample is subjected to a discrete cosine transform to obtain a Mel-frequency cepstral coefficient feature vector of each audio sample. The Mel-frequency cepstral coefficient feature vectors of all audio samples are combined to form an extended audio feature set.

[0043] In one embodiment of the present invention, the step of inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution, comprises:

[0044] Based on the depthwise separable convolution of the first scale and the depthwise separable convolution of the second scale, extracting the first feature and the second feature of the extended audio feature set accordingly; wherein the first feature and the second feature correspond to the depthwise separable convolution of the corresponding scales respectively;

[0045] Fusing the first feature and the second feature to obtain a comprehensive feature;

[0046] Performing batch normalization processing on the comprehensive features to generate normalized features;

[0047] The normalized features are subjected to nonlinear activation processing to generate and extract extended features.

[0048] The multi-scale depth-separable convolution block in the present invention includes two depth-separable convolutions with different feature scales, a batch normalization layer and a nonlinear activation function. Specifically, two depth-separable convolutions of different scales (respectively, the first scale and the second scale) are used to extract corresponding features from the extended audio feature set, respectively, to obtain the first feature and the second feature. Specifically, the extended audio feature set is convolved by the depth-separable convolution kernel of the first scale. The depth convolution of the first scale is used to perform convolution processing on each input channel separately to extract the local features of the channel. The output of the depth convolution is convolved again using point-by-point convolution, and the integrated first feature is generated by integrating the features of each channel. In addition, the depth-separable convolution of the second scale is used to perform convolution operation on the extended audio to generate the second feature. Among them, the first scale is smaller than the second scale. The extracted first feature and the second feature are feature fused, and the feature information of different scales can be combined to obtain a more comprehensive and detailed comprehensive feature. The comprehensive features are batch normalized to improve the convergence speed and effect of the model training team. The normalized features are nonlinearly activated by the nonlinear activation function to generate and extract the extended features.

[0049] The present invention realizes real-time multi-label audio classification by uniquely designing a lightweight neural network (i.e., an extended model). The lightweight neural network is based on deep separable convolution, and uses deep separable convolution of different scales to capture multi-scale frequency domain and time domain information of audio. Deep separable convolution includes two core steps: deep convolution and point-by-point convolution. In deep convolution, each input channel is convolved with an independent convolution kernel to achieve parameter sharing between channels. This strategy reduces the number of convolution kernels and significantly reduces the amount of parameters, successfully extracting important features of the input audio data at a lower computational cost. At the same time, multi-scale convolution can extract multi-scale information of audio features, and the obtained feature information is richer. Compared with traditional convolution, this innovative method not only reduces the number and complexity of parameters while maintaining model performance, but also realizes real-time audio classification, providing an efficient and feasible solution for the practical application of multi-label audio classification.

[0050] In one embodiment of the present invention, the step of inputting the extended feature into the classifier of the extended model to generate the original category label and / or the newly added category label corresponding to the extended feature includes:

[0051] Performing feature mapping on the extended features to obtain confidence of each audio category;

[0052] The categories with confidence greater than a preset threshold are selected as the original category labels and / or new category labels.

[0053] The extended features are input into the classifier of the extended model. Through each fully connected layer in the classifier, the extended features are continuously linearly transformed, and the output of the last fully connected layer is used as the output vector. Among them, each element in the output vector is used to represent the confidence score of the audio sample belonging to each category. And the output vector after linear transformation is mapped to a preset range through the activation function of the extended model to obtain the confidence of each element. Each element in the generated confidence vector is screened, and the category with a confidence greater than the preset threshold is selected as the final classification result.

[0054] It should be noted that since the feature extraction network of the original model has the same structure as that of the extended model, and the classifiers are composed of the same number of fully connected layers, when the original model is trained using the original audio set, the training process is the same as that of the above-mentioned extended model, including the same preprocessing operations on the original audio set as well as the feature extraction and classification processes, which will not be described in detail here.

[0055] S4. Based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, update the parameters of the extended model and use the trained extended model as the audio classification model.

[0056] By calculating the difference between the original category labels generated by the extended model and the reference original category labels, the difference between the extended features and the original features, and the difference between the original category labels, the newly added category labels and the real audio labels, a comprehensive loss is formed. This comprehensive loss is used to optimize the parameters of the extended model, and finally a multi-label audio classification model is trained that can maintain the ability to recognize the original categories and effectively learn the newly added categories.

[0057] In one embodiment of the present invention, the updating of the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and using the trained extended model as the audio classification model includes:

[0058] Calculating the output difference loss between the original category label and the reference original category label;

[0059] Calculating the feature difference loss between the extended feature and the original feature;

[0060] Calculating the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set;

[0061] Performing a weighted summation on the output difference loss, the feature difference loss and the autonomous learning cross entropy loss to obtain a comprehensive loss;

[0062] The parameters of the extended model are updated based on the comprehensive loss, and the trained extended model is used as an audio classification model.

[0063] Calculate the output difference loss between the original category label generated by the extended model (i.e., the prediction result of the original category by the extended model) and the reference original category label (i.e., the prediction result of the original category generated by the original model). The type of output difference loss includes but is not limited to cross entropy loss or KL (Kullback-Leibler) divergence, preferably, the output difference loss is KL divergence. Calculate the difference loss between the extended features of the extended model (features output by the extended model feature extraction network) and the original features of the original model (features output by the original model feature extraction network). The difference loss can be calculated by cosine similarity loss or mean square error loss, etc., preferably, the difference loss is cosine similarity loss. Calculate the autonomous learning cross entropy loss between the original category label generated by the extended model and the newly added category label and the corresponding real label in the audio label set. The above three losses are weighted and summed to obtain a comprehensive loss, and the parameters of the extended model are updated according to the comprehensive loss through the back propagation algorithm. Continuously iterate the training to finally obtain a trained extended model, and use it as the final multi-label audio classification model, which has the ability to recognize both the original category and the newly added category. In the present invention, the original features extracted by the original model parameters and the output results are used to calculate the cosine similarity loss and Kullback-Leibler divergence loss with the extended model to solve the problem of knowledge forgetting caused by the extended model in the process of learning new categories. It can effectively help the trained model learn new classification categories, avoid retraining of old data, greatly save the training time of the extended model, and improve the ability of the extended model to learn new classification categories. In addition, the present invention adopts an autonomous learning cross entropy loss to independently calculate the binary cross entropy loss of the original category and the newly added category. This loss can avoid the problem that after the introduction of the newly added category, the category prediction of the extended model is more inclined to the newly added category due to data imbalance.

[0064] In one embodiment of the present invention, the calculating of the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set includes:

[0065] Calculating an original category cross entropy loss between the original category label and the corresponding audio label in the audio label set;

[0066] Calculating a newly added category cross entropy loss between the newly added category label and the corresponding audio label in the audio label set;

[0067] The original category cross entropy loss and the newly added category cross entropy loss are weighted and summed to obtain the autonomous learning cross entropy loss.

[0068] Calculate the error between the prediction result of the extended model for the original category and the real original category label in the audio label set, that is, the original category cross entropy loss, to ensure that the model can still maintain the correct recognition ability of the original category during the learning process of the new category. Calculate the error between the prediction result of the extended model for the new category and the real new category label in the audio label set, that is, the new category cross entropy loss, so that the model can accurately learn and recognize the characteristics of the new category. The two losses are weighted and summed to obtain the autonomous learning cross entropy loss.

[0069] It should be noted that the extended model designed by the present invention is a lightweight model consisting of a feature extraction network and a classifier. The feature extraction network consists of a multi-scale depth-separable convolution block, which includes two depth-separable convolutions with different feature scales, a batch normalization layer, and a nonlinear activation function, and the classifier consists of a fully connected layer. The extended model has a small number of parameters and low computational complexity, and can be deployed on low-computing power devices with limited resources.

[0070] In one embodiment of the present invention, an audio classification method is also provided, the method comprising:

[0071] Get the audio to be classified;

[0072] The audio is input into any of the audio classification models described above, extended features of the audio are extracted, and original category labels and / or newly added category labels of the audio are generated based on the extended features.

[0073] After obtaining the audio to be classified, input it into the multi-label audio classification model described in any of the above items, and extract extended features from the audio through the feature extraction network. These features contain the key information of the audio signal. The extended features are input into the classifier of the audio classification model, and the audio sample is analyzed based on the extracted extended features to generate the corresponding classification label. Among them, the classification label is the original category label (such as "human voice", "dog barking") and / or the newly added category label (such as "background sound") to which the audio belongs, thereby completing the multi-category classification of the audio sample.

[0074] See also Figure 3, the training system 100 of the multi-label audio classification model includes: an audio acquisition module 110, an original label acquisition module 120, an extended label acquisition module 130 and a model training module 140. The above-mentioned audio acquisition module 110 is used to obtain an extended audio set and a corresponding audio label set. The original label acquisition module 120 is used to input the extended audio set into the trained original model, and extract the generated reference original category label and original feature. The extended label acquisition module 130 is used to input the extended audio set into the extended model, extract the generated original category label and / or the newly added category label and the extended feature generated by the same network layer as the original model; wherein the feature extraction network of the extended model is the same as the corresponding structure of the original model. The model training module 140 is used to update the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and use the trained extended model as the audio classification model.

[0075] For the specific limitations on the training system of the multi-label audio classification model, please refer to the limitations on the training method of the multi-label audio classification model mentioned above, which will not be repeated here. Each module in the above-mentioned training system of the multi-label audio classification model can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory of the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0076] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0077] See also Figure 4 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a training program for an audio classification model.

[0078] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for training the audio classification model, etc., but also can be used to temporarily store data that has been output or is to be output.

[0079] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules (such as training programs for audio classification models, etc.) stored in the memory 12, and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0080] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the training method of the audio classification model.

[0081] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an audio acquisition module 110, an original label acquisition module 120, an extended label acquisition module 130, and a model training module 140.

[0082] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the training method of the audio classification model described in each embodiment of the present application.

[0083] In summary, the present invention discloses a training method, system, device and medium for an audio classification model, and the incremental learning scheme adopted can efficiently help the existing audio classification model learn the classification function of audio data of new categories. It only needs to use the newly added category data to train the extended model on the basis of the original model to obtain the extended model that can identify the newly added category and the original category. The present invention uses the intermediate features and output results of the original model and the extended model to calculate the feature difference loss and the result difference loss, which solves the problem of knowledge forgetting caused by the extended model learning the newly added category. At the same time, the output result of the extended model calculates the autonomous learning cross entropy loss, which overcomes the problem of the extended model predicting the newly added category caused by the imbalance between the newly added category data and the original category data. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0084] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A training method for an audio classification model, characterized in that: The method comprises: Get the extended audio set and the corresponding audio tag set; Inputting the extended audio set into the trained original model, extracting the generated reference original category labels and original features; Inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same structure as the corresponding structure of the original model; Based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, the parameters of the extended model are updated, and the trained extended model is used as the audio classification model.

2. The method for training an audio classification model according to claim 1, characterized in that: The step of inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and the extended features generated by the same network layer as the original model, comprises: Preprocessing the extended audio set to generate an extended audio feature set; Inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution; The extended features are input into the classifier of the extended model to generate original category labels and / or newly added category labels corresponding to the extended features.

3. The method for training an audio classification model according to claim 2, characterized in that: The step of preprocessing the extended audio set to generate an extended audio feature set includes: Performing fast Fourier transform on each audio in the extended audio set to generate a corresponding spectrum graph; Perform Mel filter bank processing on each spectrum graph to generate the corresponding Mel spectrum graph; Logarithmically compress each Mel-spectrogram; The compressed Mel-spectrograms are respectively subjected to discrete cosine transformation to obtain the extended audio feature set constructed by the transformed Mel-spectrograms.

4. The method for training an audio classification model according to claim 2, characterized in that: The step of inputting the extended audio feature set into the feature extraction network of the extended model, and generating and extracting extended features of the extended audio feature set based on multi-scale depth-separable convolution, comprises: Based on the depthwise separable convolution of the first scale and the depthwise separable convolution of the second scale, extracting the first feature and the second feature of the extended audio feature set accordingly; wherein the first feature and the second feature correspond to the depthwise separable convolution of the corresponding scales respectively; Fusing the first feature and the second feature to obtain a comprehensive feature; Performing batch normalization processing on the comprehensive features to generate normalized features; The normalized features are subjected to nonlinear activation processing to generate and extract extended features.

5. The method for training an audio classification model according to claim 2, characterized in that: The step of inputting the extended feature into the classifier of the extended model to generate the original category label and / or the newly added category label corresponding to the extended feature includes: Performing feature mapping on the extended features to obtain confidence of each audio category; The categories with confidence greater than a preset threshold are selected as the original category labels and / or new category labels.

6. The method for training an audio classification model according to claim 1, characterized in that: The updating of the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and using the trained extended model as the audio classification model includes: Calculating the output difference loss between the original category label and the reference original category label; Calculating the feature difference loss between the extended feature and the original feature; Calculating the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set; Performing a weighted summation on the output difference loss, the feature difference loss and the autonomous learning cross entropy loss to obtain a comprehensive loss; The parameters of the extended model are updated based on the comprehensive loss, and the trained extended model is used as an audio classification model.

7. The method for training an audio classification model according to claim 6, characterized in that: The calculating the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set includes: Calculating an original category cross entropy loss between the original category label and the corresponding audio label in the audio label set; Calculating a newly added category cross entropy loss between the newly added category label and the corresponding audio label in the audio label set; The original category cross entropy loss and the newly added category cross entropy loss are weighted and summed to obtain the autonomous learning cross entropy loss.

8. An audio classification method, characterized in that: The method comprises: Get the audio to be classified; The audio is input into the audio classification model according to any one of claims 1 to 7, extended features of the audio are extracted, and original category labels and / or newly added category labels of the audio are generated based on the extended features.

9. A training system for an audio classification model, characterized in that: The system comprises: An audio acquisition module, used to acquire an extended audio set and a corresponding audio tag set; An original label acquisition module, used to input the extended audio set into the trained original model, and extract the generated reference original category labels and original features; An extended label acquisition module, used to input the extended audio set into the extended model, extract the generated original category labels and / or newly added category labels and the extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same structure as the corresponding structure of the original model; A model training module is used to update the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and use the trained extended model as the audio classification model.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the training method of the audio classification model as described in any one of claims 1 to 7 or the audio classification method as described in claim 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the training method of the audio classification model described in any one of claims 1 to 7 or the audio classification method described in claim 8.

Images