Audio classification method, model training method, system, device and medium

Abstract

The present invention relates to an audio classification method, a model training method, a system, a device and a medium. The training method comprises: obtaining an extended audio set and a corresponding audio tag set; inputting the extended audio set into a trained original model, extracting the generated reference original category labels and original features; inputting the extended audio set into an extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model; based on the difference between the original category labels and the reference original category labels, the difference between the extended features and the original features, and the difference between the original category labels, the newly added category labels and the corresponding audio tags in the audio tag set, updating the parameters of the extended model, and using the trained extended model as an audio classification model. The present invention improves the problem of knowledge forgetting that may occur when the extended model learns new categories, while enhancing its learning ability for newly added categories.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal processing, and in particular to an audio classification method, a model training method, a system, a device and a medium. Background Art

[0002] Multi-label audio classification refers to predicting the category of audio (e.g., human voice, dog barking, running water, etc.) given an audio signal. With the development of deep learning technology, multi-label audio classification tasks have gained widespread application. However, there are numerous audio categories, and the same audio segment may belong to multiple categories. After training and deploying a multi-label audio classification model using data from limited categories, subsequent applications may require introducing new audio categories through methods such as model fine-tuning. However, after adding new audio categories, the model may forget the knowledge of previously learned categories, and the classification accuracy of the model with the new categories will be severely reduced for audio from old categories. This is because the introduction of new categories affects the parameters of the already trained model. This phenomenon is more pronounced when the amount of data for the introduced categories differs significantly from that of the original categories. Therefore, it is necessary to provide an audio classification method, model training method, system, device, and medium. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide an audio classification method, model training method, system, device and medium, which improve the problem of knowledge forgetting that is prone to occur when performing incremental learning of audio categories in the prior art.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a training method for an audio classification model, comprising: obtaining an extended audio set and a corresponding audio tag set; inputting the extended audio set into a trained original model, extracting the generated reference original category labels and original features; inputting the extended audio set into an extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same corresponding structure as the original model; based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio tag set, updating the parameters of the extended model, and using the trained extended model as the audio classification model.

[0005] In one embodiment of the present invention, the step of inputting the extended audio set into the extended model and extracting the generated original category labels and / or newly added category labels as well as the extended features generated by the same network layer as the original model includes: preprocessing the extended audio set to generate an extended audio feature set; inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution; and inputting the extended features into the classifier of the extended model to generate the original category labels and / or newly added category labels corresponding to the extended features.

[0006] In one embodiment of the present invention, the extended audio set is preprocessed to generate an extended audio feature set, including: performing fast Fourier transform on each audio in the extended audio set to generate a corresponding spectrogram; performing Mel filter bank processing on each spectrogram to generate a corresponding Mel spectrogram; performing logarithmic compression on each Mel spectrogram; and performing discrete cosine transform on each compressed Mel spectrogram to obtain the extended audio feature set constructed by the transformed Mel spectrograms.

[0007] In one embodiment of the present invention, the extended audio feature set is input into the feature extraction network of the extended model, and the extended features of the extended audio feature set are generated and extracted based on multi-scale depthwise separable convolution, including: extracting the first feature and the second feature of the extended audio feature set based on the depthwise separable convolution of the first scale and the depthwise separable convolution of the second scale; wherein the first feature and the second feature correspond to the depthwise separable convolution of the corresponding scales respectively; performing feature fusion on the first feature and the second feature to obtain a comprehensive feature; performing batch normalization processing on the comprehensive feature to generate a normalized feature; performing nonlinear activation processing on the normalized feature to generate and extract the extended feature.

[0008] In one embodiment of the present invention, the extended features are input into the classifier of the extended model to generate the original category labels and / or newly added category labels corresponding to the extended features, including: feature mapping the extended features to obtain the confidence of each audio category; and selecting the category with a confidence greater than a preset threshold as the original category label and / or newly added category label.

[0009] In one embodiment of the present invention, the parameters of the extended model are updated based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and the trained extended model is used as the audio classification model, including: calculating the output difference loss between the original category label and the reference original category label; calculating the feature difference loss between the extended feature and the original feature; calculating the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set; weightedly summing the output difference loss, the feature difference loss and the autonomous learning cross entropy loss to obtain a comprehensive loss; updating the parameters of the extended model based on the comprehensive loss, and using the trained extended model as the audio classification model.

[0010] In one embodiment of the present invention, the calculating of the autonomous learning cross entropy loss between the original category label, the newly added category label and the corresponding audio label in the audio label set includes: calculating the original category cross entropy loss between the original category label and the corresponding audio label in the audio label set; calculating the newly added category cross entropy loss between the newly added category label and the corresponding audio label in the audio label set; and weighted summing the original category cross entropy loss and the newly added category cross entropy loss to obtain the autonomous learning cross entropy loss.

[0011] In one embodiment of the present invention, an audio classification method is also provided, which includes: obtaining audio to be classified; inputting the audio into any of the audio classification models described above, extracting extended features of the audio, and generating the original category label and / or new category label of the audio based on the extended features.

[0012] In one embodiment of the present invention, a training system for a multi-label audio classification model is also provided, the system comprising: an audio acquisition module for acquiring an extended audio set and a corresponding audio label set; an original label acquisition module for inputting the extended audio set into the trained original model, extracting the generated reference original category labels and original features; an extended label acquisition module for inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model is the same as the corresponding structure of the original model; a model training module for updating the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and using the trained extended model as the audio classification model.

[0013] In one embodiment of the present invention, an electronic device is also provided, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements any one of the above-mentioned audio classification model training methods or audio classification methods.

[0014] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a computer processor, the computer executes any of the audio classification model training methods or audio classification methods described above.

[0015] As described above, the audio classification method, model training method, system, device, and medium of the present invention have the following beneficial effects: By leveraging the consistency of the feature extraction network structure between the extended model and the original model, the extended model maintains its original generalization capabilities while learning new categories. Furthermore, by using an incremental learning approach, the extended model is incrementally trained using only data from newly added categories, effectively alleviating the knowledge forgetting problem that may occur when learning new categories, while enhancing its ability to learn new categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a method for training an audio classification model provided by an embodiment of the present invention;

[0017] Figure 2 This is an overall flow chart of the training method of the audio classification model of the present invention;

[0018] Figure 3Shown is a structural block diagram of a training system for an audio classification model provided by an embodiment of the present invention;

[0019] Figure 4 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0023] The inventors discovered that multi-label audio classification models are prone to knowledge forgetting after introducing new audio categories. Audio classification, as described in the present invention, refers to the classification of speech into categories, and multi-label means that one piece of data corresponds to multiple different labels. Existing technologies generally address this problem through feature extraction methods. This approach primarily involves training a model containing a feature extraction network and a classifier using audio data of a specified category. After model training is complete, data of the new audio category to be introduced is obtained. The feature extraction network parameters of the previously trained model are fixed, and the model is trained using the output of the new audio category. After training, the model's classifier is able to classify the new audio data. This approach can retain feature representations of old categories, reducing the knowledge forgetting problem caused by the introduction of new category data. However, when introducing new category data with this approach, because the feature extractor is frozen after the initial stage, the model cannot update its parameters to adapt to the new category data. This can result in insufficient feature representation of the new category, affecting the classification performance of the new category. Freezing the feature extractor means that the model's feature representation capabilities are fixed and cannot be adjusted or optimized based on new data. This can lead to insufficient feature representation when dealing with diverse and complex audio data. The feature representations of old categories may conflict with those of new categories, and freezing the feature extractor cannot resolve this conflict, potentially reducing the classifier's ability to distinguish between old and new categories. This situation can seriously impact the application of multi-label audio classification models on devices like headphones and mobile phones.

[0024] In response to the above problems, the present invention provides a training method for an audio classification model. Through a class incremental learning scheme, when a new classification category is needed, only the new category data can be used to train the extended model. This can avoid the problem of forgetting the original category knowledge caused by learning the new category, and can also prevent the new extended model from being overly biased towards the new category due to the imbalance of the new and old category data. Among them, class incremental learning is a learning method for gradually learning new categories. This method can quickly complete the learning and deployment of new categories of the existing original model, avoid the redesign and training of the new extended model, and enable the extended model after the original model is extended to efficiently learn new categories. The design of the extended model of the present invention is based on multi-scale depth-separable convolution, which greatly reduces the number of network parameters and speeds up the calculation. This design not only enables the network to be easily accelerated using arm operators during deployment to achieve low-power real-time processing, but also significantly reduces the demand for computing power and memory of the deployment device, providing excellent conditions for efficient deployment in resource-constrained environments. This extended model has low computational complexity and a small number of parameters, and can be deployed on resource-constrained edge devices such as mobile phones and headphones.

[0025] See Figure 1 ,The training method of the audio classification model includes the following steps:

[0026] S1. Obtain an extended audio set and its corresponding audio tag set.

[0027] The extended audio set is a new part of audio samples relative to the original audio set, wherein the original audio set is used to train the original model, and these samples only cover several audio categories that were initially predefined. After the original model is trained, when it is necessary to recognize audio of a new category, the audio samples containing the newly added categories are used as the extended audio set. Among them, the extended audio set only includes audio samples of categories that are newly added relative to the original audio set, and the extended audio set is composed of a mixture of single-label audio and multi-label audio. Among them, single-label audio represents audio samples that only belong to the newly added category, and multi-label audio represents audio samples that belong to both the newly added category and the original category. For each audio sample in the extended audio set, a corresponding label is generated according to its category to form an audio label set, which is used to guide the training of the extended model. Exemplarily, the original audio set includes three audio categories: "human voice", "dog barking" and "cat meowing". In addition to these three categories, the extended audio set also includes two new categories: "chicken crowing" and "background sound". For each single-label audio in the extended audio set, its label is only one of the new categories (such as "chicken crowing" or "background sound"). For each multi-label audio in the extended audio set, its label will be marked with at least one original category and at least one new category (such as labels such as "human voice", "background sound" and "chicken crowing").

[0028] S2. Input the extended audio set into the trained original model, and extract the generated reference original category labels and original features.

[0029] See Figure 2, the extended audio set is input into the trained original model, wherein the original model includes a feature extraction network and a classifier, the feature extraction network is used to extract high-level feature representations from the input audio data, and the classifier is used to classify based on the extracted features and assign the audio samples to corresponding categories (such as "human voice", "dog barking", "background sound", etc.). Since the original model is trained using the original audio set (such as "human voice", "dog barking", etc.), it can identify the relevant original categories in the extended audio set and has audio classification function, but it does not have the ability to recognize newly added categories (such as "chicken crowing" or "background sound"). The extended audio set is input into the trained original model, and for each input extended audio sample, the original model generates at least one reference original category label. Among them, the reference original category label refers to the category label output by the original model after predicting the input audio sample based on the original category it has learned (such as "human voice", "dog barking", etc.), which represents the classification result of the original model on the extended audio sample for the original category. At the same time, the output features of a layer in the original model are extracted as raw features. Considering that the features output by the feature extraction network are the best feature representation of the input data, the output of the previous layer of the original model classifier (i.e., the feature extraction network) is preferably used as the raw features. This raw feature serves as an intermediate representation of the audio sample, indicating the feature representation capability of the original model on the input data.

[0030] S3. Input the extended audio set into the extended model, extract the generated original category labels and / or newly added category labels and the extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same structure as the corresponding structure of the original model.

[0031] The structure of the extended model is the same as that of the original model, and also includes a feature extraction network and a classifier. The feature extraction network of the extended model has the same structure as that of the original model, and uses the trained feature extraction network parameters of the original model through direct migration. The classifier of the extended model is generated based on the newly added categories and the original categories in the extended audio set, so that it can simultaneously identify and classify audio samples of the original categories and the newly added categories. Specifically, the output unit of the classifier of the extended model is added to the classifier of the original model with the same number of output units as the number of audio categories that need to be newly learned.

[0032] In one embodiment of the present invention, inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model, includes:

[0033] Preprocessing the extended audio set to generate an extended audio feature set;

[0034] Inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution;

[0035] The extended features are input into the classifier of the extended model to generate original category labels and / or newly added category labels corresponding to the extended features.

[0036] Each extended audio sample in the extended audio set is preprocessed to obtain an extended audio feature set. The extended audio feature set is input into the feature extraction network of the extended model. Based on the multi-scale depth-separable convolution operation, the multi-level feature information of each extended audio sample is captured, thereby generating and extracting extended features. The extracted extended features are input into the classifier of the extended model. Since the classifier includes several cascaded fully connected layers, the fully connected layer performs a linear transformation on the input extended features and calculates the linear combination of the outputs. These outputs are input into the activation layer, and the features output by the fully connected layer are nonlinearly transformed through a nonlinear activation function. According to the probability distribution of the output, the original category label and / or the newly added category label of each sample are generated. It should be noted that since the feature extraction networks of the original model and the extended model are both composed of multi-scale depth-separable convolution blocks, the original model and the extended model are both lightweight models.

[0037] In one embodiment of the present invention, preprocessing the extended audio set to generate the extended audio feature set includes:

[0038] Performing a fast Fourier transform on each audio in the extended audio set to generate a corresponding spectrogram;

[0039] Perform Mel filter bank processing on each spectrum graph to generate the corresponding Mel spectrum graph;

[0040] Perform logarithmic compression on each Mel-spectrogram;

[0041] The compressed mel-spectrograms are respectively subjected to discrete cosine transformation to obtain the extended audio feature set constructed by the transformed mel-spectrograms.

[0042] In order to convert the original audio data into a feature representation that is more suitable for deep learning model processing, a series of preprocessing operations need to be performed on the extended audio set before the extended audio set is input into the extended model / original model. Specifically, each audio sample in the extended audio set is first subjected to a fast Fourier transform, thereby transforming the audio sample from a time domain signal to a frequency domain signal and generating a corresponding spectrogram. For the spectrogram of each audio sample: a Mel filter bank is applied for processing to generate a corresponding Mel spectrogram. Logarithmic compression is performed on the Mel spectrogram of each audio sample to obtain a logarithmic Mel spectrogram of each audio sample to reduce the dynamic range. The logarithmic Mel spectrogram of each audio sample is subjected to a discrete cosine transform to obtain a Mel-frequency cepstral coefficient feature vector for each audio sample. The Mel-frequency cepstral coefficient feature vectors of all audio samples are combined to form an extended audio feature set.

[0043] In one embodiment of the present invention, inputting the extended audio feature set into the feature extraction network of the extended model, and generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution, includes:

[0044] Based on a depthwise separable convolution of a first scale and a depthwise separable convolution of a second scale, extracting a first feature and a second feature of the extended audio feature set; wherein the first feature and the second feature correspond to the depthwise separable convolution of the corresponding scales respectively;

[0045] Fusing the first feature and the second feature to obtain a comprehensive feature;

[0046] Performing batch normalization on the comprehensive features to generate normalized features;

[0047] The normalized features are subjected to nonlinear activation processing to generate and extract extended features.

[0048] The multi-scale depthwise separable convolution block in the present invention includes two depthwise separable convolutions with different feature scales, a batch normalization layer, and a nonlinear activation function. Specifically, two depthwise separable convolutions of different scales (one at a first scale and one at a second scale) are used to extract corresponding features from the extended audio feature set, respectively, to obtain first features and second features. Specifically, a convolution operation is performed on the extended audio feature set using a depthwise separable convolution kernel of the first scale. Each input channel is convolved separately using the depthwise convolution of the first scale to extract the local features of the channel. The output of the depthwise convolution is convolved again using pointwise convolution to integrate the features of each channel to generate the integrated first feature. In addition, a second-scale depthwise separable convolution is used to convolve the extended audio to generate the second feature. The first scale is smaller than the second scale. Feature fusion of the extracted first and second features can combine feature information of different scales to obtain a more comprehensive and detailed integrated feature. The integrated features are batch normalized to improve the convergence speed and effect of the model training team. The normalized features are nonlinearly activated using a nonlinear activation function to generate and extract the extended features.

[0049] The present invention realizes real-time multi-label audio classification through a uniquely designed lightweight neural network (i.e., an extended model). The lightweight neural network is based on depthwise separable convolution, which uses depthwise separable convolution of different scales to capture multi-scale frequency domain and time domain information of audio. Depthwise separable convolution includes two core steps: depthwise convolution and pointwise convolution. In depthwise convolution, each input channel is convolved with an independent convolution kernel to achieve parameter sharing between channels. This strategy reduces the number of convolution kernels and significantly reduces the number of parameters, successfully extracting important features of the input audio data at a lower computational cost. At the same time, multi-scale convolution can extract multi-scale information of audio features, and the obtained feature information is richer. Compared with traditional convolution, this innovative method not only reduces the number of parameters and complexity while maintaining model performance, but also realizes real-time audio classification, providing an efficient and feasible solution for the practical application of multi-label audio classification.

[0050] In one embodiment of the present invention, inputting the extended features into the classifier of the extended model to generate the original category label and / or the newly added category label corresponding to the extended features includes:

[0051] Performing feature mapping on the extended features to obtain confidence scores for each audio category;

[0052] The category with a confidence level greater than a preset threshold is selected as the original category label and / or the newly added category label.

[0053] The extended features are input into the extended model's classifier. Through each fully connected layer in the classifier, the extended features are linearly transformed. The output of the last fully connected layer is used as the output vector. Each element in the output vector represents the confidence score that the audio sample belongs to each category. The extended model's activation function then maps the linearly transformed output vector to a preset range, generating the confidence score for each element. Each element in the generated confidence vector is then screened, and the category with a confidence score greater than a preset threshold is selected as the final classification result.

[0054] It should be noted that since the feature extraction network of the original model and the extended model have the same structure, and the classifiers are composed of the same number of fully connected layers, when the original audio set is used to train the original model, its training process is the same as that of the above-mentioned extended model, both including the same preprocessing operations on the original audio set as well as the feature extraction and classification processes, which will not be described in detail here.

[0055] S4. Based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio tag set, update the parameters of the extended model and use the trained extended model as the audio classification model.

[0056] By calculating the difference between the original category labels generated by the extended model and the reference original category labels, the difference between the extended features and the original features, and the difference between the original category labels, the newly added category labels and the real audio labels, a comprehensive loss is formed. This comprehensive loss is used to optimize the parameters of the extended model, and finally a multi-label audio classification model is trained that can maintain the ability to recognize the original categories while effectively learning the newly added categories.

[0057] In one embodiment of the present invention, updating the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label, and the corresponding audio label in the audio label set, and using the trained extended model as the audio classification model includes:

[0058] Calculating the output difference loss between the original category label and the reference original category label;

[0059] Calculating the feature difference loss between the expanded feature and the original feature;

[0060] Calculating the autonomous learning cross entropy loss between the original category label, the newly added category label, and the corresponding audio label in the audio label set;

[0061] Performing a weighted summation on the output difference loss, the feature difference loss, and the autonomous learning cross entropy loss to obtain a comprehensive loss;

[0062] The parameters of the extended model are updated based on the comprehensive loss, and the trained extended model is used as the audio classification model.

[0063] Calculate the output difference loss between the original category labels generated by the extended model (i.e., the prediction results of the original category by the extended model) and the reference original category labels (i.e., the prediction results of the original category generated by the original model). The types of output difference loss include, but are not limited to, cross-entropy loss or KL (Kullback-Leibler) divergence. Preferably, the output difference loss is KL divergence. Calculate the difference loss between the extended features of the extended model (features output by the extended model feature extraction network) and the original features of the original model (features output by the original model feature extraction network). The difference loss can be calculated using methods such as cosine similarity loss or mean squared error loss. Preferably, the difference loss is cosine similarity loss. Calculate the self-learning cross-entropy loss between the original category labels and the newly added category labels generated by the extended model and the corresponding true labels in the audio label set. A weighted sum of the above three losses is used to obtain a comprehensive loss. The parameters of the extended model are updated based on the comprehensive loss using a backpropagation algorithm. Continuous iterative training ultimately results in a trained extended model, which is used as the final multi-label audio classification model. The multi-label audio classification model has the ability to recognize both the original and newly added categories. In the present invention, the original features extracted by the original model parameters and the output results are used to calculate the cosine similarity loss and Kullback-Leibler divergence loss with the extended model to solve the problem of knowledge forgetting caused by the extended model in the process of learning new categories. It can effectively help the trained model learn new classification categories, avoid retraining of old data, greatly save the training time of the extended model, and improve the ability of the extended model to learn new classification categories. In addition, the present invention adopts an autonomous learning cross entropy loss to independently calculate the binary cross entropy loss of the original category and the newly added category. This loss can avoid the problem that after the introduction of the new category, the category prediction of the extended model is more inclined to the new category due to data imbalance.

[0064] In one embodiment of the present invention, calculating the autonomous learning cross entropy loss between the original category label, the newly added category label, and the corresponding audio label in the audio label set includes:

[0065] Calculating an original category cross entropy loss between the original category label and the corresponding audio label in the audio label set;

[0066] Calculating a new category cross entropy loss between the new category label and the corresponding audio label in the audio label set;

[0067] The original category cross entropy loss and the newly added category cross entropy loss are weighted and summed to obtain the autonomous learning cross entropy loss.

[0068] The error between the extended model's predictions for the original categories and the true original category labels in the audio label set, known as the original category cross-entropy loss, is calculated to ensure that the model maintains its ability to correctly identify the original categories while learning the newly added categories. The error between the extended model's predictions for the newly added categories and the true newly added category labels in the audio label set, known as the newly added category cross-entropy loss, is calculated to enable the model to accurately learn and recognize the features of the newly added categories. The weighted sum of these two losses yields the autonomous learning cross-entropy loss.

[0069] It should be noted that the extended model designed in this paper is a lightweight model consisting of a feature extraction network and a classifier. The feature extraction network is composed of a multi-scale depthwise separable convolutional block, which includes two depthwise separable convolutions at different feature scales, a batch normalization layer, and a nonlinear activation function. The classifier consists of a fully connected layer. This extended model has a small number of parameters and low computational complexity, making it suitable for deployment on resource-constrained, low-computing devices.

[0070] In one embodiment of the present invention, a method for audio classification is further provided, the method comprising:

[0071] Get the audio to be classified;

[0072] The audio is input into any of the audio classification models described above, extended features of the audio are extracted, and original category labels and / or newly added category labels of the audio are generated based on the extended features.

[0073] After obtaining the audio to be classified, it is input into the multi-label audio classification model described in any of the above items. Extended features are extracted from the audio through the feature extraction network. These features contain the key information of the audio signal. The extended features are input into the classifier of the audio classification model. Based on the extracted extended features, the audio sample is analyzed to generate the corresponding classification label. Among them, the classification label is the original category label (such as "human voice", "dog barking") and / or the newly added category label (such as "background sound") to which the audio belongs, thereby completing the multi-category classification of the audio sample.

[0074] See Figure 3The training system 100 of the multi-label audio classification model includes: an audio acquisition module 110, an original label acquisition module 120, an extended label acquisition module 130 and a model training module 140. The audio acquisition module 110 is used to obtain an extended audio set and a corresponding audio label set. The original label acquisition module 120 is used to input the extended audio set into the trained original model and extract the generated reference original category labels and original features. The extended label acquisition module 130 is used to input the extended audio set into the extended model and extract the generated original category labels and / or newly added category labels as well as the extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model is the same as the corresponding structure of the original model. The model training module 140 is used to update the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and use the trained extended model as the audio classification model.

[0075] For the specific limitations of the training system for the multi-label audio classification model, please refer to the limitations of the training method for the multi-label audio classification model mentioned above, which will not be repeated here. The various modules in the above-mentioned training system for the multi-label audio classification model can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware format, or can be stored in the memory of the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0076] It should be noted that, in order to highlight the innovative part of the present invention, this embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in this embodiment.

[0077] See Figure 4 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a training program for an audio classification model.

[0078] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Furthermore, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed on the electronic device 1, such as the code for training an audio classification model, but can also be used to temporarily store data that has been output or is to be output.

[0079] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, connecting the various components of the entire electronic device 1 using various interfaces and lines. It executes or executes programs or modules stored in the memory 12 (such as a training program for an audio classification model) and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0080] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above-mentioned audio classification model training method.

[0081] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an audio acquisition module 110, an original label acquisition module 120, an expanded label acquisition module 130, and a model training module 140.

[0082] The above-mentioned integrated unit implemented in the form of a software functional module can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The above-mentioned software functional module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to perform part of the functions of the audio classification model training method described in various embodiments of the present application.

[0083] In summary, the present invention discloses a training method, system, device and medium for an audio classification model. The incremental learning scheme adopted can efficiently help the existing audio classification model learn the classification function of new categories of audio data. It only needs to use the newly added category data to train the extended model on the basis of the original model to obtain an extended model that can identify the newly added categories and the original categories. The present invention uses the intermediate features and output results of the original model and the extended model to calculate the feature difference loss and the result difference loss, which solves the problem of knowledge forgetting caused by the extended model learning the newly added categories. At the same time, the autonomous learning cross-entropy loss is calculated for the output results of the extended model, which overcomes the problem of the extended model predicting the newly added categories caused by the imbalance between the newly added category data and the original category data. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0084] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A training method for an audio classification model, characterized in that: The method comprises: Get the extended audio set and the corresponding audio tag set; Inputting the extended audio set into the trained original model, and extracting the generated reference original category labels and original features; Inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same structure as the corresponding structure of the original model; Based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, the parameters of the extended model are updated, and the trained extended model is used as the audio classification model.

2. The method for training an audio classification model according to claim 1, wherein: Inputting the extended audio set into the extended model, extracting the generated original category labels and / or newly added category labels and extended features generated by the same network layer as the original model, includes: Preprocessing the extended audio set to generate an extended audio feature set; Inputting the extended audio feature set into the feature extraction network of the extended model, generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution; The extended features are input into the classifier of the extended model to generate original category labels and / or newly added category labels corresponding to the extended features.

3. The method for training an audio classification model according to claim 2, wherein: The preprocessing of the extended audio set to generate an extended audio feature set includes: Performing a fast Fourier transform on each audio in the extended audio set to generate a corresponding spectrogram; Perform Mel filter bank processing on each spectrum graph to generate the corresponding Mel spectrum graph; Perform logarithmic compression on each Mel-spectrogram; The compressed mel-spectrograms are respectively subjected to discrete cosine transformation to obtain the extended audio feature set constructed by the transformed mel-spectrograms.

4. The method for training an audio classification model according to claim 2, wherein: Inputting the extended audio feature set into the feature extraction network of the extended model, and generating and extracting extended features of the extended audio feature set based on multi-scale depthwise separable convolution, comprises: Based on a depthwise separable convolution of a first scale and a depthwise separable convolution of a second scale, extracting a first feature and a second feature of the extended audio feature set; wherein the first feature and the second feature correspond to the depthwise separable convolution of the corresponding scales respectively; Fusing the first feature and the second feature to obtain a comprehensive feature; Performing batch normalization on the comprehensive features to generate normalized features; The normalized features are subjected to nonlinear activation processing to generate and extract extended features.

5. The method for training an audio classification model according to claim 2, wherein: Inputting the extended features into the classifier of the extended model to generate original category labels and / or newly added category labels corresponding to the extended features includes: Performing feature mapping on the extended features to obtain confidence scores for each audio category; The category with a confidence level greater than a preset threshold is selected as the original category label and / or the newly added category label.

6. The method for training an audio classification model according to claim 1, wherein: The updating of parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label, and the corresponding audio label in the audio label set, and using the trained extended model as the audio classification model, includes: Calculating the output difference loss between the original category label and the reference original category label; Calculating the feature difference loss between the extended feature and the original feature; Calculating the autonomous learning cross entropy loss between the original category label, the newly added category label, and the corresponding audio label in the audio label set; Performing a weighted summation on the output difference loss, the feature difference loss, and the autonomous learning cross entropy loss to obtain a comprehensive loss; The parameters of the extended model are updated based on the comprehensive loss, and the trained extended model is used as the audio classification model.

7. The method for training an audio classification model according to claim 6, wherein: The calculating of the autonomous learning cross entropy loss between the original category label, the newly added category label, and the corresponding audio label in the audio label set includes: Calculating an original category cross entropy loss between the original category label and the corresponding audio label in the audio label set; Calculating a new category cross entropy loss between the new category label and the corresponding audio label in the audio label set; The original category cross entropy loss and the newly added category cross entropy loss are weighted and summed to obtain the autonomous learning cross entropy loss.

8. An audio classification method, characterized in that: The method comprises: Get the audio to be classified; The audio is input into the audio classification model according to any one of claims 1 to 7, extended features of the audio are extracted, and original category labels and / or newly added category labels of the audio are generated based on the extended features.

9. A training system for an audio classification model, characterized in that: The system comprises: An audio acquisition module is used to acquire an extended audio set and a corresponding audio tag set; an original label acquisition module, configured to input the extended audio set into a trained original model and extract the generated reference original category labels and original features; An extended label acquisition module, configured to input the extended audio set into the extended model, extract the generated original category labels and / or newly added category labels, and extended features generated by the same network layer as the original model; wherein the feature extraction network of the extended model has the same structure as the corresponding structure of the original model; A model training module is used to update the parameters of the extended model based on the difference between the original category label and the reference original category label, the difference between the extended feature and the original feature, and the difference between the original category label, the newly added category label and the corresponding audio label in the audio label set, and use the trained extended model as the audio classification model.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the training method of the audio classification model as described in any one of claims 1 to 7 or the audio classification method as described in claim 8.

11. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the training method of the audio classification model according to any one of claims 1 to 7 or the audio classification method according to claim 8.

Images