A few-sample class variable audio classification method based on prototype adaptive network

Abstract

This invention discloses a few-sample class-variable audio classification method based on a prototype adaptive network. The steps are as follows: 1) Extracting log-Mel spectrum from input audio samples; 2) In the basic stage, training an initial model consisting of a feature extractor, a prototype classifier, and a class-adjustable prototype adaptive network using base class samples; 3) Extracting representations from the base class training samples, calculating the mean of each base class representation as a prototype, and updating the prototype classifier; 4) In the incremental stage, if the number of categories increases, inputting the representations of the incremental class training samples into the class-adjustable prototype adaptive network to obtain the prototypes of each incremental class and updating the class-adjustable prototype adaptive network; if the number of categories decreases, deleting prototypes of unnecessary categories; 5) Then updating the prototype classifier using the class-adjustable prototype adaptive network; 6) Obtaining the classification category of the test sample. This invention can adaptively expand or shrink the prototype classifier, thereby continuously learning new categories while remembering or deleting old categories, achieving few-sample class-variable audio classification.

Description

Technical Field

[0001] This invention belongs to the field of few-shot learning and speech signal processing technology, specifically relating to a few-shot class variable audio classification method based on prototype adaptive networks. Background Technology

[0002] In the real world, sound is one of the most frequently encountered perceptual information types for humans, widely present in daily life and interactions with smart devices. Speech and music are two of the most representative types of audio signals, thus audio information processing occupies a crucial position in multimedia research. Audio classification, a core task in the field of audio signal processing, aims to determine the input audio sample into one of predefined categories, such as acoustic event type, instrument type, or environmental scene category. This technology supports applications such as intelligent voice assistants, environmental sound monitoring, and video content understanding and retrieval, and also provides a foundation for subsequent recognition, retrieval, and decision-making. Compared to images, audio signals exhibit strong temporal sequence and noise diversity, with significant distribution differences arising from different acquisition devices and acoustic environments. In recent years, deep neural networks, with their stronger modeling and representation learning capabilities, can learn discriminative representations from time-frequency features or raw waveforms, gradually replacing traditional methods relying on manual features and becoming the mainstream solution for audio classification systems.

[0003] However, deep learning methods typically rely heavily on large-scale, finely labeled datasets during training, resulting in significant time and computational resource consumption. In practical applications, limitations such as acquisition costs, annotation difficulty, privacy protection, and device differences often make it difficult to obtain large-scale samples that strictly match the target task. Simultaneously, datasets and business requirements are continuously updated, with new categories often appearing in small sample sizes. If the model is incrementally updated directly with a limited number of samples, it is prone to overfitting to new classes and may cause catastrophic forgetting of previously learned class knowledge. Retraining with all historical samples, on the other hand, incurs unbearable time and storage costs, and historical data is often not allowed to be stored long-term or reused repeatedly in many scenarios. Therefore, few-shot incremental learning aims to achieve effective adaptation and continuous updating of the model by relying only on a small number of new class samples. This allows for a balance between learning new knowledge and retaining old knowledge under conditions of insufficient samples and limited storage space, improving the model's continuous usability across changing scenarios.

[0004] While some work has attempted to apply incremental learning with few-shot classes to audio classification, real-world tasks often involve not only class additions but also class reductions, such as due to security policy adjustments, redundant class cleanup, device deployment, and scene migration. Improper handling can lead to model classification boundary drift and class confusion, reducing system stability and maintainability. Therefore, research on variable audio classification with few-shot classes is of great significance. In the dynamic process of simultaneous addition and deletion of classes, the model must adapt quickly with a small number of new class samples while maintaining its classification performance for existing classes. Summary of the Invention

[0005] The purpose of this invention is to address the problem of models remembering old classes and continuously learning new classes in situations where categories change dynamically. This invention provides a few-shot class-variable audio classification method based on a prototype adaptive network. The method designs a few-shot class-variable audio classification model consisting of a feature extractor, a prototype classifier, and a class-adjustable prototype adaptive network. By adjusting the prototype classifier through the class-adjustable prototype adaptive network, this invention further extends the audio classification problem to real-world environments where categories change dynamically, thus achieving few-shot class-variable audio classification.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A few-sample class-variable audio classification method based on a prototype adaptive network, the audio classification method comprising the following steps:

[0008] S1. Extracting Log-Mel spectrum features: Extracting log-Mel spectrum features from audio samples, wherein the categories of the audio samples include audio categories of the base stage and audio categories of the incremental stage. Hereinafter, the audio categories of the base stage are referred to as the base class, and the audio categories of the incremental stage are referred to as the incremental class.

[0009] S2. Initialize and train the audio classification model: In the basic stage, supervised learning is used to train the feature extractor, and a pseudo-class incremental training strategy is used to train the class-adjustable prototype adaptive network. The parameters of the feature extractor and the class-adjustable prototype adaptive network are optimized until convergence. After training, the parameters of the feature extractor and the class-adjustable prototype adaptive network are frozen.

[0010] S3. Construct a base class prototype classifier: Input the log-Mel spectrum features of the base class training samples into the feature extractor to obtain the base class representation, then calculate the mean of the representations of the same class to obtain the prototype of each base class, and then input the base class prototype into the class-adjustable prototype adaptive network for updating.

[0011] S4. Different processing is performed according to the category change: In the incremental stage where the category increases, the log-Mel spectrum of the incremental class support set is input into the feature extractor to obtain the support representation, and the support representation is input into the generation module of the class adjustable prototype adaptive network to obtain the prototype of the incremental class, while updating the class adjustable prototype adaptive network; In the incremental stage where the category decreases, the prototype of the unnecessary category is deleted.

[0012] S5. Update the prototype classifier: Input the log-Mel spectrum of the query set of the current stage into the feature extractor to obtain the query representation. Input the query representation and the prototype of the previous stage into the adaptive module of the updated class-adjustable prototype adaptive network, calculate the correlation between the query representation and the prototype, and thus update the query representation and the prototype. All updated prototypes are used as prototypes for the dynamically expanded classifier.

[0013] S6. Classify the test sample: Input the test sample, calculate the cosine similarity between the updated query representation and the prototypes of each category in the dynamic scalable classifier, and determine the category corresponding to the prototype with the largest cosine similarity as the category to which the test sample belongs, thus obtaining the audio classification result.

[0014] Furthermore, the Mel-scale can approximate the nonlinear sensitivity of the human ear to different frequencies, and is more consistent with auditory perception characteristics than the linear spectrum. This allows for more effective preservation of key auditory information needed to distinguish audio categories, improving the matching degree between extracted features and classification tasks. The log-Mel spectrum feature extraction process in step S1 is as follows:

[0015] S1.1. A first-order high-pass filter is used to pre-emphasize the input audio.

[0016] S1.2. Perform frame segmentation on the pre-emphasized input audio to obtain several short audio frames;

[0017] S1.3 Windowing process for short audio frames: Hamming window is used as the window function, and the window function is multiplied by each audio frame to obtain the windowed short audio frame;

[0018] S1.4 Perform a discrete Fourier transform on the windowed short-time audio frame to obtain the linear spectrum of the corresponding short-time audio frame. Then, use a Mel filter to map the linear spectrum to the Mel scale to obtain the Mel spectrum. Take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum feature.

[0019] Furthermore, the feature extractor employs a residual convolutional neural network ResNet18, comprising one sequentially connected convolutional layer, eight residual blocks, one average pooling layer, one fully connected layer, and one softmax layer. The softmax layer is used only for supervised learning during the feature extractor training phase and is removed after training. Since the log-Mel spectrum is a two-dimensional feature, the repeatedly stacked residual convolutional structure of ResNet18 can deeply mine the representation of audio samples, exhibiting good feature representation ability and strong generalization.

[0020] The prototype classifier consists of multiple prototypes, each corresponding to an audio class. The prototypes of the base class in the prototype classifier are obtained by calculating the average vector of the representation of the same audio class. The prototypes of the incremental class are generated by the class-adjustable prototype adaptive network. The prototype classifier uses the class center vector for measurement and discrimination. It has a small number of parameters and strong generalization ability. The addition and removal of prototypes supports the addition and deletion of categories.

[0021] The class-adaptive prototype network is a self-attention network, comprising a sequentially connected generation module, parallel-connected stability and plasticity adaptation branches, and a fusion module. The generation module includes a sequentially connected self-attention layer, a batch normalization layer, and a mean calculation layer. This module is used to obtain the interrelationships between incremental class support representations, generating more representative incremental class prototypes and improving the discriminative ability of the prototype classifier. The stability and plasticity adaptation branches have identical structures, including a sequentially connected self-attention layer and a normalization layer, both used to obtain the global position information of all prototypes in the prototype space. The system makes global adjustments to all prototypes based on information from all prototypes. The difference between the two is that the stability adaptation branch is frozen in the incremental stage, while the plasticity adaptation branch is updated. Therefore, the prototype adjusted by the stability adaptation branch can stabilize the prototype classifier's ability to distinguish the base class, while the prototype adjusted by the plasticity adaptation branch can improve the prototype classifier's ability to distinguish the incremental class. The fusion module is a multilayer perceptron consisting of a linear layer, a ReLU activation layer, another linear layer, and a Sigmoid function. The fusion module weights and fuses the outputs of the stability adaptation branch and the plasticity adaptation branch to balance the classifier's ability to distinguish the base class and the incremental class.

[0022] Furthermore, in the foundational stage, a feature extractor is first trained using supervised learning. Then, the trained feature extractor is used to extract representations, and a class-adjustable prototype adaptive network is trained using a pseudo-class incremental training strategy. Step S2 is as follows:

[0023] S2.1 The feature extractor is trained using supervised learning with all samples from the base class training set, and the optimization process employs the cross-entropy loss function. The mathematical expression is: ,

[0024] in, The number of training samples. Let m be the number of categories in the m-th stage, 0 ≤ m ≤ (M-1), where M represents the number of stages. for

[0025] The number of categories in the basic process Predict samples for the model The true label, Predict samples for the model The probability of belonging to category c;

[0026] S2.2. Train the class-adjustable prototype adaptive network using a pseudo-class variable training strategy and cross-entropy loss, and optimize the parameters of the class-adjustable prototype adaptive network until convergence.

[0027] S2.3 After the audio classification model initialization training is completed, the initial audio classification model parameters θ0 are obtained. Then, the parameters of the feature extractor, class-adjustable prototype adaptive network generation module and stability adaptation branch are frozen.

[0028] Furthermore, the class-tunable prototype adaptive network requires a large number of new class samples for training to achieve better generalization ability in the incremental stage. However, it cannot obtain information about the real new classes in the basic stage. Therefore, a "supplementary learning scenario" is constructed by sampling the base class data to synthesize pseudo-incremental classes. This allows the class-tunable prototype adaptive network to learn in advance how to update the prototype and maintain the stability of the old class under dynamic class changes and limited sample constraints. The training process of the class-tunable prototype adaptive network in step S2.2 is as follows:

[0029] S2.2.1 Sampling two datasets from the base class data and Use sampling mixing factor Synthesize pseudo-incremental classes and assign pseudo-labels to them to construct a pseudo-incremental class dataset. Then, the pseudo-incremental class dataset Classified as pseudo-incremental support set and pseudo-incremental query sets , among which, the mixing factor Values ​​between 0 and 1, and , Indicates shape parameters as The Beta distribution;

[0030] S2.2.2. Use the feature extractor trained in step S2.1 to extract the support and query representations of the log-Mel spectrum of pseudo-incremental audio samples;

[0031] S2.2.3. Input the support representation of the pseudo-increment class into the generation module of the class-adjustable prototype adaptive network to generate the support prototype of the pseudo-increment class. Then, concatenate the query representation of the pseudo-increment class and the support prototype and input them into the adaptation branch and fusion module of the class-adjustable prototype adaptive network. Finally, the adjusted prototype and query representation are obtained through vector segmentation.

[0032] S2.2.4 Calculate the cosine similarity between the adjusted query representation and each prototype. By continuously reducing the cross-entropy loss function, use the stochastic gradient descent algorithm to optimize the parameters of the class-adjustable prototype adaptive network until convergence.

[0033] Furthermore, in the basic prototype classifier, the average value of each prototype of the base class is calculated from the support representation of its class. The calculation formula for the prototype in step S3 is as follows: ,

[0034] In the formula Let c represent the prototype of category c in the m-th stage, where c = 1, 2, ... ; Let represent the log-Mel spectrum of the nth support sample, where n = 1, 2, ..., K, and K represents the number of support samples; This represents the character extraction function. This represents the first parameter used to implement the feature extraction function.

[0035] Furthermore, to simulate the dynamic changes in audio categories in a real-world scenario, the model needs to handle different situations of category changes differently in the incremental stage. In step S4, when the number of categories decreases, prototypes of unnecessary categories are deleted. In step S4, when the number of categories increases, the data input to the audio classification model includes... There are 1 audio class, each containing 1 audio class. One supporting sample, One query sample.

[0036] Furthermore, when the number of categories increases, the prototype of the current category needs to be generated using the input incremental class support set, and the class-adjustable prototype adaptive network needs to be updated. The process of step S4 is as follows:

[0037] S4.1 Input the log-Mel spectrum of the incremental support set into the feature extractor to obtain the support representation. Supporting characterization The dimension is , Supporting characteristics Constructing a supporting representation set Then, the supporting representation set As input to the generative module in a class-tunable prototypic adaptive network, After linear transformation, three matrices are obtained, namely: , and ,in , , and It consists of four trainable first, second, third, and fourth linear transformation layers. and Matrix multiplication and division Then perform a Softmax operation, and combine the matrix output by the Softmax operation with... Matrix multiplication, followed by the fourth linear transformation layer. Later obtained , ,

[0038] In the formula, the upper right corner T represents the transpose of the matrix; then... and After adding element by element and performing layer normalization, we get , ,

[0039] In the formula, ( ) represents layer normalization operation; finally, along First dimension calculation The average value is used to obtain the prototype of the current stage. ;

[0040] S4.2, Freeze the parameters of the stability adaptation branch in the adjustable prototype adaptive network;

[0041] S4.3, The prototype of the current stage Previous stage prototype and supporting characterization After being concatenated into a matrix, it is input into a class-adjustable prototype adaptive network to update all prototypes and support representation vectors;

[0042] S4.4 Predict the labels of the support set using a class-tunable prototype adaptive network and calculate the cross-entropy loss;

[0043] S4.5. Use the stochastic gradient descent algorithm to optimize the plasticity adaptation branch and fusion module in the class-adjustable prototype adaptive network.

[0044] Furthermore, in scenarios with small sample sizes and variable class representation, the incremental class prototype is susceptible to deviation due to insufficient samples and noise. The old class prototype may also become mismatched with the query samples when the class changes. To ensure that the classification boundary remains stable and discriminative after class changes, the prototype and query representation need to be synchronously calibrated. Therefore, an adaptive module of a class-adjustable prototype adaptive network is used. Through relevance modeling and attention update mechanisms, the prototype and query representation are dynamically adjusted to better fit the current class set and sample distribution, thereby improving classification accuracy and the model's class-variable adaptability. The process of step S5 is as follows:

[0045] S5.1 Input the log-Mel spectrum of the current query set into the feature extractor to obtain the query representation. Assuming the adaptive module of the class-tunable prototype adaptive network processes simultaneously Query representation The adaptive module consists of a stability adaptation branch, a plasticity adaptation branch, and a fusion module. The input variables for both branches of the adaptive module are... , This indicates the number of old categories in the current stage. This represents the total number of all categories in the current stage; the stability adaptation branch and the plasticity adaptation branch affect the input variables. The processing and generation module for The processing is the same, the outputs of the stability adaptation branch and the plastic adaptation branch are respectively and ;

[0046] S5.2, will and The fusion weights are obtained by concatenating the features along the feature dimension and inputting them into a multilayer perceptron consisting of a linear layer, a ReLU activation layer, and a sigmoid function. : ,

[0047] in, It is an activation function used to compress and map any real number to the range of 0 to 1. To represent a linear transformation operation, its form is: ,in For linear layer weight vectors, For bias vector, It is a non-linear activation function;

[0048] S5.3, Utilizing the above-mentioned fusion weights For stability output and plasticity output Perform weighted fusion to obtain the fused output. :

[0049] ;

[0050] S5.4, will It is divided into two parts: the updated query representation and the updated prototype Represent each updated query and the updated prototype The cosine similarity between samples is used to determine the category to which a sample belongs.

[0051] Furthermore, to verify the model's ability to remember or delete old classes while continuously learning new classes, the model performs audio classification on the query samples in the current stage. The process of step S6 is as follows:

[0052] S6.1 Extract the log-Mel spectrum of the sample to be tested;

[0053] S6.2 Input the log-Mel spectrum of the sample to be tested into the trained feature extractor to extract the query representation, concatenate it with the prototypes of all classes in the current stage and input it into the updated class-adjustable prototype adaptive network to obtain the adjusted query representation.

[0054] S6.3 Calculate the cosine similarity between the adjusted query representation of the test sample and each prototype in the classifier. The category corresponding to the prototype with the highest similarity is the category to which the test sample belongs, thus obtaining the audio classification result.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] 1) This invention proposes a framework for variable audio classification with a small number of samples. Audio categories can be added or deleted at each incremental stage. The model can remember existing old categories and continuously learn new categories based on a small number of training samples. This breaks the assumption of continuously increasing categories in traditional audio classification models, more closely reflects the dynamic changes in category sets in real-world applications, and expands the functionality of traditional audio classification models.

[0057] 2) This invention designs a class-adjustable prototype adaptive network. This network can dynamically update the prototypes of old and new classes, mitigating the impact of class increases or decreases on model performance. This makes the shrinking and expansion of the prototype classifier more adaptive and robust, effectively improving the model's performance in the framework of small-sample class variable audio classification.

[0058] 3) The present invention was tested on two public audio datasets (LS-100 and NSynth-100), and both achieved high average classification accuracy and small performance degradation rate, effectively realizing variable audio classification for small sample classes. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of an audio classification model network for a small-sample class variable audio classification method based on a prototype adaptive network disclosed in an embodiment of the present invention.

[0061] Figure 2 This is a flowchart illustrating the implementation method of a few-sample class variable audio classification method based on a prototype adaptive network disclosed in an embodiment of the present invention.

[0062] Figure 3 This is a network structure diagram of the feature extractor in an embodiment of the present invention;

[0063] Figure 4 This is a diagram of the class-adjustable prototype adaptive network structure in an embodiment of the present invention;

[0064] Figure 5 This is a diagram of the self-attention layer structure in an embodiment of the present invention.

[0065] Figure 6 This is an experimental result diagram of the small-sample class variable audio classification method based on prototype adaptive network in Embodiment 2 of the present invention on the LS-100 dataset. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0067] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0068] Example 1

[0069] Figure 1This is a model network structure diagram of a few-sample class-variable audio classification method based on a prototype adaptive network. The model includes a feature extractor and a classifier connected in sequence, as well as a class-tunable prototype adaptive network for adjusting the classifier. Figure 2 This is a flowchart illustrating the implementation of a few-shot class variable audio classification method based on a prototype adaptive network. It includes basic and incremental steps. The specific steps of this embodiment of the few-shot class variable audio classification method based on a prototype adaptive network are as follows:

[0070] S1. Extract log-Mel spectrum features from the input audio, where the input audio includes training audio and test audio;

[0071] In this embodiment, step S1, extracting the log-Mel spectrum features, specifically includes the following steps:

[0072] S1.1. A first-order high-pass filter is used to pre-emphasize the input audio.

[0073] S1.2. Perform frame segmentation on the pre-emphasized input audio to obtain several short audio frames;

[0074] S1.3 Windowing process for short audio frames: Select Hamming window as the window function, and multiply the window function with each audio frame to obtain the windowed short audio frame;

[0075] S1.4 Perform Discrete Fourier Transform on the windowed short-time audio frame to obtain the linear spectrum of the corresponding short-time audio frame. Then, use a Mel filter to map the linear spectrum to the Mel scale to obtain the Mel spectrum. Take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum feature.

[0076] In this embodiment, two audio datasets, LS-100 and NSynth-100, were constructed based on LibriSpeech and NSynth, respectively. LS-100 consists of speech samples from 100 speakers and is a speech corpus; NSynth-100 collects audio samples of 100 musical instruments. Each audio sample in the dataset corresponds to unique pitch, timbre, and amplitude information.

[0077] Tables 1 and 2 provide detailed information for the LS-100 and NSynth-100 datasets, respectively.

[0078] As shown in Tables 1 and 2, both the LS-100 and NSynth-100 datasets are divided into a base class dataset D0 with disjoint classes and an incremental class dataset D10. m (1≤m≤(M-1)). D0 contains the basic training dataset. and basic validation dataset D m Includes incremental training dataset and incremental validation dataset . Used for training feature extractors, classifiers, and class-tunable prototype adaptive networks. Used to update class-adjustable prototype adaptive network and prototype classifier. In each incremental step, in order to simulate the dynamic changes of audio categories in real-world scenarios, the operation of learning 5 new classes and deleting 2 old classes (1 base class and 1 incremental class) is performed alternately. The training data configuration when learning new classes is 5-way-5-shot (5 classes, and 5 samples for each class).

[0079] Table 1. Detailed information about the LS-100 dataset

[0080]

[0081] Table 2. Detailed information about the NSynth-100 dataset

[0082]

[0083] S2. Model initialization training: In the basic stage, supervised learning is used to train the feature extractor, and a pseudo-class incremental training strategy is used to train the class-adjustable prototype adaptive network. Figure 3 and Figure 4 These are schematic diagrams of the feature extractor network and the class-adjustable prototype adaptive network structure, respectively, according to embodiments of the present invention.

[0084] like Figure 3 As shown, the feature extractor adopts a cascaded structure, including one convolutional layer, eight residual blocks, one average pooling layer, one fully connected layer, and one softmax layer connected sequentially. The softmax layer is only used for supervised learning during the feature extractor training phase and is removed after training. Each residual block consists of two convolutional layers. The convolutional output is first activated by ReLU and then element-wise added to the corresponding input to form a residual connection. Figure 3 The specific parameter configurations for convolutional layers and residual blocks in ResNet are given, where "3×3 convolution, 64" indicates that the convolutional kernel size is 3×3, and both the input and output channels are 64. As the network deepens, the number of channels in the residual blocks gradually increases, with the last residual block having 512 channels. After average pooling and fully connected layers, the network outputs a 512-dimensional feature vector as the final extracted feature representation.

[0085] The classifier consists of prototypes, each representing an audio class. The prototypes of the base classes in the prototype classifier are obtained by calculating the average vector of the representations of the same audio class. Incremental class prototypes are generated through a class-adjustable prototype adaptive network, and the prototypes and sample representations are adjusted to optimize the decision results.

[0086] like Figure 4 As shown, the class-tunable prototype adaptive network is a self-attention network, including a sequentially connected generation module, a parallel-connected stability adaptation branch and a plasticity adaptation branch, and a fusion module. The generation module includes a sequentially connected self-attention layer, a batch normalization layer, and a mean calculation layer. The stability adaptation branch and the plasticity adaptation branch have the same structure, including a sequentially connected self-attention layer and a normalization layer. The fusion module is a multilayer perceptron consisting of a linear layer, a ReLU activation layer, another linear layer, and a sigmoid function.

[0087] In this embodiment, the training steps for the basic components are as follows:

[0088] S2.1 The feature extractor is trained using supervised learning with all samples from the base class training set, and the optimization process employs the cross-entropy loss function. The mathematical expression is: ,

[0089] in, The number of training samples. Let M be the number of categories in the m-th stage (0 ≤ m ≤ (M-1), where M represents the number of stages). The number of categories in the basic process. Predict samples for the model The true label, Predict samples for the model The probability of belonging to category c;

[0090] S2.2. Train the class-adjustable prototype adaptive network using a pseudo-class variable training strategy and cross-entropy loss, and optimize the parameters of the class-adjustable prototype adaptive network until convergence.

[0091] S2.2.1 Sampling two datasets from the base class data and Use sampling mixing factor Synthesize pseudo-incremental classes and assign pseudo-labels to them to construct a pseudo-incremental class dataset. Then, the pseudo-incremental class dataset Classified as pseudo-incremental support set and pseudo-incremental query sets , among which, the mixing factor Values ​​between 0 and 1, and , Indicates shape parameters as The Beta distribution;

[0092] S2.2.2. Use the feature extractor trained in step S21 to extract the support and query representations of the log-Mel spectrum of pseudo-incremental audio samples;

[0093] S2.2.3. The pseudo-incremental class support representation is generated by the generation module of the input class adjustable prototype adaptive network. Then, the pseudo-incremental class query representation and support prototype are concatenated with the adaptation branch and fusion module of the input class adjustable prototype adaptive network. Finally, the adjusted prototype and query representation are obtained through vector segmentation.

[0094] S2.2.4 Calculate the cosine similarity between the adjusted query representation and each prototype. By continuously reducing the cross-entropy loss function, use the stochastic gradient descent algorithm to optimize the parameters of the class-adjustable prototype adaptive network until convergence.

[0095] S2.3 After the audio classification model initialization training is completed, the initial audio classification model parameters θ0 are obtained. Then, the parameters of the feature extractor, class-adjustable prototype adaptive network generation module and stability adaptation branch are frozen.

[0096] S3. Input the log-Mel spectrum features of the base class training samples into the feature extractor to obtain the base class representation. Then calculate the mean of the representations of the same class to obtain the prototype of each base class. Then input the base class prototype into the prototype classifier for updating.

[0097] The formula for calculating the prototype is as follows: In the formula Let c represent the prototype of category c in the m-th stage, where c = 1, 2, ... ; Let represent the log-Mel spectrum of the nth support sample, where n = 1, 2, ..., K, and K represents the number of support samples; This represents the character extraction function. This represents the first parameter used to implement the feature extraction function;

[0098] S4. In the incremental stage, the model needs to handle different situations of category changes differently. When the number of categories decreases, simply delete the prototypes of the unnecessary categories; when the number of categories increases, the data input to the audio classification model includes... There are 1 audio class, each containing 1 audio class. One supporting sample, For each query sample, a prototype of the current category needs to be generated and the class-adjustable prototype adaptive network needs to be updated.

[0099] In this embodiment, step S4 is as follows:

[0100] S4.1 Input the log-Mel spectrum of the incremental support set into the feature extractor to obtain the support representation. Supporting characterization The dimension is , Supporting characteristics Constructing a supporting representation set Then, the supporting representation set As input to the generative module in a class-tunable prototypic adaptive network, After linear transformation, three matrices are obtained, namely: , and ,in , , and It consists of four trainable first, second, third, and fourth linear transformation layers. and Matrix multiplication and division Then perform a Softmax operation, and combine the matrix output by the Softmax operation with... Matrix multiplication, followed by the fourth linear transformation layer. Later obtained , ,

[0101] In the formula, the upper right corner T represents the transpose of the matrix; then... and After adding element by element and performing layer normalization, we get , ,

[0102] In the formula, ( ) represents layer normalization operation; finally, along First dimension calculation The average value is used to obtain the prototype of the current stage. ;

[0103] S4.2, Freeze the parameters of the stability adaptation branch in the adjustable prototype adaptive network;

[0104] S4.3, The prototype of the current stage Previous stage prototype and supporting characterization After being concatenated into a matrix, it is input into a class-adjustable prototype adaptive network to update all prototypes and support representation vectors;

[0105] S4.4 Predict the labels of the support set using a class-tunable prototype adaptive network and calculate the cross-entropy loss;

[0106] S4.5. Use the stochastic gradient descent algorithm to optimize the plasticity adaptation branch and fusion module in the class-adjustable prototype adaptive network.

[0107] S5. Use the fine-tuned and updated class-adjustable prototype adaptive network to update the prototype classifier and query representation, calculate the correlation between the query representation and the prototype, and thus update the query representation and prototype. All updated prototypes are used as prototypes for the dynamically expanded classifier.

[0108] In this embodiment, the data input to the model in the incremental step added to each category includes 5 incremental audio classes, with only 5 samples in each class.

[0109] In this embodiment, step S5 is as follows:

[0110] S5.1 Input the log-Mel spectrum of the current query set into the feature extractor to obtain the query representation. Assuming the adaptive module of the class-tunable prototype adaptive network processes simultaneously Query representation The adaptive module consists of a stability adaptation branch, a plasticity adaptation branch, and a fusion module. The input variables for both branches of the adaptive module are... , This indicates the number of old categories in the current stage. This represents the total number of all categories in the current stage; the stability adaptation branch and the plasticity adaptation branch affect the input variables. The processing and generation module for The processing is the same, the outputs of the stability adaptation branch and the plastic adaptation branch are respectively and ;

[0111] S5.2, will and The fusion weights are obtained by concatenating the features along the feature dimension and inputting them into a multilayer perceptron consisting of a linear layer, a ReLU activation layer, and a sigmoid function. : ,

[0112] in, It is an activation function used to compress and map any real number to the range of 0 to 1. To represent a linear transformation operation, its form is: ,in For linear layer weight vectors, For bias vector, It is a non-linear activation function;

[0113] S5.3, Utilizing the above-mentioned fusion weights For stability output and plasticity output Perform weighted fusion to obtain the fused output. :

[0114] ;

[0115] S5.4, will It is divided into two parts: the updated query representation and the updated prototype Represent each updated query and the updated prototype The cosine similarity between samples is used to determine the category to which a sample belongs.

[0116] S6. Verify the model's ability to continuously learn new classes while remembering or deleting old classes. The model performs audio classification on all query samples of all categories in the current stage.

[0117] In this embodiment, the model evaluation process in step S6 is as follows:

[0118] S6.1 Extract the log-Mel spectrum of the sample to be tested;

[0119] S6.2 Input the log-Mel spectrum of the sample to be tested into the feature extractor to extract the query representation, concatenate it with the prototypes of all classes in the current stage and input it into the class-adjustable prototype adaptive network to obtain the adjusted query representation.

[0120] S6.3 Calculate the cosine similarity between the adjusted query representation of the test sample and each prototype in the classifier. The category corresponding to the prototype with the highest similarity is the category to which the test sample belongs, thus obtaining the audio classification result.

[0121] This embodiment conducts experiments on two different audio datasets, LS-100 and NSynth-100, to demonstrate the effectiveness of the method of the present invention by calculating the performance of the model on various indicators in multiple incremental stages.

[0122] Accuracy is the ratio of the number of correctly classified samples to the total number of samples. To better characterize the performance of various incremental audio classification methods with small sample sizes, accuracy can be further divided into base class accuracy. Incremental accuracy Accuracy across all classes The process Base class accuracy Observable process The classification performance of the model over the existing base classes is defined as follows: In the formula, Representation Steps The number of base class samples that were correctly classified. Representation Steps The total number of base class samples tested. (Step) Incremental class accuracy Observable process The classification performance of the model for the current incremental class is defined as follows: In the formula, Representation Steps The number of incremental class samples that were correctly classified. Representation Steps The total number of incremental samples tested. (Step) All class accuracy Observable process The classification performance of the model for all existing categories is defined as follows: In the formula, Representation Steps The number of samples from all classes that were correctly classified. Representation Steps The total number of all class samples tested.

[0123] based on , and It allows for better observation of the process. The model's recognition performance on the base class, incremental class (the categories of all incremental steps), and all classes (all learned classes).

[0124] Average accuracy of all stages As an overall evaluation indicator, it is defined as follows:

[0125] ;

[0126] performance degradation rate The variation in classification performance among different methods can be measured, and it is defined as follows:

[0127] ;

[0128] The average accuracy across all classes and the performance degradation rate across all classes are the main metrics for evaluating the performance of different methods. The higher the average accuracy and the lower the performance degradation rate, the better the model's performance.

[0129] As shown in Tables 3 and 4, after applying the class-adjustable prototype adaptive network, the average accuracy of the method of this invention across all classes on the LS-100 and NSynth-100 datasets is as follows. The accuracy rates were 92.62% and 98.73% respectively, both higher than the average accuracy without the network. Furthermore, the method of this invention achieves accuracy for all classes... The values ​​are also lower than those without the network. This is because after using the class-adjustable prototype adaptive network, the model obtains a more discriminative decision boundary by adjusting the prototype position, thus improving the model's classification performance and generalization ability in scenarios with dynamic class changes. In summary, the method of this invention has the ability to model both the increase and decrease of classes simultaneously, maintains stable performance in dynamically changing class sets, and can effectively solve the problem of small-sample variable-class audio classification.

[0130] Table 3. Performance comparison results on the LS-100 dataset

[0131]

[0132] Table 4. Performance comparison results on the NSynth-100 dataset

[0133]

[0134] Example 2

[0135] This embodiment follows the few-sample class variable audio classification method based on prototype adaptive network disclosed in Embodiment 1. In step S5, t-SNE is used to map the query representations before and after the update and the prototype to a two-dimensional space. Experiments are conducted on the LS-100 dataset, and the experimental results are as follows. Figure 6 As shown, numbers 0-14 are category labels. Figure 6 Part (a) shows the prototype and query representation vectors before the class-adjustable prototype adaptive network update. Five audio classes are selected from LS-100 as base classes, with category labels ranging from 0 to 4. The pentagram represents the prototype of the corresponding category, and the dot represents the query representation. Figure 6 Part (b) shows the prototypes updated by the class-adjustable prototype adaptive network when categories are added. The added category labels are 5-9. The pentagram represents the prototype before the update, and the triangle represents the type after the update. When the prototype space is crowded, the class-adjustable prototype adaptive network increases the distance between prototypes within a certain range and adjusts the positional relationship between prototypes. Figure 6 Part (c) shows the prototype after the class-adjustable prototype adaptive network updates when the number of categories is reduced. The reduced category labels are 4 and 9. The triangle represents the prototype before the update, and the pentagon represents the prototype after the update. At this time, categories 4 and 9 have been deleted from the prototype space. The class-adjustable prototype adaptive network adjusts the prototypes of other categories to move towards the vacated space, so that the prototype space can be fully utilized. While increasing the distance between each prototype, it makes each prototype more representative of its own category's samples. Figure 6Part (d) shows the updated prototype of the class-adjustable prototype adaptive network when the number of categories increases again. The added category labels are 10-14. The pentagon represents the prototype before the update, and the rhombus represents the updated prototype. This demonstrates that the class-adjustable prototype adaptive network obtains a more discriminative decision boundary overall by adjusting the prototype position, proving the effectiveness of the invention.

[0136] Example 3

[0137] This embodiment follows the few-sample class-variable audio classification method based on prototype adaptive networks disclosed in Embodiment 1. In step S5, ablation experiments are conducted on each module of the class-adjustable prototype adaptive network to analyze the impact of the stability adaptation branch, the plasticity adaptation branch, and the fusion module on the overall model performance. Experiments are performed on the LS-100 dataset, and the results are shown in Table 5. Using the stability adaptation branch and the plasticity adaptation branch alone can improve model performance. The stability adaptation branch is trained only in the basic stage, while the incremental stage is frozen. The adjusted prototype and query representation are stable, resulting in a significant improvement in the average accuracy for the base class. The plasticity adaptation branch is fine-tuned in the incremental stage, making the adjusted prototype and query representation more plastic, resulting in a significant improvement in the average accuracy for the incremental class. The stability adaptation branch and the plasticity adaptation branch can interact and be balanced through the fusion module. As shown in Table 5, the model performance is best when all three are used simultaneously, proving the effectiveness of this invention.

[0138] Table 5. Impact of Stability Adaptation Branch, Plasticity Adaptation Branch, and Fusion Module on Model Performance

[0139]

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A few-sample class variable audio classification method based on prototype adaptive networks, characterized in that, The audio classification method includes the following steps: S1. Extracting Log-Mel spectrum features: Extracting log-Mel spectrum features from audio samples, wherein the categories of the audio samples include audio categories of the base stage and audio categories of the incremental stage. Hereinafter, the audio categories of the base stage are referred to as the base class, and the audio categories of the incremental stage are referred to as the incremental class. S2. Initialize and train the audio classification model: In the basic stage, supervised learning is used to train the feature extractor, and a pseudo-class incremental training strategy is used to train the class-adjustable prototype adaptive network. The parameters of the feature extractor and the class-adjustable prototype adaptive network are optimized until convergence. After training, the parameters of the feature extractor and the class-adjustable prototype adaptive network are frozen. S3. Construct a base class prototype classifier: Input the log-Mel spectrum features of the base class training samples into the feature extractor to obtain the base class representation, then calculate the mean of the representations of the same class to obtain the prototype of each base class, and then input the base class prototype into the class-adjustable prototype adaptive network for updating. S4. Different processing is performed according to the category change: In the incremental stage where the category increases, the log-Mel spectrum of the incremental class support set is input into the feature extractor to obtain the support representation, and the support representation is input into the generation module of the class adjustable prototype adaptive network to obtain the prototype of the incremental class, while updating the class adjustable prototype adaptive network; In the incremental stage where the category decreases, the prototype of the unnecessary category is deleted. S5. Update the prototype classifier: Input the log-Mel spectrum of the query set of the current stage into the feature extractor to obtain the query representation. Input the query representation and the prototype of the previous stage into the adaptive module of the updated class-adjustable prototype adaptive network, calculate the correlation between the query representation and the prototype, and thus update the query representation and the prototype. All updated prototypes are used as prototypes for the dynamically expanded classifier. S6. Classify the test sample: Input the test sample, calculate the cosine similarity between the updated query representation and the prototypes of each category in the dynamic scalable classifier, and determine the category corresponding to the prototype with the largest cosine similarity as the category to which the test sample belongs, thus obtaining the audio classification result.

2. The few-sample class variable audio classification method based on prototype adaptive network according to claim 1, characterized in that, The log-Mel spectrum feature extraction process in step S1 is as follows: S1.

1. A first-order high-pass filter is used to pre-emphasize the input audio. S1.

2. Perform frame segmentation on the pre-emphasized input audio to obtain several short audio frames; S1.3 Windowing process for short audio frames: Hamming window is used as the window function, and the window function is multiplied by each audio frame to obtain the windowed short audio frame; S1.4 Perform a discrete Fourier transform on the windowed short-time audio frame to obtain the linear spectrum of the corresponding short-time audio frame. Then, use a Mel filter to map the linear spectrum to the Mel scale to obtain the Mel spectrum. Take the logarithm of the Mel spectrum to obtain the logarithmic Mel spectrum feature.

3. The few-sample class variable audio classification method based on prototype adaptive networks according to claim 1, characterized in that, The feature extractor uses a residual convolutional neural network ResNet18, which includes one convolutional layer, eight residual blocks, one average pooling layer, one fully connected layer and one softmax layer connected in sequence. The softmax layer is only used for supervised learning during the training phase of the feature extractor and is removed after training. The prototype classifier consists of multiple prototypes, each corresponding to an audio class. The prototypes of the base class in the prototype classifier are obtained by calculating the average vector of the representation of the same audio class, and the prototypes of the incremental class are generated by the class-adjustable prototype adaptive network. The tunable prototype adaptive network is a self-attention network, comprising a sequentially connected generation module, a parallel-connected stability adaptation branch and a plasticity adaptation branch, and a fusion module. The generation module includes a sequentially connected self-attention layer, a batch normalization layer, and a mean calculation layer. The stability adaptation branch and the plasticity adaptation branch have the same structure, including a sequentially connected self-attention layer and a normalization layer. The fusion module is a multilayer perceptron consisting of a linear layer, a ReLU activation layer, another linear layer, and a sigmoid function.

4. The few-sample class variable audio classification method based on prototype adaptive networks according to claim 1, characterized in that, The process of step S2 is as follows: S2.1 The feature extractor is trained using supervised learning with all samples from the base class training set, and the optimization process employs the cross-entropy loss function. The mathematical expression is: , in, The number of training samples. Let m be the number of categories in the m-th stage, 0 ≤ m ≤ (M-1), where M represents the number of stages. The number of categories in the basic process. Predict samples for the model The true label, Predict samples for the model The probability of belonging to category c; S2.

2. Train the class-adjustable prototype adaptive network using a pseudo-class variable training strategy and cross-entropy loss, and optimize the parameters of the class-adjustable prototype adaptive network until convergence. S2.3 After the audio classification model initialization training is completed, the initial audio classification model parameters θ0 are obtained. Then, the parameters of the feature extractor, class-adjustable prototype adaptive network generation module and stability adaptation branch are frozen.

5. The few-sample class variable audio classification method based on prototype adaptive network according to claim 4, characterized in that, The training process of the class-tunable prototype adaptive network in step S2.2 is as follows: S2.2.1 Sampling two datasets from the base class data and Use sampling mixing factor Synthesize pseudo-incremental classes and assign pseudo-labels to them to construct a pseudo-incremental class dataset. Then, the pseudo-incremental class dataset Classified as pseudo-incremental support set and pseudo-incremental query sets , among which, the mixing factor Values ​​between 0 and 1, and , Indicates shape parameters as The Beta distribution; S2.2.

2. Use the feature extractor trained in step S2.1 to extract the support and query representations of the log-Mel spectrum of pseudo-incremental audio samples; S2.2.

3. Input the support representation of the pseudo-increment class into the generation module of the class-adjustable prototype adaptive network to generate the support prototype of the pseudo-increment class. Then, concatenate the query representation of the pseudo-increment class and the support prototype and input them into the adaptation branch and fusion module of the class-adjustable prototype adaptive network. Finally, the adjusted prototype and query representation are obtained through vector segmentation. S2.2.4 Calculate the cosine similarity between the adjusted query representation and each prototype. By continuously reducing the cross-entropy loss function, use the stochastic gradient descent algorithm to optimize the parameters of the class-adjustable prototype adaptive network until convergence.

6. The few-sample class variable audio classification method based on prototype adaptive network according to claim 1, characterized in that, The calculation formula for the prototype in step S3 is as follows: , In the formula Let c represent the prototype of category c in the m-th stage, where c = 1, 2, ... ; Let represent the log-Mel spectrum of the nth support sample, where n = 1, 2, ..., K, and K represents the number of support samples; This represents the character extraction function. This represents the first parameter used to implement the feature extraction function.

7. The few-sample class variable audio classification method based on prototype adaptive network according to claim 1, characterized in that, In step S4, during the incremental stage of category addition, the data input to the audio classification model includes... There are 1 audio class, each containing 1 audio class. One supporting sample, One query sample.

8. The few-sample class variable audio classification method based on prototype adaptive network according to claim 7, characterized in that, The process of step S4 is as follows: S4.1 Input the log-Mel spectrum of the incremental support set into the feature extractor to obtain the support representation. Supporting characterization The dimension is , Supporting characteristics Constructing a supporting representation set ; Then, the supporting representation set As input to the generative module in a class-tunable prototypic adaptive network, After linear transformation, three matrices are obtained, namely: , and ,in , , and It consists of four trainable first, second, third, and fourth linear transformation layers. and Matrix multiplication and division Then perform a Softmax operation, and combine the matrix output by the Softmax operation with... Matrix multiplication, followed by the fourth linear transformation layer. Later obtained , , In the formula, the upper right corner T represents the transpose of the matrix; then... and After adding element by element and performing layer normalization, we get , ， In the formula, ( ) represents layer normalization operation; finally, along First dimension calculation The average value is used to obtain the prototype of the current stage. ; S4.2, Freeze the parameters of the stability adaptation branch in the adjustable prototype adaptive network; S4.3, The prototype of the current stage Previous stage prototype and supporting characterization After being concatenated into a matrix, it is input into a class-adjustable prototype adaptive network to update all prototypes and support representation vectors; S4.4 Predict the labels of the support set using a class-tunable prototype adaptive network and calculate the cross-entropy loss; S4.

5. Use the stochastic gradient descent algorithm to optimize the plasticity adaptation branch and fusion module in the class-adjustable prototype adaptive network.

9. The few-sample class variable audio classification method based on prototype adaptive network according to claim 1, characterized in that, The process of step S5 is as follows: S5.1 Input the log-Mel spectrum of the current query set into the feature extractor to obtain the query representation. Assuming the adaptive module of the class-tunable prototype adaptive network processes simultaneously Query representation The adaptive module consists of a stability adaptation branch, a plasticity adaptation branch, and a fusion module. The input variables for both branches of the adaptive module are... , This indicates the number of old categories in the current stage. This represents the total number of all categories in the current stage; the stability adaptation branch and the plasticity adaptation branch affect the input variables. The processing and generation module for The processing is the same, the outputs of the stability adaptation branch and the plastic adaptation branch are respectively and ; S5.2, will and The fusion weights are obtained by concatenating the features along the feature dimension and inputting them into a multilayer perceptron consisting of a linear layer, a ReLU activation layer, and a sigmoid function. : ， in, It is an activation function used to compress and map any real number to the range of 0 to 1. To represent a linear transformation operation, its form is: ,in For linear layer weight vectors, For bias vector, It is a non-linear activation function; S5.3, Utilizing the above-mentioned fusion weights For stability output and plasticity output Perform weighted fusion to obtain the fused output. : ； S5.4, will It is divided into two parts: the updated query representation and the updated prototype Represent each updated query and the updated prototype The cosine similarity between samples is used to determine the category to which a sample belongs.

10. The few-sample class variable audio classification method based on prototype adaptive network according to claim 1, characterized in that, The process of step S6 is as follows: S6.1 Extract the log-Mel spectrum of the sample to be tested; S6.2 Input the log-Mel spectrum of the sample to be tested into the trained feature extractor to extract the query representation, concatenate it with the prototypes of all classes in the current stage and input it into the updated class-adjustable prototype adaptive network to obtain the adjusted query representation. S6.3 Calculate the cosine similarity between the adjusted query representation of the test sample and each prototype in the classifier. The category corresponding to the prototype with the highest similarity is the category to which the test sample belongs, thus obtaining the audio classification result.

Images