A Fully Few-Sample Incremental Audio Classification Method Based on Prototype Classifier

Abstract

This invention discloses a fully few-shot incremental audio classification method based on a prototype classifier, comprising the following steps: extracting a log-Mel spectrum from the input audio samples; initializing a self-attention convolutional neural network, including a representation extractor, a prototype classifier, and a prototype adaptation module; in the basic stage, training the representation extractor and the prototype adaptation module using few-shot learning and pseudo-incremental context training methods respectively; extracting representations from the audio category training samples in the basic stage and calculating the mean of the representations of the same category as prototypes to update the prototype classifier; in the incremental stage, using the prototype adaptation module to first obtain the prototypes of each incremental class, then adaptively adjusting all prototypes, and then updating the prototype classifier; during testing, evaluating the model performance using test samples of all known categories. This invention employs few-shot learning to train the model in all stages, effectively solving the problems of overfitting to incremental classes and forgetting old classes with fewer training samples.

Description

Technical Field

[0001] This invention belongs to the field of audio signal processing, specifically relating to a fully small-sample incremental audio classification method based on a prototype classifier. Background Technology

[0002] Audio classification aims to categorize various sounds in natural acoustic environments. It serves as a crucial technological foundation for fields such as intelligent assisted driving, healthcare, wildlife conservation, ecological environment monitoring, smart homes, wearable monitoring, equipment condition monitoring, and intelligent robotics. Currently, most intelligent audio terminals can only recognize a limited and fixed number of audio categories. However, in natural acoustic environments, the audio categories encountered by intelligent audio terminals are generally variable, and many application scenarios suffer from a scarcity of samples.

[0003] Current audio classification methods can be categorized into four types. Large-sample audio classification methods use a large number of training samples to build the model, enabling them to identify predefined audio categories but not incremental categories. Few-sample audio classification methods can learn incremental categories based on a small number of samples but cannot remember old categories. Category-incremental audio classification methods can continuously learn incremental categories without forgetting old categories, but both the base class and increment stages require a large number of training samples. Few-sample category-incremental audio classification methods can continuously learn incremental categories based on a small number of training samples and without forgetting old categories, but the base class stage still requires a large number of training samples. Therefore, current audio classification methods use standard supervised learning with large samples for the basic training stage. However, in many practical applications, each audio category in the base class and increment stages has only a small number of training samples, i.e., completely few samples. When both the base class and increment stages use few-sample learning methods for model training, this type of audio classification method is called completely few-sample category-incremental audio classification.

[0004] Because the base class has only a small number of training samples, the model's insufficient training leads to problems such as poor representation ability of the representation extractor and weak memory of old classes. To address these issues, there is an urgent need to propose a fully small-sample incremental audio classification method based on a prototype classifier. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned deficiencies in the prior art and provide a fully few-shot incremental audio classification method based on a prototype classifier. By organically combining few-shot learning, attention mechanisms, and prototype networks, this method enables intelligent audio terminals to continuously learn incremental classes without forgetting old classes under fully few-shot training conditions.

[0006] The objective of this invention can be achieved by adopting the following technical solutions:

[0007] A fully few-sample incremental audio classification method based on a prototype classifier, the audio classification method comprising the following steps:

[0008] S1. Extract the log-Mel spectrum from the input audio samples, wherein the input audio samples are divided into basic segment audio categories and multiple incremental segment audio categories with disjoint categories. The basic segment audio categories are referred to as base classes, and the incremental segment audio categories are referred to as incremental classes.

[0009] S2. Initialize the self-attention convolutional neural network, which includes a decoupled representation extractor, prototype classifier, and prototype adaptive module.

[0010] S3. In the basic stage, according to the training data usage strategy, the representation extractor and the prototype adaptive module are trained respectively using the few-shot learning method and the cross-entropy loss function. The parameters of the representation extractor and the prototype adaptive module are optimized until convergence. After training, the parameters of the representation extractor and the prototype adaptive module are fixed.

[0011] S4. Input the log-Mel spectrum features of the base class audio samples into the representation extractor to obtain the base class representation, then calculate the mean value of the representations of the same class to obtain different base class prototypes, and then input the base class prototypes into the prototype classifier for updating; wherein, the prototype classifier update is to replace all prototypes in the initial prototype classifier with the base class prototypes.

[0012] S5. In the incremental stage, the log-Mel spectrum features of the incremental audio samples are input into the representation extractor to obtain the incremental representation. The representation is then concatenated with all the prototypes of the prototype classifier in the previous incremental stage and input into the prototype adaptive module for prototype adaptive adjustment to obtain the adjusted base class prototype and incremental class prototype. The adjusted base class prototype and incremental class prototype are then input into the prototype classifier for update.

[0013] S6. Extract the log-Mel spectrum from the test audio sample, input it into the trained self-attention convolutional neural network, obtain the representation through the representation extractor, calculate the cosine distance between the representation and each prototype in the prototype classifier, and the category corresponding to the prototype with the smallest distance is the category to which the test audio sample belongs, thus obtaining the audio classification result.

[0014] Furthermore, the extraction process of the log-Mel spectrum in step S1 is as follows:

[0015] S1.1. A first-order high-pass filter is used to pre-emphasize the audio samples. The transfer function of the filter is H(z) = 1 - αz. -1 , where α represents the coefficient of the high-pass filter, with a value ranging from 0 to 1. Pre-emphasis can improve audio quality and extract a purer original audio signal.

[0016] S1.2. Divide the pre-emphasized audio samples into short-time frames to ensure that the input signal is stable, and then multiply them with a window function to obtain windowed short-time audio frames to avoid the Gibbs effect.

[0017] S1.3 Perform a discrete Fourier transform on the windowed short-time audio frame to obtain the linear spectrum of the frame;

[0018] S1.4. A Mel filter is used to convert the linear spectrum into a Mel spectrum, and then the logarithm of the Mel spectrum is taken to obtain the logarithmic Mel spectrum.

[0019] Furthermore, the self-attention convolutional neural network includes a representation extractor, a prototype classifier, and a prototype adaptation module.

[0020] The representation extractor is the backbone of the residual self-attention convolutional neural network. The structure of the representation extractor includes a convolutional layer, four residual layers, an average pooling layer, and a fully connected layer connected in sequence. The fully connected layer is only used for training the basic components.

[0021] The convolutional layer consists of a sequentially connected convolutional layer (64 3×3 convolutional kernels), a batch normalization layer, a ReLU non-linear activation layer, and a max pooling layer. The convolutional layer transforms the input feature map into a feature map, gradually reducing the spatial resolution and introducing non-linearity, while preserving important features for subsequent layers. This helps the network better understand the input feature map and extract appropriate features.

[0022] The four residual layers have different parameters. Each residual layer consists of two residual blocks, and each residual block includes two convolutional layers and a bitwise addition operation. A ReLU non-linear activation layer is applied after each convolutional layer. By progressively extracting semantic features and increasing the network depth through these four residual layers, the model can better learn and represent the features of the audio.

[0023] Average pooling layers are used to perform average pooling operations, reducing the dimensionality and computational complexity of feature maps, which helps to reduce the number of parameters and improve the generalization ability of the model.

[0024] The fully connected layer linearly maps the feature vectors after the max pooling operation to the final output value.

[0025] The prototype adaptation module includes an attention-based prototype generation submodule and a prototype query representation adaptation submodule. The attention-based prototype generation module generates representative prototypes; its input is the incremental class support representation, and its output is the incremental class prototype. The prototype query representation adaptation module updates all prototypes and query representations; its input is the old class prototype, the incremental class prototype, and the query representation, and its output is the updated prototype and query representation.

[0026] The attention prototype generation module includes a self-attention layer, a batch normalization layer, and a mean calculation layer connected in sequence. The self-attention layer is used to obtain the interrelationships between incremental class support representations, which helps to generate more representative incremental class prototypes. The batch normalization layer is used to increase the stability of training, improve generalization ability, maintain feature diversity, and help gradient propagation. The mean calculation layer is used to calculate the mean vector of the incremental class support representations to obtain the incremental class prototype.

[0027] The prototype query representation adaptation module includes a self-attention layer, a normalization layer, a vector segmentation layer, and a cosine similarity calculation layer connected in sequence. The self-attention layer is used to obtain the relationship between the old class prototype, the incremental class prototype, and the incremental class query representation, so that the updated prototype is compact within the class and dispersed between classes. The batch normalization layer is used to increase the stability of training, improve the generalization ability, maintain feature diversity, and help gradient propagation. The vector segmentation layer divides the output of the batch normalization layer into the adjusted prototype and query representation.

[0028] Furthermore, step S3 is as follows:

[0029] S3.1 Using a subset of samples from the base class audio samples, a few-shot learning method and cross-entropy loss function are used to pre-train the representation extractor. The pre-trained representation extractor is used to simulate the representation extractor in pseudo-incremental context training, in preparation for step S3.2.

[0030] S3.2 Using all samples in the base class audio samples, the prototype adaptive module is trained using the small sample pseudo-incremental context training method and the cross-entropy loss function. The small sample pseudo-incremental context training method can simulate the incremental recognition task and train the model's incremental recognition capability.

[0031] S3.3 Using all samples in the base class audio samples, repeat step S3.1 to retrain the representation extractor. The parameters of the representation extractor remain unchanged in all incremental steps and are used to extract the depth representation of the incremental audio samples.

[0032] Furthermore, the characterization extractor pre-training process in step S3.1 is as follows:

[0033] S3.1.1, Transfer the base class audio samples Divide into two subsets with non-overlapping categories and use Pre-trained representation extractor, Used to simulate base class audio samples in incremental tasks;

[0034] S3.1.2, The representation extractor is trained using a few-sample learning method. The model is randomly divided into multiple subsets, each containing N classes with K support samples and K query samples in each class. The K support samples constitute the pre-training support set, and the K query samples constitute the pre-training query set. Through multiple rounds of training on different few-shot tasks, the model is helped to adapt to new few-shot tasks more quickly.

[0035] S3.1.3 Input the log-Mel spectrum of audio samples in the pre-trained support set and the pre-trained query set into the representation extractor to obtain the support representation and the query representation. Calculate the mean vector of the support representation of different categories of audio samples in the support set and use it as the prototype of different categories of audio samples in the support set.

[0036] S3.1.4 Calculate the distance between the query representation and each prototype, and iteratively update the representation extractor parameters by minimizing the cross-entropy loss function to obtain the pre-trained representation extractor.

[0037] Furthermore, the prototype calculation formula in step S3.1 is as follows:

[0038]

[0039] In the formula p n This represents the prototype of category n, where n = 1, 2, ..., N. Let f represent the log-Mel spectrum of the i-th support sample, i = 1, 2, ..., K. α (·) represents the extraction function, and α represents the parameters of the extractor;

[0040] The cross-entropy loss function used to train the representation extractor in step S3.1 is defined as follows:

[0041]

[0042] In the formula Let y represent the log-Mel spectrum of the i-th query sample. Category tags, express The representation of p y and p n Let represent the prototypes of categories y and n, respectively. This represents the cosine distance between the query representation and the prototype.

[0043] Furthermore, the prototype adaptive module training process in step S3.2 is as follows:

[0044] S3.2.1, Set the base class audio sample set Divide into two subsets and from Randomly select N classes, with K audio samples from each class, as the first support set S1. Then, randomly select K audio samples from the remaining samples of the selected N classes as the first query set Q1. Use the same method to... The second support set S2 and the second query set Q2 are obtained by randomly selecting audio samples. S1 and S2 constitute the third support set S3, and Q1 and Q2 constitute the third query set Q3. The first support set S1 and the second support set S2 are used to simulate the base class support set and the incremental class support set, respectively. The first query set Q1 and the second query set Q2 are used to simulate the base class query set and the incremental query set, respectively. The third support set S3 and the third query set Q3 constitute the pseudo-incremental training task.

[0045] S3.2.2 Extract the log-Mel spectrum of audio samples from the third support set S3 and the query set Q3;

[0046] S3.2.3. Use the representation extractor pre-trained in step S3.1 to extract the support and query representations of the log-Mel spectra of audio samples in the third support and query sets.

[0047] S3.2.4. Input the support representations of the samples in the third support set S3 into the attention prototype generation module of the prototype adaptive network. First, X1, X2, and X3 are obtained by passing them through three trainable linear transformation matrices ψ1, ψ2, and ψ3 respectively. Then, X1 and X2 are passed through matrix multiplication, scaling, and sofmax layers in sequence and then multiplied by X3. After passing through the trainable linear transformation matrix ψ4, X′ is obtained. X′ is added element-wise with the input support representation vector, then normalized, and finally the mean is calculated to obtain the support prototype.

[0048] S3.2.5. The query representation and supporting prototype of the samples in the third query set Q3 are concatenated as input vectors and input into the prototype query representation adaptation module of the prototype adaptive network. After passing through three trainable linear transformation matrices ψ5, ψ6, and ψ7 respectively, X4, X5, and X6 are obtained. Then, X4 and X5 are passed through matrix multiplication, scaling, and sofmax layers in sequence and then multiplied by X6. After passing through the trainable linear transformation matrix ψ8, X″ is obtained. X″ is added element-wise with the input vector and then normalized. Finally, the adjusted prototype and query representation are obtained through vector segmentation.

[0049] S3.2.6 Calculate the cosine distance between the adjusted query representation and each prototype. Optimize the parameters ψ1 to ψ8 of the prototype adaptive module by continuously reducing the cross-entropy loss function until convergence.

[0050] Furthermore, the prototype adaptive adjustment and prototype classifier update process in step S5 is as follows:

[0051] S5.1 Input the log-Mel spectrum features of the incremental audio samples into the representation extractor to obtain the incremental representation, and divide the representations of the same category into incremental support representations and incremental query representations.

[0052] S5.2 The incremental class support representation is input into the attention prototype generation submodule of the prototype adaptive module. First, self-attention calculation is performed, then the calculation result is added to the input support representation vector element-wise, then normalized, and finally the mean is calculated to obtain the incremental class prototype. The self-attention calculation is used to obtain the mutual relationship between incremental class support representations, which helps to obtain a more representative incremental class prototype.

[0053] S5.3. The incremental class query representation, the incremental class prototype, and all prototypes in the prototype classifier of the previous step are concatenated and input into the prototype query representation adaptation submodule of the prototype adaptation module. Self-attention calculation is performed first, and then the calculation result is added element-wise to the input concatenated vector. Normalization is then performed, and finally, the adjusted base class prototype and incremental class prototype are obtained through vector segmentation. The self-attention calculation is used to obtain the relationship between the old class prototype, the incremental class prototype, and the incremental class query representation, so that the adjusted prototype is more distinguishable.

[0054] S5.4 Replace all prototypes in the prototype classifier with the adjusted base class prototype and incremental class prototype to complete the prototype classifier update.

[0055] Furthermore, the self-attention calculation formula in step S5 is as follows:

[0056]

[0057] In the formula, X′ represents the self-attention calculation result, X1, X2, and X3 are obtained by applying three trainable linear transformation matrices ψ1, ψ2, and ψ3 to the input vector, respectively, and D represents the dimension of the input vector. ψ4 is a scaling factor, T represents the matrix transpose operation, and ψ4 is also a trainable linear transformation matrix. ψ1, ψ2, ψ3 and ψ4 are obtained by training in step S3.2.

[0058] Furthermore, the audio classification test process in step S6 is as follows:

[0059] S6.1 Extract the log-Mel spectrum features of the test audio samples;

[0060] S6.2 Input the log-Mel spectrum features of the test audio samples into the trained self-attention convolutional neural network, and obtain the representation through the representation extractor;

[0061] S6.3 Calculate the cosine distance between each prototype in the representation and prototype classifier. The category corresponding to the prototype with the smallest distance is the category to which the test audio sample belongs, and the audio classification result is obtained.

[0062] The present invention has the following advantages and effects compared with the prior art:

[0063] (1) This invention proposes a fully small-sample incremental audio classification method that uses small-sample training for both the base class stage and the incremental stage, which can solve the problem of insufficient base class training samples.

[0064] (2) The present invention decouples the characterization extractor and the classifier, and uses a few-sample learning training method in the base class to improve the generalization ability of the characterization extractor.

[0065] (3) In the incremental stage, this invention combines the attention mechanism and uses the prototype adaptive module to adaptively adjust the old class prototype and the old class prototype, thereby improving the distinguishability of various prototypes.

[0066] (4) Experimental results show that the method of the present invention achieves high average accuracy and low forgetting degree on NSynth-100 and LS-100 datasets, respectively, effectively solving the problems of model overfitting to incremental classes and forgetting old classes. Attached Figure Description

[0067] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0068] Figure 1 This is a flowchart of a fully small-sample incremental audio classification method based on a prototype classifier according to an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of the characterization extractor structure according to an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the prototype adaptive module structure according to an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of the attention prototype generation module according to an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the prototype query representation adaptation module in an embodiment of the present invention;

[0073] Figure 6 This is a schematic diagram of the training of the characterization extractor according to an embodiment of the present invention;

[0074] Figure 7This is a schematic diagram of the prototype adaptive module training in an embodiment of the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0076] Example 1

[0077] Figure 1 This is a flowchart illustrating a fully small-sample incremental audio classification method based on a prototype classifier, according to an embodiment of the present invention. Figure 1 As shown, a fully few-sample incremental audio classification method based on a prototype classifier includes the following steps:

[0078] S1. Extract the log-Mel spectrum from the input audio samples, wherein the input audio samples are divided into basic segment audio categories and multiple incremental segment audio categories with disjoint categories. The basic segment audio categories are referred to as base classes, and the incremental segment audio categories are referred to as incremental classes.

[0079] In this embodiment, step S1, extracting the log-Mel spectrum, specifically includes the following steps:

[0080] S1.1. A first-order high-pass filter is used to pre-emphasize the audio samples. The transfer function of the filter is H(z) = 1 - αz. -1 , where α represents the coefficient of the high-pass filter, with a value of 0.95;

[0081] S1.2. Divide the pre-emphasized audio samples into short-time frames, and then multiply them with a Hamming window to obtain windowed short-time audio frames.

[0082] S1.3 Perform a discrete Fourier transform on the windowed short-time audio frame to obtain the linear spectrum of the frame;

[0083] S1.4. A Mel filter is used to convert the linear spectrum into a Mel spectrum, and then the logarithm of the Mel spectrum is taken to obtain the logarithmic Mel spectrum.

[0084] In this embodiment, audio datasets NSynth-100 and LS-100 are constructed from NSynth and LibriSpeech, respectively. NSynth is a dataset of musical instrument sounds containing 305,979 audio samples, which are non-uniformly distributed across 1,006 instrument categories. Each audio sample in the dataset has a unique pitch, timbre, and amplitude. LibriSpeech is a corpus containing approximately 1,000 hours of audio recordings by 2,484 speakers.

[0085] Tables 1 and 2 provide detailed information for the NSynth-100 and LS-100 datasets, respectively.

[0086] As shown in Tables 1 and 2, both the NSynth-100 and LS-100 datasets are divided into a base class dataset D0 with disjoint classes and an incremental class dataset D10. i (1≤i≤(M-1)). D0 contains the basic training dataset. and basic validation dataset D i Includes incremental training dataset and incremental validation dataset Used for training the representation extractor and the prototype adaptive module. Used to update the prototype of the classifier.

[0087] Table 1. Detailed information about the NSynth-100 dataset

[0088]

[0089] Table 2. Detailed information about the LS-100 dataset

[0090]

[0091] S2. Initialize the self-attention convolutional neural network, which includes a decoupled representation extractor, prototype classifier, and prototype adaptive module.

[0092] In this embodiment, the parameters of the self-attention convolutional neural network are randomly initialized, the representation extractor is the backbone of the residual self-attention convolutional neural network ResNet18, the prototype classifier is initialized as a W×512 dimensional random vector, where W is the number of categories in the basic link, and the prototype adaptive module is a self-attention module. Figure 2 and Figure 3 These are schematic diagrams of the characterization extractor and prototype adaptive module structures, respectively, according to embodiments of the present invention.

[0093] like Figure 2As shown, the structure of the characterization extractor includes a sequentially connected convolutional layer, four residual layers, an average pooling layer, and a fully connected layer, wherein the fully connected layer is only used for training the basic steps.

[0094] The input convolutional layer consists of a sequentially connected convolutional layer (64 3×3 convolutional kernels), a batch normalization layer, a ReLU non-linear activation layer, and a max pooling layer.

[0095] The four residual layers have different parameters. Each residual layer consists of two residual blocks. Each residual block includes two convolutional layers and a bitwise addition operation. A ReLU nonlinear activation layer is applied after each convolutional layer.

[0096] The average pooling layer is used to perform the average pooling operation.

[0097] The fully connected layer linearly maps the feature vectors after the max pooling operation to the final output value.

[0098] like Figure 3 As shown, the prototype adaptation module is a self-attention module, which includes a sequentially connected attention prototype generation submodule and a prototype query representation adaptation submodule.

[0099] (1) Prototype Adaptive Module

[0100] like Figure 3 As shown, the prototype adaptation module includes an attention-based prototype generation submodule and a prototype query representation adaptation submodule. The attention-based prototype generation module generates representative prototypes; its input is the incremental class support representation, and its output is the incremental class prototype. The prototype query representation adaptation module updates all prototypes and query representations; its input is the old class prototype, the incremental class prototype, and the query representation, and its output is the updated prototype and query representation.

[0101] (2) Attention Prototype Generation Module

[0102] like Figure 4 As shown, the attention prototype generation module includes a self-attention layer, a batch normalization layer, and a mean calculation layer connected in sequence. The self-attention layer is used to obtain the interrelationships between the features of the incremental class support set samples, which helps to generate more representative incremental class prototypes. The batch normalization layer is used to increase the stability of training, improve generalization ability, maintain feature diversity, and help gradient propagation. The mean calculation layer is used to calculate the mean vector of the incremental class support representation to obtain the incremental class prototype.

[0103] (3) Prototype query representation adaptation module

[0104] like Figure 5As shown, the prototype query representation adaptation module consists of a self-attention layer, a normalization layer, a vector segmentation layer, and a cosine similarity calculation layer connected in sequence. The self-attention layer is used to obtain the relationship between the old class prototype, the incremental class prototype, and the incremental class query representation, so that the updated prototype is compact within the class and dispersed between classes. The batch normalization layer is used to increase the stability of training, improve the generalization ability, maintain feature diversity, and help gradient propagation. The vector segmentation layer divides the output of the batch normalization layer into the updated prototype and the query representation.

[0105] S3. In the basic stage, according to the training data usage strategy, the representation extractor and the prototype adaptive module are trained respectively using the few-shot learning method and the cross-entropy loss function. The parameters of the representation extractor and the prototype adaptive module are optimized until convergence. After training, the parameters of the representation extractor and the prototype adaptive module are fixed.

[0106] In this embodiment, the training steps for the basic components are as follows:

[0107] S3.1. Using a subset of samples from the base class audio samples, a few-shot learning method and cross-entropy loss function are used to pre-train the representation extractor.

[0108] S3.1.1, Transfer the base class audio samples Divide into two subsets with non-overlapping categories and use Pre-trained representation extractor;

[0109] S3.1.2, The representation extractor is trained using a few-sample learning method. The dataset is randomly divided into multiple subsets, each containing 5 classes with 5 support samples and 5 query samples per class. The 5 support samples constitute the pre-trained support set, and the 5 query samples constitute the pre-trained query set.

[0110] S3.1.3 Input the log-Mel spectrum of audio samples in the pre-trained support set and the pre-trained query set into the representation extractor to obtain the support representation and the query representation. Calculate the mean vector of the support representation of different categories of audio samples in the support set and use it as the prototype of different categories of audio samples in the support set.

[0111] The prototype calculation formula is as follows:

[0112] In the formula p n This represents the prototype of category n, where n = 1, 2, ..., N. Let f represent the log-Mel spectrum of the i-th support sample, i = 1, 2, ..., K. α (·) represents the extraction function, and α represents the parameters of the extractor;

[0113] S3.1.4 Calculate the distance between the query representation and each prototype, and iteratively update the representation extractor parameters by minimizing the cross-entropy loss function to obtain the pre-trained representation extractor.

[0114] The cross-entropy loss function is defined as follows:

[0115]

[0116] In the formula Let y represent the log-Mel spectrum of the i-th query sample. Category tags, express The representation of p y and p n Let represent the prototypes of categories y and n, respectively. This represents the cosine distance between the query representation and the prototype.

[0117] S3.2. Using all samples in the base class audio samples, train the prototype adaptive module using the small sample pseudo-incremental context training method and cross-entropy loss function.

[0118] S3.2.1, Set the base class audio sample set Divide into two subsets and from Randomly select N classes, with K audio samples from each class, as the first support set S1. Then, randomly select K audio samples from the remaining samples of the selected N classes as the first query set Q1. Use the same method to... The second support set S2 and the second query set Q2 are obtained by randomly selecting audio samples. S1 and S2 constitute the third support set S3, and Q1 and Q2 constitute the third query set Q3.

[0119] S3.2.2 Extract the log-Mel spectrum of audio samples from the third support set S3 and the query set Q3;

[0120] S3.2.3. Use the representation extractor pre-trained in step S3.1 to extract the support and query representations of the log-Mel spectra of audio samples in the third support and query sets.

[0121] S3.2.4. Input the support representations of the samples in the third support set S3 into the attention prototype generation module of the prototype adaptive network. First, X1, X2, and X3 are obtained by passing them through three trainable linear transformation matrices ψ1, ψ2, and ψ3 respectively. Then, X1 and X2 are passed through matrix multiplication, scaling, and sofmax layers in sequence and then multiplied by X3. After passing through the trainable linear transformation matrix ψ4, X′ is obtained. X′ is added element-wise with the input support representation vector, then normalized, and finally the mean is calculated to obtain the support prototype.

[0122] S3.2.5. The query representation and supporting prototype of the samples in the third query set Q3 are concatenated as input vectors and input into the prototype query representation adaptation module of the prototype adaptive network. They are passed through three trainable linear transformation matrices ψ5, ψ6, and ψ7 to obtain X4, X5, and X6 respectively. Then, X4 and X5 are passed through matrix multiplication, scaling, and sofmax layers in sequence and then multiplied by X6. After passing through the trainable linear transformation matrix ψ8, X″ is obtained. X″ is added element-wise with the input vector and then normalized. Finally, the vector is split to obtain the adjusted prototype and query representation.

[0123] S3.2.6 Calculate the cosine distance between the adjusted query representation and each prototype. Optimize the parameters ψ1 to ψ8 of the prototype adaptive module by continuously reducing the cross-entropy loss function until convergence.

[0124] S3.3 Using all samples in the base class audio samples, repeat step S3.1 to retrain the representation extractor. The parameters of the representation extractor remain unchanged in all incremental steps and are used to extract the depth representation of the incremental audio samples.

[0125] S4. Input the log-Mel spectrum features of the base class audio samples into the representation extractor to obtain the base class representation, and then calculate the mean value of the representations of the same class to obtain different base class prototypes p0, p1, ..., p w-1 Then, input the base class prototype into the prototype classifier, and replace the random vector in the initialized prototype classifier to obtain the base class prototype classifier C0 = (p0, p1, ..., p w-1 );

[0126] S5. In the incremental stage, the log-Mel spectrum features of the incremental audio samples are input into the representation extractor to obtain the incremental representation. The representation is then concatenated with all the base class prototypes of the prototype classifier in the previous incremental stage as an input vector and input into the prototype adaptive module for prototype adaptive adjustment. The adjusted base class prototype and incremental class prototype are then input into the prototype classifier for update.

[0127] In this embodiment, the specific steps for prototype adaptive adjustment and prototype classifier update are as follows:

[0128] S5.1 Input the log-Mel spectrum features of the incremental audio samples into the representation extractor to obtain the incremental representation, and divide the representations of the same category into incremental support representations f. α (x s ) and incremental query representation f α (x q );

[0129] S5.2, Incremental class support characterization f α (x s The attention prototype generation submodule of the input prototype adaptive module first performs self-attention calculation, and then compares the calculation result X′ with the input support representation vector representation f. α (x s ) Perform element-wise addition, then normalize, and finally calculate the average to obtain the incremental class prototype p. novel ;

[0130] The formula for calculating self-attention is as follows:

[0131] In the formula, X′ is the result of self-attention calculation, and X1, X2, and X3 are incremental class support representations f. α (x s The vectors are obtained by applying three trainable linear transformation matrices ψ1, ψ2, and ψ3, respectively, where D is the dimension of the input vector (512). ψ4 is a scaling factor, T represents the matrix transpose operation, and ψ4 is also a trainable linear transformation matrix. ψ1, ψ2, ψ3 and ψ4 are obtained by training in step S3.2.

[0132] S5.3, Represent incremental query f α (x q ), Incremental class prototype p novel And all prototypes p0, p1, ..., p in the prototype classifier of the previous stage w-1 The prototype query representation adaptation submodule, which uses the concatenated input vector as input to the prototype adaptation module, first performs self-attention calculation, then adds the calculation result to the input vector element-wise, performs normalization, and finally obtains the adjusted prototypes p′0, p′1, ..., p′ after vector segmentation. w-1 ,p′ novel ;

[0133] The formula for calculating self-attention is as follows:

[0134] In the formula, X″ represents the result of self-attention calculation, and X4, X5, and X6 are the values ​​of [p0, p1, ..., p6]. w-1 ,p novel ,f α (xq The vectors were obtained by applying three trainable linear transformation matrices ψ5, ψ6, and ψ7, respectively, where D is the dimension of the input vector (512). ψ8 is a scaling factor, T represents the matrix transpose operation, and ψ8 is also a trainable linear transformation matrix. ψ5, ψ6, ψ7 and ψ8 are obtained by training in step S3.2.

[0135] S5.4 Replace all prototypes in the prototype classifier with the adjusted base class prototype and incremental class prototype to obtain the updated prototype classifier C. m =(p′0,p′1,……,[′ w-1 ,p′ novel ).

[0136] S6. Extract the log-Mel spectrum from the test audio sample, input it into the trained self-attention convolutional neural network, obtain the representation through the representation extractor, calculate the cosine distance between the representation and each prototype in the prototype classifier, and the category corresponding to the prototype with the smallest distance is the category to which the test audio sample belongs, thus obtaining the audio classification result.

[0137] In this embodiment, the specific steps for testing audio classification are as follows:

[0138] S6.1 Extract the log-Mel spectrum features of the test audio samples;

[0139] S6.2 Input the log-Mel spectrum features of the test audio samples into the trained self-attention convolutional neural network, and obtain the representation through the representation extractor;

[0140] S6.3 Calculate the cosine distance between each prototype in the representation and prototype classifier. The category corresponding to the prototype with the smallest distance is the category to which the test audio sample belongs, and the audio classification result is obtained.

[0141] This embodiment compares the method with the CEC method (C. Zhang, N. Song, G. Lin, Y. Zheng, P. Pan and Y. Xu, "Few-Shot Incremental Learning with Continually Evolved Classifiers," 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, pp. 12450-12459, doi:10.1109 / CVPR46437.2021.01227) on two different audio datasets, NSynth-100 and LS-100. The CEC method uses a graph model to propagate contextual information between classifiers and update prototypes, thereby achieving continuous incremental class recognition. The effectiveness of the method is demonstrated by calculating the average accuracy and forgetting level of the model across all classes in multiple incremental stages.

[0142] Among them, the classification accuracy A of the m-th step m The definition is as follows:

[0143] In the formula, H′ represents the number of samples in all classes that are correctly classified, and H represents the total number of samples in all classes.

[0144] The average precision (AA) of the model across all classes is defined as follows:

[0145] In the formula, M represents the number of steps.

[0146] The forgetting level (PD) of the model across all classes is defined as follows: PD = A0 - A M-1 ,

[0147] In the formula, M represents the number of steps, A0 represents the classification precision of the base class steps, and A M-1 This indicates the classification accuracy of the final step.

[0148] As shown in Tables 3 and 4, the method of this invention achieved average accuracies of 85.1% and 78.2% on the NSynth-100 and LS-100 datasets, respectively, both higher than the corresponding values ​​of the CEC method. The forgetting rates of the method of this invention on the NSynth-100 and LS-100 datasets were 14.28% and 11.23%, respectively, both lower than the corresponding values ​​of the CEC method. The main reason why the method of this invention outperforms the CEC method is that, in the base class stage, the method of this invention uses different datasets for small-sample pre-training and updating of the representation extractor, improving the generalization of the representation extractor; in the incremental stage, the method of this invention uses an attention prototype generation module and a prototype query representation adaptation module to adaptively adjust and update the prototype classifier, improving the discriminative power of various prototypes. In summary, the method proposed in this invention can effectively solve the problems of model overfitting to the incremental class and forgetting of the old class.

[0149] Table 3. Performance comparison results on the NSynth-100 dataset

[0150]

[0151] Table 4. Performance comparison results on the LS-100 dataset

[0152]

[0153] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A fully small-sample incremental audio classification method based on a prototype classifier, characterized in that, Includes the following steps: S1. Extract the log-Mel spectrum from the input audio samples, wherein the input audio samples are divided into basic segment audio categories and multiple incremental segment audio categories with disjoint categories. The basic segment audio categories are referred to as base classes, and the incremental segment audio categories are referred to as incremental classes. S2. Initialize the self-attention convolutional neural network, which includes a decoupled representation extractor, prototype classifier, and prototype adaptive module. S3. In the basic stage, according to the training data usage strategy, the representation extractor and the prototype adaptive module are trained respectively using the few-shot learning method and the cross-entropy loss function. The parameters of the representation extractor and the prototype adaptive module are optimized until convergence. After training, the parameters of the representation extractor and the prototype adaptive module are fixed. S4. Input the log-Mel spectrum features of the base class audio samples into the representation extractor to obtain the base class representation, then calculate the mean value of the representations of the same class to obtain different base class prototypes, and then input the base class prototypes into the prototype classifier for updating; wherein, the prototype classifier update is to replace all prototypes in the initial prototype classifier with the base class prototypes. S5. In the incremental stage, the log-Mel spectrum features of the incremental audio samples are input into the representation extractor to obtain the incremental representation. The representation is then concatenated with all the prototypes of the prototype classifier in the previous incremental stage and input into the prototype adaptive module for prototype adaptive adjustment to obtain the adjusted base class prototype and incremental class prototype. The adjusted base class prototype and incremental class prototype are then input into the prototype classifier for update. S6. Extract the log-Mel spectrum from the test audio sample, input it into the trained self-attention convolutional neural network, obtain the representation through the representation extractor, calculate the cosine distance between the representation and each prototype in the prototype classifier, and the category corresponding to the prototype with the smallest distance is the category to which the test audio sample belongs, thus obtaining the audio classification result.

2. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 1, characterized in that, The extraction process of the log-Mel spectrum in step S1 is as follows: S1.

1. A first-order high-pass filter is used to pre-emphasize the audio samples. The transfer function of the filter is: ,in These represent the coefficients of the high-pass filter, with values ​​ranging from 0 to 1. S1.

2. Divide the pre-emphasized audio samples into short-time frames, and then multiply them with a window function to obtain windowed short-time audio frames; S1.3 Perform a discrete Fourier transform on the windowed short-time audio frame to obtain the linear spectrum of the frame; S1.

4. A Mel filter is used to convert the linear spectrum into a Mel spectrum, and then the logarithm of the Mel spectrum is taken to obtain the logarithmic Mel spectrum.

3. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 1, characterized in that, The representation extractor consists of a sequentially connected convolutional layer, four residual layers, an average pooling layer, and a fully connected layer, where the fully connected layer is used only for training the basic steps. The prototype classifier is composed of prototype vectors for each category, where each prototype vector is the center vector of the representation of each category's samples. The prototype adaptation module is a self-attention module, comprising a sequentially connected attention prototype generation submodule and a prototype query representation adaptation submodule. The attention prototype generation submodule includes a sequentially connected self-attention layer, a batch normalization layer, and a mean calculation layer. The prototype query representation adaptation module consists of a sequentially connected self-attention layer, a normalization layer, and a vector segmentation layer.

4. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 1, characterized in that, The process of step S3 is as follows: S3.

1. Using a subset of samples from the base class audio samples, a few-shot learning method and cross-entropy loss function are used to pre-train the representation extractor. S3.

2. Using all samples in the base class audio samples, train the prototype adaptive module using the small sample pseudo-incremental context training method and cross-entropy loss function. S3.3 Using all samples in the base class audio samples, repeat step S3.1 to retrain the representation extractor. The parameters of the representation extractor remain unchanged in all incremental steps and are used to extract the depth representation of the incremental audio samples.

5. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 4, characterized in that, The characterization extractor pre-training process in step S3.1 is as follows: S3.1.1, Transfer the base class audio samples Divide into two subsets with non-overlapping categories and ,use Pre-trained representation extractor; S3.1.2, The representation extractor is trained using a few-sample learning method. The dataset is randomly divided into multiple subsets. Each subset contains N classes, with K support samples and K query samples for each class. The K support samples constitute the pre-trained support set, and the K query samples constitute the pre-trained query set. S3.1.3 Input the log-Mel spectrum of audio samples in the pre-trained support set and the pre-trained query set into the representation extractor to obtain the support representation and the query representation. Calculate the mean vector of the support representation of different categories of audio samples in the support set and use it as the prototype of different categories of audio samples in the support set. S3.1.4 Calculate the distance between the query representation and each prototype, and iteratively update the representation extractor parameters by minimizing the cross-entropy loss function to obtain the pre-trained representation extractor.

6. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 5, characterized in that, The prototype calculation formula in step S3.1 is as follows: ， In the formula This represents the prototype of category n, where n = 1, 2, ..., N. Let i represent the log-Mel spectrum of the i-th support sample, i = 1, 2, ..., K. This represents the character extraction function. Indicates the parameters characterizing the extractor; The cross-entropy loss function used to train the representation extractor in step S3.1 is defined as follows: ， In the formula Let y represent the log-Mel spectrum of the i-th query sample. Category tags, express The representation, and Let represent the prototypes of categories y and n, respectively. This represents the cosine distance between the query representation and the prototype.

7. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 5, characterized in that, The prototype adaptive module training process in step S3.2 is as follows: S3.2.1, Set the base class audio sample set Divide into two subsets and ,from Randomly select N classes, with K audio samples from each class, as the first support set S1. Then, randomly select K audio samples from the remaining samples of the N classes as the first query set Q1. Use the same method to... The second support set S2 and the second query set Q2 are obtained by randomly selecting audio samples. S1 and S2 constitute the third support set S3, and Q1 and Q2 constitute the third query set Q3. S3.2.2 Extract the log-Mel spectrum of audio samples from the third support set S3 and the query set Q3; S3.2.

3. Use the representation extractor pre-trained in step S3.1 to extract the support and query representations of the log-Mel spectra of audio samples in the third support and query sets. S3.2.

4. Input the support representations of the samples in the third support set S3 into the attention prototype generation module of the prototype adaptive network, and first pass them through three trainable linear transformation matrices. , , We obtain X1, X2, and X3. Then, X1 and X2 are successively processed through matrix multiplication, scaling, and sofmax layers before being multiplied by X3. Finally, they are processed through a trainable linear transformation matrix. get , The support prototype is obtained by element-wise addition with the input support representation vector, normalization, and finally averaging. S3.2.

5. The query representation and supporting prototype of the samples in the third query set Q3 are concatenated and input as the input vector to the prototype query representation adaptation module of the prototype adaptive network. This vector is then processed by three trainable linear transformation matrices. , , We obtain X4, X5, and X6. Then, X4 and X5 are successively processed through matrix multiplication, scaling, and sofmax layers before being multiplied by X6. Finally, they are processed through a trainable linear transformation matrix. get , The prototype and query representation are obtained by element-wise addition with the input vector, normalization, and finally vector segmentation. S3.2.6 Calculate the cosine distance between the adjusted query representation and each prototype. By continuously reducing the cross-entropy loss function, adjust the parameters of the prototype adaptive module. to Optimize until convergence.

8. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 1, characterized in that, The prototype adaptive adjustment and prototype classifier update process in step S5 is as follows: S5.1 Input the log-Mel spectrum features of the incremental audio samples into the representation extractor to obtain the incremental representation, and divide the representations of the same category into incremental support representations and incremental query representations. S5.

2. The attention prototype generation submodule of the incremental class support representation input prototype adaptive module first performs self-attention calculation, then adds the calculation result to the input support representation vector element-wise, then performs normalization processing, and finally calculates the mean to obtain the incremental class prototype. S5.

3. The incremental class query representation, the incremental class prototype, and all prototypes in the prototype classifier of the previous step are concatenated as input vectors and input into the prototype query representation adaptation submodule of the prototype adaptation module. Self-attention calculation is performed first, and then the calculation result is added to the input vector element-wise. Then, normalization is performed, and finally, the adjusted base class prototype and incremental class prototype are obtained through vector segmentation. S5.4 Replace all prototypes in the prototype classifier with the adjusted base class prototype and incremental class prototype to complete the prototype classifier update.

9. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 8, characterized in that, The formula for calculating self-attention in step S5 is: In the formula For the self-attention calculation results, X1, X2, and X3 are the input vectors after undergoing three trainable linear transformation matrices. , , The obtained value is D, which is the dimension of the input vector. Scaling factor This represents the matrix transpose operation. It is also a trainable linear transformation matrix. , , and It is obtained from the training in step S3.

2.

10. The fully small-sample incremental audio classification method based on a prototype classifier according to claim 8, characterized in that, The audio classification process in step S6 is as follows: S6.1 Extract the log-Mel spectrum features of the test audio samples; S6.2 Input the log-Mel spectrum features of the test audio samples into the trained self-attention convolutional neural network, and obtain the representation through the representation extractor; S6.3 Calculate the cosine distance between each prototype in the representation and prototype classifier. The category corresponding to the prototype with the smallest distance is the category to which the test audio sample belongs, and the audio classification result is obtained.

Images