An uncertainty-guided few-shot class-incremental audio classification method and device

Abstract

The present application relates to a kind of uncertainty guided few-shot class incremental audio classification method and device.The method includes: reconstructing audio sample using mask technology, generate mask audio sample, after the audio feature of mask audio sample is obtained by feature extractor, audio feature is input to Dropout random classifier to obtain prediction matrix.The uncertainty of prediction matrix is evaluated based on kernel norm to obtain the importance weight corresponding to class.Categorize the audio sample and the newly added audio sample to the feature extractor to extract the audio feature, to classify the extracted audio feature by random classifier, output the prediction result corresponding to the current sample category.According to the importance weight, adjust the class prototype loss function between prediction result and audio sample, then jointly optimize random classifier, obtain the optimized few-shot class incremental learning model.The audio classification precision under the condition of few-shot audio signal can be improved by using the method.

Description

Technical Field

[0001] The present invention relates to the technical field of audio classification, and in particular to a method and device for uncertainty-guided small-sample incremental audio classification. Background Art

[0002] Audio classification technology has been widely used in many fields, such as speech recognition, environmental sound detection, and music style classification. Its application areas usually involve the following key technical means and their combination with audio classification:

[0003] The first is audio signal preprocessing, the cornerstone of audio classification. It aims to improve the signal-to-noise ratio (SNR) and the stability of subsequent processing. This primarily involves denoising, which uses techniques such as frequency-domain filtering or adaptive filtering to remove noise and improve signal purity. It also involves normalization, which scales the audio signal's amplitude to a fixed range to eliminate amplitude variations under different recording conditions and ensure stability and consistency in subsequent processing.

[0004] The second step is feature extraction. Feature extraction is the core step of audio classification and determines the performance of the classification model. Common feature extraction methods include time domain features, frequency domain features, and time-frequency domain features.

[0005] The third is the selection and optimization of the classification model. After extracting rich audio features, it is necessary to select a suitable classification model for training and optimization, which usually includes rule-based classification methods, machine learning methods (such as support vector machines, random forests, etc.) and deep learning methods (such as convolutional neural networks, recurrent neural networks, etc.).

[0006] The fourth is the combined application of technical means, including the combination of preprocessing and feature extraction, the combination of traditional methods and deep learning, and the fusion of multimodal information.

[0007] Audio classification is a key task in both acoustic and multimedia analysis, and is a fundamental component of numerous audio-related applications, such as multimedia content analysis and search, music recommendation, audiovisual speech recognition, acoustic scene classification, sound event detection, and bioacoustic monitoring. Typically, these methods employ traditional supervised learning algorithms. Despite their impressive performance, these methods face two significant limitations: the need for large amounts of labeled data and the restriction to classifying only predefined classes (called base classes). In other words, they are constrained by a fixed class vocabulary. It can be seen that when new classes emerge in real scenarios or when users require a personalized class vocabulary, the number of samples for the new classes is usually limited.

[0008] In the field of audio classification, traditional methods typically operate under the assumption of a predetermined number of classes, a limitation that proves inadequate in real-world scenarios characterized by the unpredictability of target classes. A significant challenge arises when the model needs to incrementally adapt to new classes, as this process is prone to catastrophic forgetting, characterized by a large drop in the model's test performance on previously learned classes.

[0009] Incremental learning (also known as continuous learning or lifelong learning) refers to a learning paradigm in which a model continuously adapts to a series of new classes without forgetting previously acquired knowledge. For example, in sound classification and acoustic scene classification tasks, it continuously retains previously learned knowledge while learning new knowledge. However, traditional incremental learning methods often require large amounts of training data, complex training processes, and relatively high performance requirements for experimental equipment. These are not suitable for practical scenarios in resource-constrained acoustic fields or computing environments.

[0010] In contrast, Few-shot Class-incremental Audio Classification (FCAC) aims to perform class-incremental learning tasks in the absence of sufficient data. Current FCAC methods can generally be divided into two categories. The first category continuously expands the classifier in the incremental training phase, while the other category aims to improve the classifier prototype in the incremental training phase. Although these methods have shown excellent performance in accurately identifying new classes and base classes, they still have some common shortcomings that limit their performance. For example, these existing methods ignore the differences between samples of different categories and do not consider the impact of different samples during the training phase. In fact, the model effect in the field of artificial intelligence is easily affected by the uncertainty of training order and training samples. Summary of the Invention

[0011] Based on this, it is necessary to provide an uncertainty-guided few-sample incremental audio classification method and device that can overcome catastrophic forgetting and improve the accuracy of audio sample classification to address the above technical problems.

[0012] An uncertainty-guided few-sample incremental audio classification method, the method comprising:

[0013] Get the audio signal as audio samples.

[0014] The masking technology is used to reconstruct the audio sample to generate a masked audio sample. After obtaining the audio features of the masked audio sample through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0015] The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0016] The audio sample and the newly added audio sample are input into the feature extractor for audio feature extraction to obtain a first audio feature, the first audio feature is classified by a random classifier, and a prediction result corresponding to the category of the current audio sample is output.

[0017] The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample. The random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-shot class incremental learning model.

[0018] An uncertainty-guided, few-sample incremental audio classification device, comprising:

[0019] The data processing module is used to obtain the audio signal as an audio sample.

[0020] The prediction matrix acquisition module is used to reconstruct the audio sample using masking technology to generate a masked audio sample. After obtaining the audio features of the masked audio sample through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0021] The weight labeling module is used to evaluate the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0022] The prediction result acquisition module is used to input the audio sample and the newly added audio sample into the feature extractor to extract the audio features, obtain the first audio feature, classify the first audio feature through the random classifier, and output the prediction result corresponding to the category of the current audio sample.

[0023] The classification model optimization module is used to adjust the prototype loss function between the prediction results and the audio samples according to the importance weight to obtain the category prototype loss function of the predicted samples. The random classifier is jointly tuned according to the category prototype loss function of the predicted samples to obtain an optimized few-shot class incremental learning model.

[0024] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0025] Get the audio signal as audio samples.

[0026] The masking technology is used to reconstruct the audio sample to generate a masked audio sample. After obtaining the audio features of the masked audio sample through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0027] The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0028] The audio sample and the newly added audio sample are input into the feature extractor for audio feature extraction to obtain a first audio feature, the first audio feature is classified by a random classifier, and a prediction result corresponding to the category of the current audio sample is output.

[0029] The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample. The random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-shot class incremental learning model.

[0030] The uncertainty-guided few-shot class-incremental audio classification method and device first reconstructs the audio samples through a mask technique to generate masked audio samples. The features of the masked audio samples are extracted by a feature extractor and input into a Dropout random classifier to generate a prediction matrix corresponding to the class. Then, the uncertainty of the prediction matrix is evaluated by a kernel norm to obtain an importance weight corresponding to the class. This weight reflects the confidence of sample classification and is used to guide the subsequent training steps. Next, the original audio samples and the new audio samples are input into the feature extractor to extract the first audio features. The features are classified by the random classifier to output the class prediction result of the current audio sample. In this process, the method adjusts the prototype loss function between the prediction result and the current audio sample using the importance weight to construct a new class prototype loss function of the prediction sample. In this way, different weights are allocated to audio samples of different classes in the loss optimization, ensuring that the model can pay more attention to important samples, thereby effectively suppressing the catastrophic forgetting phenomenon. Finally, the random classifier is jointly optimized based on the class prototype loss function of the prediction sample to complete the training of the optimized few-shot class-incremental learning model. The optimized model can better adapt to new audio classes while maintaining the memory ability for the original class knowledge. This uncertainty-guided training strategy not only improves the adaptability and stability of the model in incremental learning, but also significantly reduces the dependence on a large amount of labeled data and the complexity of the training process. Through the above innovative steps, the method solves the common problems in traditional incremental learning and performs outstandingly in audio signal classification and denoising tasks. Especially in resource-constrained acoustic application scenarios, it can achieve efficient incremental learning and classification. At the same time, by using the mask technique and uncertainty guidance mechanism, the method greatly reduces noise interference and improves the processing accuracy of audio signals, providing an efficient and robust solution for the audio analysis field. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of an uncertainty-guided few-shot class-incremental audio classification method in one embodiment;

[0032] Figure 2 A backbone model framework diagram in one embodiment;

[0033] Figure 3 An uncertainty-guided few-shot class-incremental audio classification framework diagram in one embodiment;

[0034] Figure 4 A structural block diagram of an uncertainty-guided few-shot class-incremental audio classification device in one embodiment;

[0035] Figure 5 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] In one embodiment, Figure 1 As shown, an uncertainty-guided few-shot incremental audio classification method is provided, comprising the following steps:

[0038] Step 102: Acquire an audio signal as an audio sample.

[0039] In step 104, the audio sample is reconstructed using a masking technique to generate a masked audio sample. After the audio features of the masked audio sample are obtained through a feature extractor, the audio features are input into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample.

[0040] Step 106 : Evaluate the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0041] The importance weights are used to adjust the joint training loss function.

[0042] In step 108 , the audio sample and the newly added audio sample are input into a feature extractor for audio feature extraction to obtain a first audio feature. The first audio feature is classified by a random classifier, and a prediction result corresponding to the category of the current audio sample is output.

[0043] Step 110, adjusting the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tuning the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0044] In the above uncertainty-guided few-shot incremental audio classification method,

[0045] An audio signal is used as an input sample and reconstructed using a masking technique to generate a masked audio sample. First, the audio sample is reconstructed using a masking technique to generate a masked audio sample. Features of the masked audio sample are extracted using a feature extractor and fed into a dropout random classifier to generate a prediction matrix corresponding to each class. The uncertainty of the prediction matrix is ​​then evaluated using the nuclear norm to obtain the importance weights corresponding to each class. These weights reflect the confidence of the sample classification and are used to guide subsequent training steps. Next, the original audio sample and the newly added audio sample are fed into the feature extractor to extract the first audio feature. These features are then classified using a random classifier, which outputs a class prediction for the current audio sample. During this process, the method uses the importance weights to adjust the prototype loss function between the prediction result and the current audio sample, constructing a new class prototype loss function for the predicted sample. This results in different weights being assigned to audio samples of different classes in the loss optimization, ensuring that the model focuses more on important samples and effectively suppressing catastrophic forgetting. Finally, the random classifier is jointly tuned based on the class prototype loss function for the predicted samples to complete the training of the optimized few-shot incremental learning model. The optimized model can better adapt to newly added audio categories while maintaining its ability to remember knowledge of existing categories. This uncertainty-guided training strategy not only improves the adaptability and stability of the model in incremental learning, but also significantly reduces the reliance on large amounts of labeled data and the complexity of the training process. Through the above innovative steps, this method solves common problems in traditional incremental learning and demonstrates excellent performance in audio signal classification and denoising tasks. Especially in resource-constrained acoustic application scenarios, efficient incremental learning and classification can be achieved. At the same time, by utilizing masking technology and uncertainty-guided mechanisms, this method significantly reduces noise interference and improves the processing accuracy of audio signals, providing an efficient and robust solution for the field of audio analysis.

[0046] In one embodiment, an original audio signal transmitted by an external audio device is obtained, and the original audio signal is processed using a mel-spectrogram to obtain an audio signal, which is used as an audio sample.

[0047] In one embodiment, a masking technique is used to reconstruct an audio sample to generate a masked audio sample, and a feature extractor is used to extract audio features of an embedding layer from a log-mel spectrogram of the masked audio sample to obtain a masked audio feature.

[0048] In one embodiment, the audio features are input into a Dropout random classifier, and a Monte Carlo dropout algorithm is used to define the forward propagation method of each hidden layer in the Dropout random classifier:

[0049]

[0050] Among them, y h is the audio feature output by the hidden layer h in the random classifier, is the Hadamard product, w h is the weight, b h is the bias term, σ is the activation function, A vector whose elements are 0 or 1 and whose dimension is the same as the number of neurons in the hidden layer h-1 is set to 0. Calculate the output feature vector y of this vector and the hidden layer h-1. h-1 The Hadamard product of the h-1 layer is used to randomly discard the output of neurons in the h-1 layer, thereby introducing model uncertainty and preventing overfitting. The Dropout operation is used to classify the audio features according to the forward propagation method, and the prediction matrix corresponding to the category of the masked audio sample is output:

[0051]

[0052] Among them, P x is the prediction matrix of the audio sample x consisting of K×A prediction results, where x is the audio sample, x A is the Ath masked audio sample, is the K-th prediction result of the A-th masked audio sample, is the K-th prediction result of the 1st masked audio sample, K is the number of Dropout operations, and A is the number of mask augmentations.

[0053] In one embodiment, the uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain an uncertainty value:

[0054]

[0055] Among them, U is the uncertainty value, P is the prediction matrix, tr(.) is the trace of the prediction matrix, σ j is the jth largest singular value of the prediction matrix, ‖P‖ * is the nuclear norm of the prediction matrix, σ j Dim=min(K×A,D). Define the importance weight of each audio sample category according to the uncertainty value:

[0056]

[0057] Among them, B is the number of audio samples in each batch, U x is the uncertainty value of the audio sample x, W x is the importance weight of the audio sample x.

[0058] It is worth noting that through K dropout operations and A mask augmentations, the prediction matrix P for each sample x can be obtained, which consists of K×A prediction results. Each prediction result is a D-dimensional vector, where D represents the number of all visible classes. Therefore, the size of P is (K×A)×D. Since each row of the matrix represents the probability distribution of the prediction, higher inconsistency means greater differences between rows. Therefore, a simple way to measure the uncertainty value is the rank of the matrix P, which can represent the linear irrelevance between rows. Compared with generating uncertainty values ​​by maximizing the rank of the matrix P, this is an NP-difficult non-convex problem because the value of the matrix rank is discrete. A series of studies have shown that the matrix rank can be replaced by the nuclear norm, the convex envelope of the rank of P. Therefore, the uncertainty value U can be redefined using NN (nuclear norm). Mathematically, the nuclear norm and the Frobenius norm are bounds of each other. The Frobenius norm can be defined as:

[0059]

[0060] And, it can be expressed by the following formula ‖P‖ * and ‖P‖ F The mutual boundary relationship between them and ‖P‖ F It can be constructed by the following formula:

[0061]

[0062] In addition, existing researchers have proved that in terms of monotonicity, ‖P‖ F Strictly opposite to Shannon entropy, maximizing ‖P‖ F =Equal to minimizing entropy. Therefore, the nuclear norm can be used not only to analyze diversity but also to analyze irrelevance in order to quantify the inconsistency of the prediction matrix P and estimate the uncertainty value.

[0063]

[0064] In one embodiment, the mel-spectrogram of the audio sample and the mel-spectrogram sample of the newly added audio sample are input to the feature extractor. During the incremental training process, the loss function of the feature extractor is reconstructed based on the audio sample:

[0065]

[0066] Among them, x is the audio sample, y is the label corresponding to the audio sample, fe is the feature extractor, and N is the number of all categories seen up to the current incremental session. is the weight of the random classifier, fe(x) is the encoding vector of the audio sample x, and λ is a constant coefficient. The audio features of the newly added audio sample are extracted according to the reconstructed loss function to obtain the first audio features.

[0067] In one embodiment, in an incremental session, the first audio feature is classified by a random classifier, the prototypes of the categories of the previous batch of audio samples are spliced ​​with the encoding vectors of the categories of the current audio sample to train the random classifier, the scores of each category in the first audio feature are calculated by the trained random classifier, and the prediction results corresponding to the category of the current audio sample are obtained based on the scores.

[0068] In one embodiment, the prototype loss function between the category prototype of the prediction result and the category prototype of the current audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample:

[0069]

[0070] Among them, L p is the category prototype loss function of the predicted sample, y is the label of the previous batch of audio samples, p c is the prototype of category c of the current audio sample in the incremental session, O is the number of old categories of the previous batch of audio samples, is the weight of the random classifier, N is the number of categories seen up to the current incremental session, and p o For the category prototype of the current audio sample in the base session, a joint loss function is constructed based on the category prototype loss function of the predicted sample and the true label of the current audio sample:

[0071]

[0072] in, is the joint loss function, is the reconstruction loss function, x is the audio signal, y is the label of the historical audio signal, fe is the feature extractor, N is the number of all categories seen up to the current incremental session, is the weight of the random classifier, fe(x) is the feature extractor, and λ is a constant coefficient. The model parameters of the random classifier are optimized according to the joint loss function to obtain the optimized few-shot incremental learning model.

[0073] In one embodiment, a backbone model framework proposed in Few-shot Incremental Audio Classification Based on Random Classifier is followed, where the model f consists of a feature extractor (FE) and a random classifier. Figure 2 As shown in Figure 2, we first train the FE in a supervised manner using a multilayer perceptron classifier and abundant samples in the base training phase. The FE is then frozen to mitigate catastrophic forgetting in the following training phase and used to extract embedding layer features from the log-mel spectrogram of the samples. Next, we train a random classifier (SC) in the incremental training phase using the following loss function:

[0074]

[0075] Where x and y are training samples and corresponding labels; fe represents the feature extractor; N represents the number of all visible classes; represents the weight of SC; fe(x) represents the embedding layer feature of x.

[0076] To overcome catastrophic forgetting, the prototype is the average vector of the embedding layer for each class, which is saved after each training phase and used to train SC and calculate the class score in the incremental training phase. p Defined as:

[0077]

[0078] where p c is the prototype of class c, y is the label, and O represents the number of old classes. In the mth training stage, both the prototype and the samples in the current training stage are used to train SC with a joint loss:

[0079]

[0080] where λ is the coefficient.

[0081] In one embodiment, Figure 3 As shown in the figure, an uncertainty-guided few-shot incremental audio classification framework is provided, which includes three main components: (A) dropout SC; (B) masked samples; (C) nuclear norm. First, we perform A data augmentation on the samples with different random masks of fixed size. Then, we collect the prediction results of K random forward passes with MCD. sg represents the stopping gradient. In this way, we can obtain a prediction matrix consisting of (K×A) prediction results. D is the dimension of each prediction result vector. In addition, we use the nuclear norm instead of variance to maintain a high tolerance to noise and outliers to calculate the uncertainty value of the sample. Based on the uncertainty value, we can calculate the importance weight of each sample to control the size of the training loss for each new sample in the batch, which implicitly changes the learning order.

[0082] First, assume that SC consists of H hidden layers. For each hidden layer h, y h represents the output of this layer, w h and b h represents weights and biases. Then y h Can be

[0083] y h =σ(w h y h-1 +b h )

[0084] Where σ represents the activation function. The dropout operation refers to dropping certain neural units in a specific layer; temporarily canceling the connections between these units and other units. Figure 3 As shown in (A), the same number of units are randomly dropped each time, and the SC after the dropout operation is only used to estimate uncertainty, not for training. Specifically, for layer h, the generated vector r h , which has the same dimension as layer h, and r h Each element of follows the Bernoulli distribution with probability p, that is, After the dropout operation, a 1-p fraction of the output neural units are set to zero and the forward pass can be redefined as:

[0085]

[0086] in, Denotes Hadamard product. For each batch of samples, K forward propagations and different dropout operations are implemented. The number of neural units dropped in the same layer is random each time, but the number is fixed. Therefore, K prediction results can be obtained. For each sample x, the uncertainty value U for each sample x can then be estimated using the variance of the prediction results x :

[0087]

[0088] If SC has high confidence in the training samples, randomly dropping a small number of neurons will not significantly affect the prediction results. Therefore, the prediction results after different dropout operations will be similar and the variance will be low. For each batch of B samples, we can get the uncertainty values ​​U1, U2, ..., U B And define the importance weight of each sample x:

[0089]

[0090] It is worth noting that the use of the term max(U1,U2,…,U B ) data. Then, the SC loss of the batch can be redefined as:

[0091]

[0092] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0093] In one embodiment, Figure 4 As shown, an uncertainty-guided few-sample incremental audio classification device is provided, comprising: a data processing module 402, a prediction matrix acquisition module 404, a weight marking module 406, a prediction result acquisition module 408, and a classification model optimization module 410, wherein:

[0094] The data processing module 402 is configured to obtain an audio signal as an audio sample.

[0095] The prediction matrix acquisition module 404 is used to reconstruct the audio sample using a masking technique to generate a masked audio sample. After obtaining the audio features of the masked audio sample through a feature extractor, the audio features are input into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample.

[0096] The weight marking module 406 is used to evaluate the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weight corresponding to the category.

[0097] Prediction result acquisition module 408 is configured to input the audio sample and the newly added audio sample into a feature extractor for audio feature extraction, obtain a first audio feature, classify the first audio feature using a random classifier, and output a prediction result corresponding to the category of the current audio sample.

[0098] The classification model optimization module 410 is used to adjust the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tune the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0099] For the specific limitations of the uncertainty-guided small-sample incremental audio classification device, please refer to the limitations of the uncertainty-guided small-sample incremental audio classification method above, which will not be repeated here. The various modules in the above-mentioned uncertainty-guided small-sample incremental audio classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0100] In an embodiment, a computer device, which can be a terminal, has an internal structure diagram as shown in Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement an uncertainty-guided few-shot class-incremental audio classification method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0101] Those skilled in the art can understand that Figure 4-5 the structure shown in the above embodiment is only a block diagram of part of the structure related to the present application, and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0102] In an embodiment, a computer device is provided, which includes a memory storing a computer program and a processor executing the computer program to implement the following steps:

[0103] An audio signal is obtained as an audio sample.

[0104] The audio sample is reconstructed using a mask technique to generate a masked audio sample. After obtaining audio features of the masked audio sample through a feature extractor, the audio features are input into a Dropout random classifier to obtain a prediction matrix corresponding to the class of the masked audio sample.

[0105] The uncertainty of the prediction matrix is evaluated based on a kernel norm to obtain an importance weight corresponding to the class.

[0106] The audio sample and the new audio sample are input into the feature extractor for audio feature extraction to obtain first audio features. The first audio features are classified by the random classifier to output a prediction result corresponding to the class of the current audio sample.

[0107] The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample. The random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-shot class incremental learning model.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0109] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. An uncertainty-guided few-shot incremental audio classification method, characterized by: The method comprises: Obtaining an audio signal as audio samples; Reconstructing the audio sample using a masking technique to generate a masked audio sample, obtaining audio features of the masked audio sample through a feature extractor, and then inputting the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample; The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the importance weights corresponding to the categories; the uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the uncertainty value: Among them, U is the uncertainty value, P is the prediction matrix, is the trace of the prediction matrix, is the jth largest singular value of the prediction matrix, is the nuclear norm of the prediction matrix; The importance weight corresponding to the category of each audio sample is defined according to the uncertainty value: Among them, B is the number of audio samples in each batch, is the uncertainty value of the audio sample x, is the importance weight of audio sample x; Inputting the audio sample and the newly added audio sample into the feature extractor to extract audio features to obtain a first audio feature, classifying the first audio feature using a random classifier, and outputting a prediction result corresponding to the category of the current audio sample; The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample, and the random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

2. The method according to claim 1, characterized in that Get the audio signal as audio samples, including: An original audio signal transmitted by an external audio device is obtained, and the original audio signal is processed using a mel-spectrogram to obtain an audio signal, which is used as an audio sample.

3. The method according to claim 2, characterized in that Reconstructing the audio sample using a masking technique to generate a masked audio sample, and obtaining audio features of the masked audio sample using a feature extractor, including: The audio sample is reconstructed using a masking technique to generate a masked audio sample, and audio features of an embedding layer are extracted from a log-mel spectrogram of the masked audio sample using a feature extractor to obtain a masked audio feature.

4. The method according to claim 3, characterized in that Inputting the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample includes: The audio features are input into the Dropout random classifier, and the Monte Carlo dropout algorithm is used to define the forward propagation mode of each hidden layer in the Dropout random classifier: in, is the audio feature output by the hidden layer h in the random classifier, is the Hadamard product, is the importance weight, is the activation function, is a vector with the same dimension as the number of neurons in the hidden layer h-1, and the jth element is set to 0. is the audio feature output by the hidden layer h-1 in the random classifier, is the bias term; The audio features are classified using the Dropout operation according to the forward propagation method, and a prediction matrix corresponding to the category of the masked audio sample is output: in, for The prediction matrix composed of prediction results, For audio samples, is the Ath masked audio sample, is the K-th prediction result of the A-th masked audio sample, is the K-th prediction result of the 1st masked audio sample, K is the number of Dropout operations, and A is the number of mask augmentations.

5. The method according to claim 4, characterized in that Inputting the audio sample and the newly added audio sample into the feature extractor to extract audio features to obtain a first audio feature, including: The mel-spectrogram of the audio sample and the mel-spectrogram sample of the newly added audio sample are input to the feature extractor. During the incremental training process, the loss function of the feature extractor is reconstructed according to the audio sample: Among them, x is the audio sample, y is the label corresponding to the audio sample, is the feature extractor, N is the number of categories seen up to the current incremental session, is the weight of the random classifier, is the encoding vector of audio sample x; The audio feature of the newly added audio sample is extracted according to the reconstructed loss function to obtain a first audio feature.

6. The method according to claim 5, characterized in that Classify the first audio feature using a random classifier and output a prediction result corresponding to the category of the current audio sample, including: In the incremental session, the first audio feature is classified by a random classifier, and the prototypes of the categories of the previous batch of audio samples are spliced ​​with the encoding vectors of the categories of the current audio sample to train the random classifier. The scores of each category in the first audio feature are calculated by the trained random classifier, and the prediction results corresponding to the category of the current audio sample are obtained based on the scores.

7. The method according to claim 6, characterized in that Adjusting the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain a category prototype loss function of the predicted sample, and jointly tuning the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model, including: The prototype loss function between the category prototype of the prediction result and the category prototype of the current audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample: in, is the category prototype loss function of the predicted sample, y is the label of the previous batch of audio samples, is the prototype of category c of the current audio sample in the incremental session, is the number of old categories of the previous batch of audio samples, is the weight of the random classifier, N is the number of categories seen up to the current incremental session, is the category prototype of the current audio sample in the base session; A joint loss function is constructed based on the category prototype loss function of the predicted sample and the true label of the current audio sample: in, is the joint loss function, is the reconstruction loss function, λ is a constant coefficient; The model parameters of the random classifier are optimized according to the joint loss function to obtain an optimized few-shot class incremental learning model.

8. An uncertainty-guided few-sample incremental audio classification device, characterized by: For implementing the method according to any one of claims 1 to 7, the device comprises: A data processing module, configured to obtain an audio signal as an audio sample; A prediction matrix acquisition module is configured to reconstruct the audio sample using a masking technique to generate a masked audio sample, obtain audio features of the masked audio sample through a feature extractor, and then input the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample; A weight labeling module, configured to evaluate the uncertainty of the prediction matrix based on a nuclear norm to obtain importance weights corresponding to the categories; a prediction result acquisition module, configured to input the audio sample and the newly added audio sample into the feature extractor to extract audio features, obtain a first audio feature, classify the first audio feature using a random classifier, and output a prediction result corresponding to the category of the current audio sample; A classification model optimization module is used to adjust the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tune the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

Images