Uncertainty-guided few-sample class incremental audio classification method and device

Abstract

The invention relates to an uncertainty-guided few-sample class incremental audio classification method and device. The method comprises the following steps: reconstructing an audio sample by adopting a mask technology, generating a mask audio sample, obtaining audio features of the mask audio sample through a feature extractor, and inputting the audio features into a Dropout random classifier to obtain a prediction matrix; and evaluating the uncertainty of the prediction matrix based on a nuclear norm so as to obtain an importance weight corresponding to the category. And inputting the audio sample and the newly added audio sample into a feature extractor for audio feature extraction, classifying the extracted audio features through a random classifier, and outputting a prediction result corresponding to the current sample category. And after a category prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight, joint tuning is performed on the random classifier, and an optimized few-sample class incremental learning model is obtained. By adopting the method, the audio classification precision under the condition of few sample quantity audio signals can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio classification, and in particular to an uncertainty-guided few-sample incremental audio classification method and device. Background Art

[0002] Audio classification technology has a wide range of applications in many fields, such as speech recognition, environmental sound detection, music style classification, etc. In its application field, it usually involves the following key technical means and their combined application with audio classification:

[0003] The first is audio signal preprocessing, which is the cornerstone of audio classification and aims to improve the signal-to-noise ratio (SNR) of the signal and the stability of subsequent processing. It mainly includes denoising, which is to remove noise from the audio signal and improve the purity of the signal through frequency domain filtering or adaptive filtering and other technologies. And normalization, which is to scale the amplitude of the audio signal to a fixed range to eliminate the amplitude difference under different recording conditions and ensure the stability and consistency of subsequent processing.

[0004] The second is feature extraction, which is the core step of audio classification and determines the performance of the classification model. Commonly used feature extraction methods include time domain features, frequency domain features, and time-frequency domain features.

[0005] The third is the selection and optimization of the classification model. After extracting rich audio features, it is necessary to select a suitable classification model for training and optimization, which usually includes rule-based classification methods, machine learning methods (such as support vector machines, random forests, etc.) and deep learning methods (such as convolutional neural networks, recurrent neural networks, etc.).

[0006] Fourth, the combined application of technical means, including the combination of preprocessing and feature extraction, the combination of traditional methods and deep learning, and the fusion of multimodal information.

[0007] Audio classification is a key task in both acoustic and multimedia analysis, and is a fundamental component of numerous audio-related applications, such as multimedia content analysis and search, music recommendation, audiovisual speech recognition, acoustic scene classification, sound event detection, and bioacoustic monitoring. Typically, these methods employ traditional supervised learning algorithms. Despite their impressive performance, these methods face two significant limitations: the large demand for labeled data and the restriction to classify only predefined classes (called base classes). In other words, they are constrained by a fixed class vocabulary. It can be seen that when new classes appear in real scenarios or when users require a personalized class vocabulary, the number of samples for the new classes is usually limited.

[0008] In the field of audio classification, traditional methods typically operate under the assumption of a predetermined number of classes, a limitation that proves inadequate for current approaches in real-world scenarios characterized by the unpredictability of target classes. A significant challenge arises when models need to incrementally adapt to new classes, as this process is prone to catastrophic forgetting, characterized by a large drop in the model's test performance on previously learned classes.

[0009] Class incremental learning (incremental learning is also called continuous learning or lifelong learning) refers to a learning paradigm in which the model continuously adapts to a series of new classes without forgetting previously acquired knowledge. For example, in sound classification and acoustic scene classification tasks, new knowledge is learned while retaining the learned knowledge. However, traditional incremental learning methods often require a large amount of training data, a complex training process, and relatively high requirements for experimental equipment performance, which are not suitable for actual scenarios in resource-constrained acoustic fields or computing environments.

[0010] In contrast, Few-shot Class-incremental Audio Classification (FCAC) aims to perform class incremental learning tasks when data is insufficient. Current FCAC methods can generally be divided into two categories. The first category continuously expands the classifier in the incremental training phase, while the other category aims to improve the classifier prototype in the incremental training phase. Although these methods have shown excellent performance in accurately identifying new classes and base classes, they still have some common shortcomings that limit their performance. For example, these existing methods ignore the differences in samples of different categories and do not consider the impact of different samples during the training phase. In fact, the model effect in the field of artificial intelligence is easily affected by the uncertainty of training order and training samples. Summary of the invention

[0011] Based on this, it is necessary to provide an uncertainty-guided few-sample incremental audio classification method and device that can overcome catastrophic forgetting and improve the classification accuracy of audio samples in response to the above technical problems.

[0012] An uncertainty-guided few-sample incremental audio classification method, the method comprising:

[0013] Get the audio signal as audio samples.

[0014] The masking technology is used to reconstruct the audio sample to generate a masked audio sample. After the audio features of the masked audio sample are obtained through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0015] The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the importance weights of the class correspondences.

[0016] The audio sample and the newly added audio sample are input into the feature extractor to extract the audio features, obtain the first audio feature, classify the first audio feature through the random classifier, and output the prediction result corresponding to the category of the current audio sample.

[0017] The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample. The random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0018] An uncertainty-guided few-sample incremental audio classification device, the device comprising:

[0019] The data processing module is used to obtain the audio signal as an audio sample.

[0020] The prediction matrix acquisition module is used to reconstruct the audio sample by using the masking technology to generate the masked audio sample. After obtaining the audio features of the masked audio sample through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0021] The weight labeling module is used to evaluate the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0022] The prediction result acquisition module is used to input the audio sample and the newly added audio sample into the feature extractor to extract the audio feature, obtain the first audio feature, classify the first audio feature through the random classifier, and output the prediction result corresponding to the category of the current audio sample.

[0023] The classification model optimization module is used to adjust the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tune the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0024] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0025] Get the audio signal as audio samples.

[0026] The masking technology is used to reconstruct the audio sample to generate a masked audio sample. After the audio features of the masked audio sample are obtained through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0027] The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the importance weights of the class correspondences.

[0028] The audio sample and the newly added audio sample are input into the feature extractor to extract the audio features, obtain the first audio feature, classify the first audio feature through the random classifier, and output the prediction result corresponding to the category of the current audio sample.

[0029] The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample. The random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0030] The above-mentioned uncertainty-guided few-sample incremental audio classification method and device first reconstructs the audio sample through the masking technology to generate a masked audio sample. The features of the masked audio sample are extracted by the feature extractor and input into the Dropout random classifier to generate a prediction matrix corresponding to the category. Then, the uncertainty of the prediction matrix is ​​evaluated by the nuclear norm to obtain the importance weight corresponding to the category. This weight reflects the confidence of the sample classification and is used to guide the subsequent training steps. Then, the original audio sample and the newly added audio sample are input into the feature extractor together to extract the first audio feature. These features are classified by the random classifier, and the category prediction result of the current audio sample is output. In this process, the method uses the importance weight to adjust the prototype loss function between the prediction result and the current audio sample, and constructs a new category prototype loss function of the predicted sample. In this way, different categories of audio samples are assigned different weights in the loss optimization, ensuring that the model can pay more attention to important samples, thereby effectively suppressing the catastrophic forgetting phenomenon. Finally, the random classifier is jointly tuned based on the category prototype loss function of the predicted sample to complete the training of the optimized few-sample incremental learning model. The optimized model can better adapt to new audio categories while maintaining the ability to remember the original category knowledge. This uncertainty-guided training strategy not only improves the adaptability and stability of the model in incremental learning, but also significantly reduces the dependence on large amounts of labeled data and the complexity of the training process. Through the above innovative steps, this method solves common problems in traditional incremental learning and demonstrates excellent performance in audio signal classification and denoising tasks. Especially in resource-constrained acoustic application scenarios, efficient incremental learning and classification can be achieved. At the same time, by utilizing masking technology and uncertainty-guided mechanisms, this method greatly reduces noise interference and improves the processing accuracy of audio signals, providing an efficient and robust solution for the field of audio analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a flow chart of an uncertainty-guided few-sample incremental audio classification method in one embodiment;

[0032] Figure 2 A schematic diagram of a backbone model framework in an embodiment;

[0033] Figure 3 A schematic diagram of an uncertainty-guided few-sample incremental audio classification framework in one embodiment;

[0034] Figure 4 A structural block diagram of an uncertainty-guided few-sample incremental audio classification device in one embodiment;

[0035] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] In one embodiment, Figure 1 As shown, an uncertainty-guided few-sample incremental audio classification method is provided, comprising the following steps:

[0038] Step 102: Acquire an audio signal as an audio sample.

[0039] Step 104, reconstructing the audio sample using masking technology to generate a masked audio sample, obtaining the audio features of the masked audio sample through a feature extractor, and then inputting the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample.

[0040] Step 106 , evaluating the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0041] The importance weights are used to adjust the joint training loss function.

[0042] Step 108: Input the audio sample and the newly added audio sample into a feature extractor to extract audio features, obtain a first audio feature, classify the first audio feature through a random classifier, and output a prediction result corresponding to the category of the current audio sample.

[0043] Step 110, adjusting the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tuning the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0044] In the above uncertainty-guided few-sample incremental audio classification method,

[0045] The audio signal is used as an input sample and reconstructed by masking technology to generate a masked audio sample. First, the audio sample is reconstructed by masking technology to generate a masked audio sample. The features of the masked audio sample are extracted by the feature extractor and input into the Dropout random classifier to generate a prediction matrix corresponding to the category. Then, the uncertainty of the prediction matrix is ​​evaluated by the nuclear norm to obtain the importance weight corresponding to the category. This weight reflects the confidence of the sample classification and is used to guide the subsequent training steps. Next, the original audio sample and the newly added audio sample are input into the feature extractor to extract the first audio feature. These features are classified by the random classifier, and the category prediction result of the current audio sample is output. In this process, the method uses the importance weight to adjust the prototype loss function between the prediction result and the current audio sample, and constructs a new category prototype loss function of the predicted sample. In this way, different categories of audio samples are assigned different weights in the loss optimization, ensuring that the model can pay more attention to important samples, thereby effectively suppressing the catastrophic forgetting phenomenon. Finally, the random classifier is jointly tuned based on the category prototype loss function of the predicted sample to complete the training of the optimized few-sample incremental learning model. The optimized model can better adapt to new audio categories while maintaining the ability to remember the original category knowledge. This uncertainty-guided training strategy not only improves the adaptability and stability of the model in incremental learning, but also significantly reduces the dependence on large amounts of labeled data and the complexity of the training process. Through the above innovative steps, this method solves common problems in traditional incremental learning and demonstrates excellent performance in audio signal classification and denoising tasks. Especially in resource-constrained acoustic application scenarios, efficient incremental learning and classification can be achieved. At the same time, by utilizing masking technology and uncertainty-guided mechanisms, this method greatly reduces noise interference and improves the processing accuracy of audio signals, providing an efficient and robust solution for the field of audio analysis.

[0046] In one of the embodiments, an original audio signal transmitted by an external audio device is obtained, and the original audio signal is processed using a mel-spectrogram to obtain an audio signal, and the audio signal is used as an audio sample.

[0047] In one of the embodiments, the masking technology is used to reconstruct the audio sample to generate the masked audio sample, and the audio features of the embedding layer are extracted from the logarithmic Mel spectrogram of the masked audio sample by a feature extractor to obtain the masked audio features.

[0048] In one embodiment, the audio features are input into a Dropout random classifier, and a Monte Carlo dropout algorithm is used to define the forward propagation method of each hidden layer in the Dropout random classifier:

[0049]

[0050] Among them, y h is the audio feature output by the hidden layer h in the random classifier, is the Hadamard product, w h is the weight, b h is the bias term, σ is the activation function, is a vector whose element value is 0 or 1 and whose dimension is the same as the number of neurons in the hidden layer h-1. Its jth element is set to 0 and the output feature vector y of the hidden layer h-1 is calculated. h-1 The Hadamard product of , that is, the effect of randomly discarding the output of the h-1 layer neurons is achieved, so as to introduce model uncertainty and prevent overfitting. According to the forward propagation method, the Dropout operation is used to classify the audio features, and the prediction matrix corresponding to the category of the masked audio sample is output:

[0051]

[0052] Among them, P x is the prediction matrix of audio sample x consisting of K×A prediction results, where x is the audio sample, x A is the Ath masked audio sample, is the K-th prediction result of the A-th masked audio sample, is the Kth prediction result of the 1st masked audio sample, K is the number of Dropout operations, and A is the number of mask augmentations.

[0053] In one embodiment, the uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain an uncertainty value:

[0054]

[0055] Among them, U is the uncertainty value, P is the prediction matrix, tr(.) is the trace of the prediction matrix, σ j is the jth largest singular value of the prediction matrix, ‖P‖ * is the nuclear norm of the prediction matrix, σ j Dim = min (K × A, D). Define the importance weight of each audio sample category according to the uncertainty value:

[0056]

[0057] Among them, B is the number of audio samples in each batch, U x is the uncertainty value of the audio sample x, W x is the importance weight of audio sample x.

[0058] It is worth noting that through K dropout operations and A mask augmentations, the prediction matrix P for each sample x can be obtained, which consists of K×A prediction results. Each prediction result is a D-dimensional vector, where D represents the number of all visible classes. Therefore, the size of P is (K×A)×D. Since each row of the matrix represents the probability distribution of the prediction, higher inconsistency means greater differences between rows. Therefore, a simple way to measure the uncertainty value is the rank of the matrix P, which can represent the linear uncorrelation between rows. Compared with generating uncertainty values ​​by maximizing the rank of the matrix P, this is an NP-difficult non-convex problem because the value of the matrix rank is discrete. A series of studies have shown that the matrix rank can be replaced by the nuclear norm, the convex envelope of the rank of P. Therefore, the uncertainty value U can be redefined using NN (nuclear norm). Mathematically, the nuclear norm and the Frobenius norm are the boundaries of each other. The Frobenius norm can be defined as:

[0059]

[0060] And, it can be expressed by the following formula ‖P‖ * and ‖P‖ F The mutual boundary relationship between them and ‖P‖ F It can be constructed by the following formula:

[0061]

[0062] In addition, existing researchers have proved that in terms of monotonicity, ‖P‖ F Strictly opposite to Shannon entropy, maximizing ‖P‖ F is equal to minimizing entropy. Therefore, the nuclear norm can be used not only to analyze diversity but also to analyze irrelevance in order to quantify the inconsistency of the prediction matrix P and estimate the uncertainty value.

[0063]

[0064] In one embodiment, the mel-spectrogram of the audio sample and the mel-spectrogram sample of the newly added audio sample are input into the feature extractor, and during the incremental training process, the loss function of the feature extractor is reconstructed according to the audio sample:

[0065]

[0066] Among them, x is the audio sample, y is the label corresponding to the audio sample, fe is the feature extractor, and N is the number of all categories seen up to the current incremental session. is the weight of the random classifier, fe(x) is the encoding vector of the audio sample x, and λ is a constant coefficient. The audio features of the newly added audio sample are extracted according to the reconstructed loss function to obtain the first audio feature.

[0067] In one of the embodiments, in an incremental session, a first audio feature is classified by a random classifier, and the prototypes of the categories of the previous batch of audio samples are concatenated with the encoding vectors of the categories of the current audio samples to train the random classifier. The scores of each category in the first audio feature are calculated by the trained random classifier, and a prediction result corresponding to the category of the current audio sample is obtained based on the scores.

[0068] In one embodiment, the prototype loss function between the category prototype of the prediction result and the category prototype of the current audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample:

[0069]

[0070] Among them, L p is the category prototype loss function of the predicted sample, y is the label of the previous batch of audio samples, p c is the prototype of category c of the current audio sample in the incremental session, O is the number of old categories of the previous batch of audio samples, is the weight of the random classifier, N is the number of categories seen up to the current incremental session, and p o For the category prototype of the current audio sample in the base session, a joint loss function is constructed based on the category prototype loss function of the predicted sample and the true label of the current audio sample:

[0071]

[0072] in, is the joint loss function, is the reconstruction loss function, x is the audio signal, y is the label of the historical audio signal, fe is the feature extractor, N is the number of all categories seen up to the current incremental session, is the weight of the random classifier, fe(x) is the feature extractor, and λ is a constant coefficient. The model parameters of the random classifier are optimized according to the joint loss function to obtain an optimized incremental learning model for the few-shot class.

[0073] In one embodiment, a backbone model framework proposed in Few-shot Incremental Audio Classification Based on Random Classifier is followed, where the model f consists of a feature extractor (FE) and a random classifier. Figure 2 As shown in Figure 1, first in the basic training phase, we train the FE in a supervised manner with a multilayer perceptron classifier and abundant samples. Then, the FE is frozen to mitigate catastrophic forgetting in the following training phase and is used to extract embedding layer features from the log-mel spectrogram of the samples. Next, we train a random classifier (SC) in the incremental training phase with the following loss function:

[0074]

[0075] Where x and y are training samples and corresponding labels; fe represents the feature extractor; N represents the number of all visible classes; represents the weight of SC; fe(x) represents the embedding layer feature of x.

[0076] To overcome catastrophic forgetting, the prototype is the average vector of the embedding layer for each class, which is saved after each training stage and used to train the SC and calculate the class scores in the incremental training stage. p Defined as:

[0077]

[0078] where p c is the prototype of class c, y is the label, and O represents the number of old classes. In the mth training stage, both the prototype and the samples in the current training stage are used to train SC with a joint loss:

[0079]

[0080] where λ is the coefficient.

[0081] In one embodiment, if Figure 3 As shown in the figure, an uncertainty-guided few-class incremental audio classification framework is provided, which includes three main components: (A) dropout SC; (B) masked samples; (C) nuclear norm. First, we perform A data augmentation on the samples with different random masks of fixed size. Then, we collect the prediction results of K random forward passes with MCD. sg represents the stopping gradient. In this way, we can get a prediction matrix consisting of (K×A) prediction results. D is the dimension of each prediction result vector. In addition, we use the nuclear norm instead of variance to maintain a high tolerance to noise and outliers to calculate the uncertainty value of the sample. Based on the uncertainty value, we can calculate the importance weight of each sample to control the size of the training loss for each new sample in the batch, which implicitly changes the learning order.

[0082] First, assume that SC consists of H hidden layers. For each hidden layer h, y h represents the output of this layer, w h and b h represents weights and biases. Then y h Can be

[0083] y h =σ(w h y h-1 +b h )

[0084] where σ represents the activation function. The dropout operation refers to dropping certain neural units in a specific layer; temporarily canceling the connection between these units and other units. Figure 3 As shown in (A), the same number of units are randomly dropped each time, and the SC after the dropout operation is only used to estimate uncertainty but not for training. Specifically, for layer h, the generated vector r h , which has the same dimension as layer h, and r h Each element of follows the Bernoulli distribution with probability p, that is, After the dropout operation, a 1-p ratio of the output neural units is set to zero and the forward pass can be redefined as:

[0085]

[0086] in, represents the Hadamard product. For each batch of samples, K forward propagations and different dropout operations are implemented. The number of neural units dropped in the same layer is random each time, but the number is fixed. Therefore, K prediction results can be obtained For each sample x, the uncertainty value U for each sample x can then be estimated using the variance of the prediction results x :

[0087]

[0088] If SC has high confidence in the training samples, randomly dropping a small number of neurons will not significantly affect the prediction results. Therefore, the prediction results after different dropout operations will be similar and the variance will be low. For each batch of B samples, we can get the uncertainty value U 1 ,U 2 ,…,U B And define the importance weight of each sample x:

[0089]

[0090] It is worth noting that the use of the term max(U 1 ,U 2 ,…,U B ). Then, the SC loss of the batch can be redefined as:

[0091]

[0092] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0093] In one embodiment, Figure 4 As shown, an uncertainty-guided small sample class incremental audio classification device is provided, comprising: a data processing module 402, a prediction matrix acquisition module 404, a weight marking module 406, a prediction result acquisition module 408 and a classification model optimization module 410, wherein:

[0094] The data processing module 402 is used to obtain an audio signal as an audio sample.

[0095] The prediction matrix acquisition module 404 is used to reconstruct the audio sample by using the masking technology to generate a masked audio sample. After obtaining the audio features of the masked audio sample through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0096] The weight marking module 406 is used to evaluate the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories.

[0097] The prediction result acquisition module 408 is used to input the audio sample and the newly added audio sample into the feature extractor to extract the audio feature, obtain the first audio feature, classify the first audio feature through the random classifier, and output the prediction result corresponding to the category of the current audio sample.

[0098] The classification model optimization module 410 is used to adjust the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tune the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0099] For the specific limitations of the uncertainty-guided small-sample incremental audio classification device, please refer to the limitations of the uncertainty-guided small-sample incremental audio classification method above, which will not be repeated here. Each module in the above-mentioned uncertainty-guided small-sample incremental audio classification device can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0100] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an uncertainty-guided small sample class incremental audio classification method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0101] Those skilled in the art will understand that Figure 4-5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0102] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0103] Get the audio signal as audio samples.

[0104] The masking technology is used to reconstruct the audio sample to generate a masked audio sample. After the audio features of the masked audio sample are obtained through the feature extractor, the audio features are input into the Dropout random classifier to obtain the prediction matrix corresponding to the category of the masked audio sample.

[0105] The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the importance weights of the class correspondences.

[0106] The audio sample and the newly added audio sample are input into the feature extractor to extract the audio features, obtain the first audio feature, classify the first audio feature through the random classifier, and output the prediction result corresponding to the category of the current audio sample.

[0107] The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample. The random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0109] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An uncertainty-guided few-sample incremental audio classification method, characterized in that: The method comprises: Obtaining an audio signal as audio samples; Reconstructing the audio sample by using a masking technique to generate a masked audio sample, obtaining audio features of the masked audio sample by a feature extractor, and then inputting the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample; Evaluating the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories; Inputting the audio sample and the newly added audio sample into the feature extractor to extract audio features to obtain a first audio feature, classifying the first audio feature through a random classifier, and outputting a prediction result corresponding to the category of the current audio sample; The prototype loss function between the prediction result and the audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the predicted sample, and the random classifier is jointly tuned according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

2. The method according to claim 1, characterized in that Get the audio signal as audio samples, including: An original audio signal transmitted by an external audio device is obtained, and the original audio signal is processed by using a mel-spectrogram to obtain an audio signal, and the audio signal is used as an audio sample.

3. The method according to claim 2, characterized in that Reconstructing the audio sample by using a masking technique to generate a masked audio sample, and obtaining audio features of the masked audio sample by using a feature extractor, including: The audio sample is reconstructed by using a masking technique to generate a masked audio sample, and the audio features of the embedding layer are extracted from the logarithmic Mel spectrogram of the masked audio sample by a feature extractor to obtain the masked audio features.

4. The method according to claim 3, characterized in that Inputting the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample includes: The audio features are input into the Dropout random classifier, and the Monte Carlo dropout algorithm is used to define the forward propagation mode of each hidden layer in the Dropout random classifier: Among them, y h is the audio feature output by the hidden layer h in the random classifier, is the Hadamard product, w h is the importance weight, σ is the activation function, is a vector with the same dimension as the number of neurons in the hidden layer h-1, and the jth element is set to 0; y h-1 is the audio feature output by the hidden layer h-1 in the random classifier, b h is the bias term; The audio features are classified using the Dropout operation according to the forward propagation method, and a prediction matrix corresponding to the category of the masked audio sample is output: Among them, P x is a prediction matrix consisting of K×A prediction results, x is an audio sample, x A is the Ath masked audio sample, is the K-th prediction result of the A-th masked audio sample, is the Kth prediction result of the 1st masked audio sample, K is the number of Dropout operations, and A is the number of mask augmentations.

5. The method according to claim 4, characterized in that Evaluating the uncertainty value of the prediction matrix based on the nuclear norm to obtain the importance weight corresponding to the category includes: The uncertainty of the prediction matrix is ​​evaluated based on the nuclear norm to obtain the uncertainty value: Among them, U is the uncertainty value, P is the prediction matrix, tr(.) is the trace of the prediction matrix, σ j is the jth largest singular value of the prediction matrix, ‖P‖ * is the nuclear norm of the prediction matrix, σ j Dim = min (K × A, D) is; The importance weight corresponding to the category of each audio sample is defined according to the uncertainty value: Among them, B is the number of audio samples in each batch, U x is the uncertainty value of the audio sample x, and Wx is the importance weight of the audio sample x.

6. The method according to claim 5, characterized in that Inputting the audio sample and the newly added audio sample into the feature extractor to extract audio features to obtain a first audio feature, including: The mel-spectrogram of the audio sample and the mel-spectrogram sample of the newly added audio sample are input into the feature extractor. During the incremental training process, the loss function of the feature extractor is reconstructed according to the audio sample: Among them, x is the audio sample, y is the label corresponding to the audio sample, and f e is the feature extractor, N is the number of all categories seen up to the current incremental session, is the weight of the random classifier, f e (x) is the encoding vector of audio sample x, λ is a constant coefficient; The audio feature of the newly added audio sample is extracted according to the reconstructed loss function to obtain a first audio feature.

7. The method according to claim 6, characterized in that Classifying the first audio feature through a random classifier and outputting a prediction result corresponding to the category of the current audio sample, including: In an incremental session, the first audio feature is classified by a random classifier, and the prototype of the category of the previous batch of audio samples is concatenated with the encoding vector of the category of the current audio sample to train the random classifier. The score of each category in the first audio feature is calculated by the trained random classifier, and the prediction result corresponding to the category of the current audio sample is obtained according to the score.

8. The method according to claim 7, characterized in that The method further comprises: adjusting the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the prediction sample, and jointly tuning the random classifier according to the category prototype loss function of the prediction sample to obtain an optimized small sample class incremental learning model, including: The prototype loss function between the category prototype of the prediction result and the category prototype of the current audio sample is adjusted according to the importance weight to obtain the category prototype loss function of the prediction sample: Among them, L p is the category prototype loss function of the predicted sample, y is the label of the previous batch of audio samples, p c is the prototype of category c of the current audio sample in the incremental session, O is the number of old categories of the previous batch of audio samples, is the weight of the random classifier, N is the number of categories seen up to the current incremental session, and p o is the category prototype of the current audio sample in the base session; A joint loss function is constructed based on the category prototype loss function of the predicted sample and the true label of the current audio sample: in, is the joint loss function, is the reconstruction loss function, x is the audio signal, y is the label of the historical audio signal, fe is the feature extractor, N is the number of all categories seen up to the current incremental session, is the weight of the random classifier, fe(x) is the feature extractor, and λ is a constant coefficient; The model parameters of the random classifier are optimized according to the joint loss function to obtain an optimized small-sample class incremental learning model.

9. An uncertainty-guided few-sample incremental audio classification device, characterized in that: The device comprises: A data processing module, used for acquiring an audio signal as an audio sample; A prediction matrix acquisition module, used to reconstruct the audio sample by using a masking technique to generate a masked audio sample, obtain the audio features of the masked audio sample by a feature extractor, and then input the audio features into a Dropout random classifier to obtain a prediction matrix corresponding to the category of the masked audio sample; A weight labeling module, used for evaluating the uncertainty of the prediction matrix based on the nuclear norm to obtain the importance weights corresponding to the categories; A prediction result acquisition module, used to input the audio sample and the newly added audio sample into the feature extractor to extract audio features, obtain a first audio feature, classify the first audio feature through a random classifier, and output a prediction result corresponding to the category of the current audio sample; A classification model optimization module is used to adjust the prototype loss function between the prediction result and the audio sample according to the importance weight to obtain the category prototype loss function of the predicted sample, and jointly tune the random classifier according to the category prototype loss function of the predicted sample to obtain an optimized few-sample class incremental learning model.

Images