Audio classification method and apparatus based on semi-supervised class incremental learning

Abstract

The application relates to an audio classification method and device based on semi-supervised class incremental learning. The audio data to be classified is obtained, the audio data to be classified is input into a model obtained through task-by-task training on a data set corresponding to an ordered sequence task set based on a time sequence consistency regularization method and a loss function algorithm, and semi-supervised class incremental learning is performed to obtain the classification label of the audio data to be classified, thereby completing the classification of the audio. The time sequence consistency regularization method of the semi-supervised class incremental audio classification model is used to continuously update the parameters of the model, so that the matching degree between the model and the audio signal samples in the data set is higher and higher, thereby ensuring that the model is more and more accurate during classification. The audio samples generated in the loss function algorithm process of the semi-supervised class incremental audio classification model are stored, thereby effectively preventing the problem of catastrophic forgetting of the model during the learning process, and the stability of the model during audio classification is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio classification, and particularly relates to an audio classification method and device based on semi-supervised class-incremental learning. BACKGROUND

[0002] Audio data is a kind of unstructured binary stream in addition to the information such as encoding mode and sampling rate, and has the characteristics of large data volume, complex processing, high information correlation, etc., which makes the processing of audio signals more complex and brings great difficulties to audio classification and other applications. In the early stage, people tried to use manually crafted features and shallow architecture-based classifiers to classify audio, such as Gaussian mixture model, hidden Markov model and non-negative matrix factorization. With the progress of science and technology, new audio classification technology has emerged, that is, a deep neural network model is constructed to classify audio.

[0003] The current technical personnel in this field have made continuous efforts according to the construction technology of deep neural network model, but still encounter some technical problems in the attempt to classify audio. First, most of the traditional technologies assume that the audio classification task is based on a fixed vocabulary, but the number of audio target classes may not be determined in advance, and the system needs to be constantly obtained from a dynamic data stream, and new audio target classes need to be learned from a dynamic data stream, that is, incremental learning. Secondly, in the whole learning process, the traditional technology generally assumes that the input data is completely annotated, but in many actual application scenarios, most of the data are not annotated. At present, the research on incremental learning technology for image classification mainly focuses on replay-based, regularization-based and parameter isolation-based technologies, and there is no much research on catastrophic forgetting.

[0004] This directly leads to the problem that the traditional model construction method for audio classification cannot prevent the model from producing catastrophic forgetting in the learning process. SUMMARY

[0005] Based on the above technical problems, an audio classification method based on semi-supervised class-incremental learning and an audio classification device based on semi-supervised class-incremental learning are provided to solve the above problems.

[0006] An audio classification method based on semi-supervised class-incremental learning, the method comprising:

[0007] obtaining audio data to be classified;

[0008] calling a trained semi-supervised class-incremental audio classification model; the semi-supervised class-incremental audio classification model is obtained by training a deep neural network model item by item based on a time sequence consistency regularization method and a loss function algorithm using an ordered sequence task set and a data set corresponding to the sequence task set;

[0009] input the audio data to be classified into the semi-supervised class-incremental audio classification model for semi-supervised class-incremental learning to obtain a classification label of the audio data to be classified.

[0010] In one of the embodiments, the training process of the semi-supervised class-incremental audio classification model comprises:

[0011] obtaining a sequence task set and a data set corresponding to the sequence task set, initializing a deep neural network model and a replay sample buffer;

[0012] extracting a current data set corresponding to a current task in the sequence task set, calculating the current data set by using a loss function algorithm to obtain a total loss function, and storing samples corresponding to effective pseudo labels generated in the calculation process to the replay sample buffer; the total loss function comprises a supervised learning cross-entropy loss function and a semi-supervised cross-entropy loss function;

[0013] updating the gradient of the parameters of the deep neural network model by using the total loss function combined with a time consistency regularization method, completing the current training of the deep neural network model and obtaining new parameters;

[0014] after obtaining the new parameters, returning to the step of extracting the current data set corresponding to the current task in the sequence task set, calculating the current data set by using the loss function algorithm to obtain the total loss function, and performing the next data set training of the model;

[0015] after the deep neural network model completes the training of all sequence tasks in the sequence task set, obtaining the semi-supervised class-incremental audio classification model.

[0016] In one of the embodiments, the process of the loss function algorithm comprises:

[0017] converting the current data set from an audio signal into a mel spectrogram; the mel spectrogram comprises a first mel spectrogram corresponding to a label-free audio signal and a second mel spectrogram corresponding to a labeled audio signal;

[0018] inputting the first mel spectrogram into the deep neural network model to calculate a semi-supervised learning cross-entropy loss function;

[0019] inputting the second mel spectrogram into the deep neural network model to calculate a supervised learning cross-entropy loss function;

[0020] summing the semi-supervised learning cross-entropy loss function and the supervised cross-entropy loss function to obtain a total loss function.

[0021] In one of the embodiments, the process of inputting the first mel spectrogram into the deep neural network model for calculation comprises:

[0022] The first Mel-scale time-frequency diagram is subjected to weak enhancement and strong enhancement to obtain a weakly enhanced first Mel-scale time-frequency diagram and a strongly enhanced first Mel-scale time-frequency diagram;

[0023] The weakly enhanced first Mel-scale time-frequency diagram is input into a deep neural network model to obtain pseudo labels, and the pseudo labels generated by the deep neural network model when a high-confidence prediction is made are retained as valid pseudo labels;

[0024] The strongly enhanced first Mel-scale time-frequency diagram is input into the deep neural network model in combination with the valid pseudo labels to obtain a semi-supervised learning cross-entropy loss function.

[0025] In one embodiment, the process based on the time sequence consistency regularization method includes:

[0026] A set number of samples corresponding to the valid pseudo labels are extracted from the replay sample buffer.

[0027] The samples corresponding to the valid pseudo labels are input into the deep neural network model trained in the last iteration to obtain a first high-level feature vector value;

[0028] The samples corresponding to the valid pseudo labels are input into the deep neural network model trained in the current iteration to obtain a second high-level feature vector value;

[0029] A set number of loss functions are calculated based on the first high-level feature vector value and the second high-level feature vector value.

[0030] In one embodiment, the number of audio signals with labels in the data set is less than the number of audio signals without labels.

[0031] In one embodiment, the audio signals of different tasks in the sequence task set do not overlap.

[0032] An audio classification device based on semi-supervised class incremental learning includes:

[0033] A data acquisition module is configured to acquire audio data to be classified.

[0034] A model calling module is configured to call a semi-supervised class incremental audio classification model that has been trained.

[0035] An audio classification module is configured to input the audio data to be classified into the semi-supervised class incremental audio classification model for semi-supervised class incremental learning to obtain a classification label of the audio data to be classified.

[0036] In another aspect, a computer device is also provided, comprising a memory and a processor, the memory having stored thereon a computer program, the processor implementing the steps of the audio classification method based on semi-supervised class-incremental learning when executing the computer program.

[0037] In yet another aspect, a readable storage medium is also provided, having stored thereon a computer program, the program implementing the steps of the audio classification method based on semi-supervised class-incremental learning when executed by a processor.

[0038] The audio classification method and device based on semi-supervised class-incremental learning described above, by obtaining the audio data to be classified, inputting the audio data to be classified into the trained semi-supervised class-incremental audio classification model for comparison and matching, obtaining the classification label of the audio data to be classified, and completing the classification of the audio. Since the semi-supervised class-incremental audio classification model is obtained by training the deep neural network model on the data sets corresponding to the ordered sequence task set one by one, to ensure that the audio labels obtained by training in the semi-supervised class-incremental learning classification model are regular and ordered, which is conducive to the extraction and comparison of audio classification labels. In the training process of the semi-supervised class-incremental audio classification model, the loss function algorithm is used to obtain the loss function of the data sets corresponding to all sequence task sets, and the deep neural network model is continuously updated and iterated by using the loss function combined with the temporal consistency regularization method, which ensures that the semi-supervised class-incremental learning classification model can more accurately complete the classification of the audio data to be classified in actual application.

[0039] In this way, the semi-supervised learning process completes the information conversion of the audio to be classified, so that it conforms to the type of audio label matching in the semi-supervised class-incremental learning classification model, ensuring that the audio data to be classified can be quickly classified in the semi-supervised class-incremental learning classification model. In addition, since the loss function algorithm can also obtain and store the audio labels in the data sets corresponding to all sequence task sets in its processing process, it can ensure that the semi-supervised class-incremental learning classification model will not have the problem of catastrophic forgetting, so that the semi-supervised class-incremental learning classification model can more stably classify the audio data to be classified. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0041] Figure 1 A flowchart of the audio classification method based on semi-supervised class-incremental learning in an embodiment;

[0042] Figure 2 A flowchart of a training process of a semi-supervised class-incremental audio classification model in one embodiment;

[0043] Figure 3 A flowchart of a loss function algorithm process in one embodiment;

[0044] Figure 4 A flowchart of a process of inputting a first mel-scale time-frequency map into a deep neural network model for calculation in one embodiment;

[0045] Figure 5 A flowchart of a process of a temporal consistency regularization method in one embodiment;

[0046] Figure 6 A schematic diagram of a module structure of an audio classification device based on semi-supervised class-incremental learning in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application.

[0048] It should be noted that the term "embodiment" mentioned herein means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The phrase is shown at various places in the specification does not necessarily refer to the same embodiment, nor is it independent or alternative to other embodiments. Those skilled in the art can understand that the embodiments described herein can be combined with other embodiments. The term "and / or" used in the specification and the appended claims means one or more of the associated listed terms in any combination and all possible combinations, and includes these combinations.

[0049] The embodiments of the present application will be described in detail below with reference to the accompanying drawings of the embodiments of the present application.

[0050] In the research of the present application, the inventors found that in the traditional audio classification model, a fixed vocabulary is used and insufficient attention is paid to the problem of catastrophic forgetting, resulting in that the traditional audio classification model has limitations in audio classification and is unstable, and needs to relearn from dynamic data streams to obtain audio categories, thereby causing long time consumption, inaccurate classification or inability to classify in audio classification.

[0051] In one embodiment, as shown in Figure 1 a semi-supervised class-incremental learning-based audio classification method is provided, which comprises the following steps:

[0052] Step 102, obtaining audio data to be classified.

[0053] In this step, audio data is obtained from the outside world, usually by implanting a trained semi-supervised class-incremental audio classification model in a certain software application, and inputting audio data to be classified from the input port of the software application. For example, in daily life, a music software application has a function of listening to songs and recognizing songs, and using this function can receive audio data from the outside world.

[0054] Step 104, calling a trained semi-supervised class-incremental audio classification model; the semi-supervised class-incremental audio classification model is obtained by training a deep neural network model item by item based on a time consistency regularization method and a loss function algorithm using an ordered sequence task set and a data set corresponding to the sequence task set.

[0055] In this step, the ordered sequence task set is a set of tasks with sequence characteristics, which can be time sequence or data set size inverse sequence. The size and arrangement order of the ordered sequence task set can be set by the staff in advance. When training with a deep neural network model, each sequence task is trained in strict accordance with the arrangement order of each sequence task in the ordered sequence task set, and each sequence task corresponds to a data set, so as to ensure the ordered training of the deep neural network model. The data set generally includes a small amount of labeled samples and a large amount of unlabeled samples. The loss function algorithm can calculate all the labeled samples and unlabeled samples in the data set to obtain the loss function and the samples, so as to depict the matching degree of the samples generated in the deep neural network model and the samples in the data set, and store the samples. The stored audio samples can be used as historical data to continuously optimize and update the training process, and can also be used as traceability data to effectively prevent catastrophic forgetting in the learning process of the model. The time consistency regularization method iteratively optimizes the training of the current deep neural network model using historical data, so as to ensure that the matching degree of the samples generated in the above deep neural network model and the samples in the data set is getting higher and higher, thereby ensuring more and more accurate classification of audio.

[0056] Step 106, input the audio data to be classified into the semi-supervised class incremental audio classification model for semi-supervised class incremental learning, to obtain the classification label of the audio data to be classified.

[0057] In this step, the semi-supervised class incremental learning classifies the audio data to be classified by signal enhancement, and then calculates the classification label of the audio data to be classified by the loss function of the semi-supervised class incremental audio classification model. When the comparison result is matched, the classification of the audio data to be classified is completed. When the comparison result is not matched, the obtained classification label is stored to achieve the purpose of incremental learning. It is worth noting that the classification label can be a feature set of the audio data, which can be used to represent the type of the audio data.

[0058] In the above audio classification method based on semi-supervised class incremental learning, the data sets corresponding to the sequence task set are trained one by one according to the arrangement order in the ordered sequence task set, to ensure that the whole training process is accurate and efficient. The loss function of the semi-supervised class incremental audio classification model is used to obtain the loss function of the data set corresponding to all sequence task sets. The loss function is combined with the time consistency regularization method to continuously update and iterate the deep neural network model, so that the semi-supervised class incremental learning classification model can more accurately complete the classification of the audio data to be classified. The classification label of the audio data to be classified is calculated by the loss function of the semi-supervised class incremental audio classification model, and then the obtained classification label is compared with the classification label in the semi-supervised class incremental audio classification model. When the comparison result is matched, the classification of the audio data to be classified is completed. When the comparison result is not matched, the obtained classification label is stored to achieve the purpose of incremental learning. The loss function algorithm of the semi-supervised class incremental audio classification model can also obtain and store the audio labels in the data set corresponding to all sequence task sets in the process of the algorithm, so that the semi-supervised class incremental learning classification model will not have the problem of catastrophic forgetting, and the semi-supervised class incremental learning classification model can more stably classify the audio data to be classified.

[0059] In one embodiment, the training process of the semi-supervised class incremental audio classification model includes:

[0060] Step 202: Obtain the sequence task set and the data set corresponding to the sequence task set, initialize the deep neural network model and the replay sample buffer.

[0061] In this step, the replay sample buffer is used to store the updated model parameters obtained after each training, has the characteristics of fast reading data, ensures the speed of model training, and at the same time, the updated model parameters stored therein provide reliable data support for the subsequent process of continuous iterative updating of the deep neural network model. It is worth noting that, since the training process of the deep neural network model is orderly and comprehensive, the efficiency of the training process can be guaranteed, and the problem of catastrophic forgetting can be solved.

[0062] Step 204: Extract the current data set corresponding to the current task in the sequence task set, calculate the total loss function using the loss function algorithm on the current data set, and store the samples corresponding to the effective pseudo label generated in the calculation process to the replay sample buffer; the total loss function includes the supervised learning cross-entropy loss function and the semi-supervised cross-entropy loss function.

[0063] The current task and the current data set in this step are based on the current training, and with the advancement of the training round, the current task and the current data set can change according to the pre-arranged order in the ordered sequence task set. One training extracts only the data set represented by one sequence task, i.e. the current data set, to ensure the ordered and efficient training. It is worth noting that since the audio signals in the data set are often mixed with labeled audio signals and unlabeled audio signals, the current data set needs to be classified and calculated when using the loss function algorithm, i.e. using labeled audio signals to calculate the supervised learning cross-entropy loss function and using unlabeled audio signals to calculate the semi-supervised cross-entropy loss function. The calculation process usually converts the original audio signal into a form that can be read by the deep data network model, and then uses the audio data in this form to train the deep data network model, so that the deep data network model retains the classification label of the audio signal. The effective pseudo label in this step is obtained in the process of calculating the unlabeled audio signal. Since the unlabeled audio signal is not completely annotated, it needs to be processed to obtain a form that can be operated on the loss function during the calculation process, so that the semi-supervised cross-entropy loss function can be obtained, which is also the fundamental of the semi-supervised incremental learning audio classification model to process unlabeled samples in the classification audio data. Different audio signals in the current data set are classified and calculated to obtain loss functions of different types of audio signals, keeping the calculation process rigorous and making the classification labels in the trained model more accurate and reliable.

[0064] Step 206: Update the parameters of the deep neural network model using the total loss function combined with the time sequence consistency regularization method, complete the current training of the deep neural network model and obtain new parameters.

[0065] The total loss function in this step is the sum of the supervised learning cross-entropy loss function and the semi-supervised cross-entropy loss function, that is, the distortion rate of all kinds of audio signal classification labels obtained by training the deep neural network model on all audio signals in the current data set. The temporal consistency regularization method iteratively optimizes the training of the current deep neural network model using historical data to ensure that the deep neural network model becomes more and more accurate in the subsequent training process. Gradient update is to calculate the loss function of the part of audio data by using the deep neural network model to extract part of the unlabeled audio data in the current data set. It is worth noting that the deep neural network model is trained for the first time, so only part of the unlabeled audio data in the first training is used in the first gradient update.

[0066] Step 208: After obtaining the new parameters, return to the step of calculating the total loss function by using the loss function algorithm on the current data set corresponding to the current task in the extraction sequence task set to perform the next data set training of the model.

[0067] This step is a loop step. It is worth noting that since the first training of the deep neural network model has been completed, the loop step will combine the loss function of the last gradient update and the total loss function of the current time to perform gradient update, which can ensure that the new parameters obtained by each update iteration are in the direction of more accurate iteration.

[0068] Step 210: When the deep neural network model completes the training of all sequence tasks on the sequence task set, a semi-supervised incremental audio classification model is obtained.

[0069] The final deep neural network model obtained in this step is the semi-supervised incremental audio classification model.

[0070] The training of the above semi-supervised incremental audio classification model uses the replay sample buffer with fast reading characteristics to efficiently train the deep neural network model; by classifying and calculating the audio samples in the data set corresponding to the sequence task set, the integrity of the classification labels in the model is ensured, making the deep neural network model more applicable. At the same time, by storing the samples corresponding to the effective pseudo-labels, the problem of catastrophic forgetting generated by the model during learning is solved, making the semi-supervised incremental audio classification model obtained by training more stable; by setting the replay sample buffer to store and retain the classification labels, the temporal consistency regularization method is used to continuously update and iterate the classification labels, making the accuracy of the classification labels higher, which can make the deep neural network model reliable.

[0071] In one embodiment, the process of the loss function algorithm includes:

[0072] Step 302: converting the current data set from the audio signal to the mel spectrogram; the mel spectrogram includes a first mel spectrogram corresponding to the unlabeled audio signal and a second mel spectrogram corresponding to the labeled audio signal.

[0073] In this step, the data form in the current data set is converted from the audio signal to the mel spectrogram, which aims to filter the audio signal that interferes with the judgment, and obtain a pure audio signal for feature extraction of the deep neural network model. Since the audio signal includes unlabeled audio signal and labeled audio signal, the conversion process is also divided into mel spectrogram corresponding to the unlabeled audio signal and mel spectrogram corresponding to the labeled audio signal. In this way, the accuracy of the audio signal during data transmission can be ensured, and data distortion can be effectively reduced.

[0074] Step 304: inputting the first mel spectrogram into the deep neural network model to calculate the semi-supervised learning cross-entropy loss function.

[0075] In this step, because the unlabeled audio signal is audio data that has not been completely annotated, the unlabeled audio signal needs to be processed first. This allows the unlabeled audio signal to obtain an audio label in a special form in the semi-supervised incremental audio classification model. This special form of audio label can be used to classify the unlabeled audio signal, which is closer to real-life scenarios and is conducive to the widespread use of the semi-supervised incremental audio classification model.

[0076] Step 306: inputting the second mel spectrogram into the deep neural network model to calculate the supervised learning cross-entropy loss function.

[0077] The calculation process in this step is to first input the mel spectrogram of the labeled audio signal into the deep neural network model to obtain the high-level feature vector value of the audio signal, and then combine the label value of the labeled audio signal itself to calculate the cross-entropy loss. Finally, the supervised learning cross-entropy loss function is obtained. Although the labeled audio signal only accounts for a small proportion in real-life scenarios, it still needs to be calculated to ensure the integrity of the final semi-supervised incremental audio classification model.

[0078] Step 308: summing the supervised learning cross-entropy loss function and the semi-supervised cross-entropy loss function to obtain the total loss function.

[0079] In this step, the total loss function is a sequence task representative of the distorted audio classification label of the entire audio signal in the data set during the training process when converting to the corresponding audio classification label. It immediately depicts the matching degree of the deep neural network model and the audio signal sample in the data set. The smaller this value is, the higher the matching degree is, and the more reliable the model obtained by training is.

[0080] The algorithm of the loss function described above converts the data form in the current data set from an audio signal to a Mel-frequency spectrogram, filters out audio frequency bands that interfere with classification, obtains a pure audio signal, reduces the amount of calculation in the training process, and makes the calculation result more accurate and reliable. By classifying and calculating different types of audio signals, the results obtained are more effective and reliable, thereby improving the reliability of the semi-supervised incremental audio classification model. By applying signal enhancement combined with the semi-supervised loss function algorithm, the semi-supervised incremental audio classification model is more suitable for real-world scenarios.

[0081] In one embodiment, the first Mel-frequency spectrogram is input into a deep neural network model calculation process, including:

[0082] Step 402, signal weak enhancement and strong enhancement are performed on the first Mel-frequency spectrogram to obtain a weakly enhanced first Mel-frequency spectrogram and a strongly enhanced first Mel-frequency spectrogram.

[0083] In this step, the first Mel-frequency spectrogram is corresponded to the unlabeled audio signal, and the unlabeled audio signal is audio data without complete annotation. Therefore, the first Mel-frequency spectrogram needs to be weakly enhanced and strongly enhanced first. The role of weak enhancement and strong enhancement is to extract audio features in the unlabeled audio signal, so that the deep neural network model can obtain effective audio information through training, and then obtain effective audio classification labels.

[0084] Step 404, the weakly enhanced first Mel-frequency spectrogram is input into a deep neural network model to obtain pseudo labels, and the pseudo labels generated by the deep neural network model when producing a high-confidence prediction are reserved as effective pseudo labels.

[0085] In this step, the pseudo label can represent some indeterminate audio features in the unlabeled audio signal, and the role is to enable the deep neural network model to learn some hidden audio features in the unlabeled audio signal. The high-confidence prediction is a prediction produced by the deep neural network model when learning the above-mentioned hidden audio features. It is worth noting that the prerequisite for producing a high-confidence prediction is that part of the hidden audio features can represent the unlabeled audio signal to a high degree. This type of label is referred to as an effective pseudo label in this step.

[0086] Step 406, the strongly enhanced first Mel-frequency spectrogram is input into a deep neural network model to calculate a semi-supervised learning cross-entropy loss function in combination with the effective pseudo label.

[0087] In this step, the audio label obtained by combining strong enhancement is compared with the effective pseudo label, and whether the hidden audio features of the unlabeled audio signal can represent the unlabeled audio signal is verified again to ensure the accuracy of the semi-supervised learning cross-entropy loss function obtained by calculation.

[0088] In this embodiment, the hidden audio features in the audio signal are fully mined by weak enhancement and strong enhancement of the signal of the first Mel-scale time-frequency diagram, and the effective pseudo-labels are obtained by predicting the audio features through the deep neural network model. Finally, the audio labels obtained by the strong enhancement are verified to obtain the audio signals that can be used for training and learning of the deep neural network model.

[0089] In one embodiment, the process based on the timing consistency regularization method comprises:

[0090] Step 502: Extracting a set number of valid pseudo-label corresponding samples of different types of tasks in the replay sample buffer.

[0091] The purpose of this step is to make the unlabeled samples stored in the replay sample buffer be the minimum number of unlabeled samples, containing each type of unlabeled audio signal, so that the calculation is the lowest under the premise of accuracy in the process of gradient update, thereby ensuring the speed of training. It is worth noting that the valid pseudo-label corresponding sample is generally an unlabeled sample,

[0092] Step 504: Inputting the valid pseudo-label corresponding sample into the deep neural network model of the last training to calculate the first high-level feature vector value.

[0093] In this step, the first high-level feature vector value is the audio feature of the classification label obtained after the last deep neural network model training.

[0094] Step 506: Inputting the valid pseudo-label corresponding sample into the deep neural network model of the current training to calculate the second high-level feature vector value.

[0095] In this step, the second high-level feature vector value is the audio feature of the classification label obtained after the current deep neural network model training.

[0096] Step 508: Calculating a set number of loss functions according to the first high-level feature vector value and the second high-level feature vector value.

[0097] The calculation of this step is to compare the absolute values of the first high-level feature vector value and the second high-level feature vector value, and the difference between the absolute values of the two is taken as the set number of loss functions. It is intended to correct the deep neural network model of the current training, and at the same time obtain an accurate model parameter to provide reliable historical data for the gradient update of the next deep neural network model training.

[0098] The embodiment extracts the minimum number of unlabeled samples, so that the calculation is the lowest under the premise of accuracy in the process of gradient update, and the speed of gradient update is ensured; the difference between the absolute values of the first high-level feature vector value and the second high-level feature vector value is used as the set number loss function, the deep neural network model of the current training is corrected, and reliable data support is also provided for subsequent gradient update.

[0099] In one embodiment, the number of labeled audio signals in the data set is less than the number of unlabeled audio signals.

[0100] The embodiment aims to simulate a real situation. In the real situation, when performing audio classification, the provided original audio signals are mostly unlabeled audio signals, and the labeled audio signals only account for a small part. Therefore, when designing the audio signals in the data set, the real scene is simulated, and the number of unlabeled audio signals is designed to be greater than the number of labeled audio signals. In this way, the applicability of the trained semi-supervised class-incremental audio classification model can be fundamentally improved.

[0101] In one embodiment, the audio signals of different tasks in the sequence task set do not overlap.

[0102] The embodiment aims to illustrate that the same audio signal will not be repeatedly updated during the training process, reducing the calculation amount of the deep neural network model during training, thereby ensuring the training speed.

[0103] It should be understood that, although Figure 1 The steps in the flowchart of the embodiment are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in the embodiment can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0104] Please refer to Figure 2In one embodiment, an audio classification device 100 based on semi-supervised class-incremental learning is provided, comprising a data acquisition module 11, a model calling module 12 and an audio classification module 13. Among them: the data acquisition module 11 is used to acquire audio data to be classified. The model calling module 12 is used to call a trained semi-supervised class-incremental audio classification model; the semi-supervised class-incremental audio classification model is obtained by training a deep neural network model item by item based on a time sequence consistency regularization method and a loss function algorithm on a data set corresponding to an ordered sequence task set. The audio classification module 13 is used to input the audio data to be classified into the semi-supervised class-incremental audio classification model for semi-supervised class-incremental learning to obtain the classification label of the audio data to be classified.

[0105] The above-mentioned audio classification device 100 based on semi-supervised class-incremental learning, through the cooperation of each module, by acquiring audio data to be classified, inputting the audio data to be classified into the trained semi-supervised class-incremental audio classification model for comparison and matching, obtaining the classification label of the audio data to be classified, and completing the classification of the audio. Since the semi-supervised class-incremental audio classification model is obtained by training a deep neural network model item by item on a data set corresponding to an ordered sequence task set, to ensure that the audio labels obtained by training in the semi-supervised class-incremental learning classification model are regular and ordered, which is conducive to the extraction and comparison of audio classification labels. In the training process of the semi-supervised class-incremental audio classification model, the loss function of the data set corresponding to all sequence task sets is obtained by using the loss function algorithm, and the deep neural network model is continuously updated and iterated by using the loss function combined with the time sequence consistency regularization method, which ensures that the semi-supervised class-incremental learning classification model can more accurately complete the classification of the audio data to be classified in actual application.

[0106] In this way, the semi-supervised learning process completes the information conversion of the audio to be classified, so that it conforms to the type of audio label matching in the semi-supervised class-incremental learning classification model, ensuring that the audio data to be classified can be quickly classified in the semi-supervised class-incremental learning classification model; in addition, since the loss function algorithm can also obtain and store the audio labels in the data set corresponding to all sequence task sets in its processing process, it can ensure that the semi-supervised class-incremental learning classification model will not have the problem of catastrophic forgetting, so that the semi-supervised class-incremental learning classification model can more stably classify the audio data to be classified.

[0107] In one embodiment, the modules of the above-mentioned audio classification device 100 based on semi-supervised class-incremental learning can also be used to realize the corresponding processing steps of other embodiments of the above-mentioned audio classification method based on semi-supervised class-incremental learning.

[0108] As to the specific definition of the audio classification device 100 based on semi-supervised class-incremental learning, reference can be made to the corresponding definition of the audio classification method based on semi-supervised class-incremental learning in the foregoing, which will not be repeated here. Each module in the audio classification device 100 based on semi-supervised class-incremental learning described above can be realized by software, hardware, and a combination thereof, in whole or in part. The modules described above can be embedded in or independent of a device with a specific data processing function in the form of hardware, or stored in the memory of the aforementioned device in the form of software, so that the processor invokes and executes the operations corresponding to each of the modules. The aforementioned device can be, but is not limited to, various types of portable data analysis and processing devices in the art.

[0109] In one embodiment, a computer device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following processing steps when executing the computer program:

[0110] obtaining audio data to be classified; calling a trained semi-supervised class-incremental audio classification model; the semi-supervised class-incremental audio classification model being obtained by training a deep neural network model item by item based on a temporal consistency regularization method and a loss function algorithm, using an ordered sequence task set and a data set corresponding to the sequence task set; inputting the audio data to be classified into the semi-supervised class-incremental audio classification model for semi-supervised class-incremental learning, to obtain a classification label of the audio data to be classified.

[0111] It can be understood that, in addition to the memory and the processor described above, the computer device described above also includes other software and hardware components not listed in the specification, which can be determined according to the specific data processing device model in different application scenarios, and will not be listed in detail in the specification.

[0112] In one embodiment, the processor, when executing the computer program, can also implement the steps or sub-steps added in each embodiment of the audio classification method based on semi-supervised class-incremental learning described above.

[0113] In one embodiment, a computer readable storage medium is also provided, which stores a computer program, and the computer program, when executed by a processor, implements the following processing steps:

[0114] obtaining audio data to be classified; calling a trained semi-supervised class-incremental audio classification model; the semi-supervised class-incremental audio classification model being obtained by training a deep neural network model item by item based on a temporal consistency regularization method and a loss function algorithm, using an ordered sequence task set and a data set corresponding to the sequence task set; inputting the audio data to be classified into the semi-supervised class-incremental audio classification model for semi-supervised class-incremental learning, to obtain a classification label of the audio data to be classified.

[0115] In one embodiment, the computer program, when executed by the processor, further implements the steps or sub-steps added in each of the above-mentioned embodiments of the audio classification method based on semi-supervised class incremental learning.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, RDRAM) and interface dynamic random access memory (DRDRAM).

[0117] Each technical feature of the above-mentioned embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of each technical feature in the above-mentioned embodiments are not described, however, as long as the combination of technical features does not exist, it should be considered as the scope of the present application.

[0118] The above-mentioned embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. An audio classification method based on semi-supervised incremental learning, characterized in that, The method includes: Obtain the audio data to be categorized; The trained semi-supervised incremental audio classification model is invoked; the semi-supervised incremental audio classification model is obtained by training a deep neural network model task by task using an ordered sequence task set and the dataset corresponding to the sequence task set, based on the temporal consistency regularization method and loss function algorithm; The audio data to be classified is input into the semi-supervised incremental audio classification model for semi-supervised incremental learning to obtain the classification label of the audio data to be classified. The training process of the semi-supervised incremental audio classification model includes: Obtain the sequence task set and the dataset corresponding to the sequence task set, and initialize the deep neural network model and the replay sample buffer pool; Extract the current dataset corresponding to the current task from the sequence task set, calculate the total loss function on the current dataset using the loss function algorithm, and store the samples corresponding to the valid pseudo-labels generated during the calculation process into the replay sample buffer pool; the total loss function includes the supervised learning cross-entropy loss function and the semi-supervised cross-entropy loss function; The total loss function combined with the temporal consistency regularization method is used to update the gradient of the parameters of the deep neural network model, thereby completing the current training of the deep neural network model and obtaining new parameters. After obtaining the new parameters, the process returns to the step of extracting the current dataset corresponding to the current task in the sequence task set, calculating the total loss function on the current dataset using the loss function algorithm, and then training the model on the next dataset. After the deep neural network model has completed training on all sequence tasks in the set of sequence tasks, the semi-supervised incremental audio classification model is obtained.

2. The audio classification method based on semi-supervised incremental learning according to claim 1, characterized in that, The process of the loss function algorithm includes: The current dataset is converted from audio signals into a Megel time-frequency plot; the Megel time-frequency plot includes a first Megel time-frequency plot corresponding to unlabeled audio signals and a second Megel time-frequency plot corresponding to labeled audio signals; The first Meger time-frequency plot is input into the deep neural network model to calculate the semi-supervised learning cross-entropy loss function; The second Meger time-frequency plot is input into the deep neural network model to calculate the supervised learning cross-entropy loss function; The total loss function is obtained by summing the semi-supervised learning cross-entropy loss function and the supervised cross-entropy loss function.

3. The audio classification method based on semi-supervised incremental learning according to claim 2, characterized in that, The process of inputting the first Meger time-frequency map into the deep neural network model for calculation includes: The first Meger time-frequency plot is subjected to weak signal enhancement and strong signal enhancement to obtain a weakly enhanced first Meger time-frequency plot and a strongly enhanced first Meger time-frequency plot. The weakly enhanced first Meger time-frequency map is input into the deep neural network model to predict pseudo-labels, and the pseudo-labels generated by the deep neural network model when it produces high-confidence predictions are retained as valid pseudo-labels. The strongly enhanced first Meger time-frequency map is input into the deep neural network model and combined with the effective pseudo-labels to calculate the semi-supervised learning cross-entropy loss function.

4. The audio classification method based on semi-supervised incremental learning according to claim 1, characterized in that, The process of the time-series consistency regularization method includes: Extract a set number of samples corresponding to the valid pseudo-labels from the replay sample buffer pool for different types of tasks; The sample corresponding to the effective pseudo-label is input into the previously trained deep neural network model to calculate the first high-level feature vector value; The sample corresponding to the effective pseudo-label is input into the deep neural network model of the current training to calculate the second high-level feature vector value; The set quantity loss function is calculated based on the first high-level feature vector value and the second high-level feature vector value.

5. The audio classification method based on semi-supervised incremental learning according to claim 1, characterized in that, The dataset contains fewer labeled audio signals than unlabeled audio signals.

6. The audio classification method based on semi-supervised incremental learning according to claim 1, characterized in that, The audio signals of different tasks in the sequence task set do not overlap.

7. An audio classification device based on semi-supervised incremental learning, characterized in that, include: The data acquisition module is used to acquire the audio data to be classified. The model calling module is used to call the trained semi-supervised incremental audio classification model. The semi-supervised incremental audio classification model is obtained by training a deep neural network model task by task using an ordered sequence task set and the dataset corresponding to the sequence task set, based on the temporal consistency regularization method and loss function algorithm. The audio classification module is used to input the audio data to be classified into the semi-supervised incremental audio classification model for semi-supervised incremental learning, and obtain the classification label of the audio data to be classified. The training process of the semi-supervised incremental audio classification model includes: Obtain the sequence task set and the dataset corresponding to the sequence task set, and initialize the deep neural network model and the replay sample buffer pool; Extract the current dataset corresponding to the current task from the sequence task set, calculate the total loss function on the current dataset using the loss function algorithm, and store the samples corresponding to the valid pseudo-labels generated during the calculation process into the replay sample buffer pool; the total loss function includes the supervised learning cross-entropy loss function and the semi-supervised cross-entropy loss function; The total loss function combined with the temporal consistency regularization method is used to update the gradient of the parameters of the deep neural network model, thereby completing the current training of the deep neural network model and obtaining new parameters. After obtaining the new parameters, the process returns to the step of extracting the current dataset corresponding to the current task in the sequence task set, calculating the total loss function on the current dataset using the loss function algorithm, and then training the model on the next dataset. After the deep neural network model has completed training on all sequence tasks in the set of sequence tasks, the semi-supervised incremental audio classification model is obtained.

8. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the audio classification method based on semi-supervised incremental learning as described in any one of claims 1-6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the audio classification method based on semi-supervised incremental learning as described in any one of claims 1-6.

Images