Sound event classification model training method, electronic equipment, storage medium and computer program product
By using two models of the network with the same structure and adjusting parameters through prediction loss, the problem of the model's classification accuracy of the old category when introducing new sound event categories is solved, and a high-accurate sound event classification is achieved.
Patent Information
- Application Number
- CN202411627722.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-25
AI Technical Summary
When introducing new sound event categories, the prior art tends to cause the model to reduce the classification accuracy of sound events in the old category, which is called the knowledge forgetting problem.
By using two models of the feature extraction network with the same structure, one of the model's classifiers adds output units for the new category and adjusts model parameters by predicting losses, maintaining the model's classification performance for the old category while learning the new category.
It effectively avoids the problem of knowledge forgetting, improves the classification accuracy of new categories, and allows feature extraction networks to be adjusted and optimized based on new categories data.
Smart Images
Figure CN119517083A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and in particular to a training method for a sound event classification model, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] Sound event classification refers to the process of predicting the sound event category (such as gun, whistle, wind, etc.) to which an audio signal belongs given an audio signal. With the development of deep learning technology, the task of sound event classification has been widely used. There are many word segments for audio sound event categories, and the same audio may belong to multiple categories. After training and deploying a sound event classification model using data of limited categories, it may be necessary to introduce new categories of sound events in subsequent applications through methods such as model fine-tuning. However, after adding new sound event categories, the category knowledge that the model has already learned will be catastrophically forgotten, and the classification accuracy of the model that introduces new categories for sound events of old categories will be seriously reduced. This problem occurs because it is difficult for the model to learn new categories without losing the accuracy of known categories.
[0003] Related solutions generally use feature extraction methods to solve this problem: use audio data of a specified category to train a model that includes a feature extraction network and a classifier. After the model training is completed, obtain the audio data of the new category that needs to be introduced. Fix the parameters of the feature extraction network in the trained model, and use the audio data of the new category to train the model so that the classifier of the retrained model can classify the new category. This method can retain the feature representation of the old category and reduce the problem of knowledge forgetting caused by the introduction of new category data.
[0004] However, this approach still has some problems: since the feature extractor is frozen in the initial stage when introducing new category data, the model cannot update the parameters of the feature extractor to adapt to the new category data, which may lead to insufficient feature representation of the new category and affect the classification performance of the new category. Frozen feature extractors mean that the model's feature representation capabilities are fixed and cannot be adjusted and optimized according to new category data. This may lead to insufficient feature representation when facing diverse and complex audio data, and then the accuracy is low when using the feature representation of new category data to perform classification tasks. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a training method for a sound event classification model and an electronic device, a storage medium, and a computer program product, which are used to avoid the problem of knowledge forgetting caused by introducing new category data and improve the classification accuracy of new categories.
[0006] On the one hand, the present application provides a method for training a sound event classification model, comprising:
[0007] S1: Input the audio features of multiple audio data in the specified data set into the first original model and the second original model respectively, and obtain the first predicted category output by the first original model and the second predicted category output by the second original model; wherein the specified data set includes original category data, new category data and unsupervised data; the first original model and the second original model have the same feature extraction network structure, the classifier of the second original model adds an output unit of the new category compared with the classifier of the first original model, and the second original model and the first original model have the same initialization parameters;
[0008] S2: determining a prediction loss according to the first prediction category, the second prediction category, the category label of the original category data, and the category label of the new category data, and adjusting the parameters of the second original model based on the prediction loss;
[0009] S3: Return to step S1, and after multiple iterations, use the converged second original model as the sound event classification model.
[0010] In one embodiment, before step S1, the method further includes:
[0011] Performing a fast Fourier transform on the audio data in the specified data set to obtain a transformation result;
[0012] Performing Mel filter bank processing on the transformation result to obtain a processing result;
[0013] The logarithm of the processing result is taken, and discrete cosine transform is performed on the logarithm to obtain the audio features of the audio data.
[0014] In one embodiment, before step S1, the method further includes:
[0015] Inputting audio features of multiple audio data in the original data set into a deep learning model to obtain a third predicted category output by the deep learning model; wherein the original data set includes original category data; and the deep learning model includes a feature extraction network and a classifier;
[0016] Determining a prediction loss according to the third predicted category and the category label of the original category data, and adjusting parameters of the deep learning model based on the prediction loss;
[0017] Repeat the above process for multiple iterations, and use the converged deep learning model as the first original model;
[0018] Based on the first original model, a second original model including a new category output unit is generated.
[0019] In one embodiment, the specified data set is constructed in the following manner:
[0020] Uniformly and randomly extracting original category data of each category from the original data set to form an original data subset;
[0021] The original data subset, the preset new category data and the preset unsupervised data constitute the specified data set.
[0022] In one embodiment, determining the prediction loss according to the first predicted category, the second predicted category, the category label of the original category data, and the category label of the new category data includes:
[0023] Determine a first sub-loss according to the first prediction result and the second prediction result of the original category data;
[0024] Determine a second sub-loss according to the first prediction result and the second prediction result of the unsupervised data;
[0025] Determine a third sub-loss according to the second prediction result of the original category data, the category label of the original category data, the second prediction result of the new category data, and the category label of the new category data;
[0026] The predicted loss is determined according to the first sub-loss, the second sub-loss and the third sub-loss.
[0027] In one embodiment, determining the first sub-loss according to the first prediction result and the second prediction result of the original category data includes:
[0028] Calculating a relative entropy loss as a first sub-loss according to the first prediction result and the second prediction result of the original category data;
[0029] The determining the second sub-loss according to the first prediction result and the second prediction result of the unsupervised data comprises:
[0030] According to the first prediction result and the second prediction result of the unsupervised data, the mean square error loss is calculated as the second sub-loss;
[0031] The determining the third sub-loss according to the second prediction result of the original category data, the category label of the original category data, the second prediction result of the new category data, and the category label of the new category data includes:
[0032] Calculating a first cross entropy loss according to the second prediction result and the category label of the original category data;
[0033] Calculating a second cross entropy loss according to the second prediction result and the category label of the new category data;
[0034] A third sub-loss is determined according to the first cross entropy loss and the second cross entropy loss.
[0035] In one embodiment, the method further comprises:
[0036] The audio data to be classified is preprocessed to obtain corresponding audio features, and the audio features are input into the sound event classification model to obtain the sound event category information output by the sound event classification model.
[0037] On the other hand, the present application provides an electronic device, the electronic device comprising:
[0038] processor;
[0039] a memory for storing processor-executable instructions;
[0040] Wherein, the processor is configured to execute the training method of the above-mentioned sound event classification model.
[0041] In addition, the present application provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete the training method of the above-mentioned sound event classification model.
[0042] Furthermore, the present application provides a computer program product, including a computer program / instruction, which implements the above-mentioned training method of the sound event classification model when executed by a processor.
[0043] The present application scheme calculates the prediction loss through the first prediction result output by the first original model, which can constrain the prediction difference between the new model (second original model) and the old model (first original model) on the original category, and introduces unsupervised data to constrain the prediction difference between the new model (second original model) and the old model (first original model) on the unknown category data, maintain the consistency of the first original model and the second original model, alleviate the problem of the new model forgetting the original category knowledge, and maintain the classification performance of the new model for the original category. In addition, this process does not require the parameters of the feature extraction network to be fixed, so that the feature extraction network after training can be adjusted and optimized according to the new category data, ensuring that its feature representation ability meets the classification requirements. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.
[0045] Figure 1A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;
[0046] Figure 2 A flowchart of a method for training a sound event classification model provided in one embodiment of the present application;
[0047] Figure 3 A schematic diagram of a flow chart of a method for extracting audio features provided in an embodiment of the present application;
[0048] Figure 4 A schematic diagram of a flow chart of a method for generating an original model provided in an embodiment of the present application;
[0049] Figure 5 A schematic diagram of a flow chart of a method for calculating predicted losses provided in an embodiment of the present application;
[0050] Figure 6 A schematic diagram of a method for training a sound event classification model provided in one embodiment of the present application;
[0051] Figure 7 A block diagram of a training device for a sound event classification model provided in one embodiment of the present application. DETAILED DESCRIPTION
[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0053] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0054] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1 A processor 11 is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment. In one embodiment, the electronic device 1 can be a host, a server, a server cluster or a cloud computing center, which is used to execute the training method of the sound event classification model.
[0055] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.
[0056] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the training method of the sound event classification model provided in the present application.
[0057] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training method of the sound event classification model provided in the present application.
[0058] See also Figure 2 , is a flow chart of a method for training a sound event classification model provided in an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps S1-S3.
[0059] S1: Input the audio features of multiple audio data in the specified data set into the first original model and the second original model respectively to obtain the first prediction category output by the first original model and the second prediction category output by the second original model; wherein the specified data set includes original category data, new category data and unsupervised data; the first original model and the second original model have a feature extraction network with the same structure, the classifier of the second original model adds an output unit of the new category compared to the classifier of the first original model, and the second original model and the first original model have the same initialization parameters.
[0060] The original category data is audio data of the original category, and the original category is the sound event category that the first original model and the second original model can already classify.
[0061] The new category data is audio data of a new category, and the new category is a sound event category that cannot be distinguished by the first original model and the second original model.
[0062] The audio data in both the original category data and the new category data carry one or more category labels. The category label of any sound event category indicates whether the audio data belongs to the sound event category. Exemplarily, if the category label corresponding to the sound event category "wind sound" of any audio data is "yes", it means that the audio data belongs to "wind sound". Alternatively, if the category label corresponding to the sound event category "gun shooting" of any audio data is "yes", the category label corresponding to the sound event category "whistle" is "yes", and the category label corresponding to the sound category label "wind sound" is "no", it means that the audio data belongs to "gun shooting" and "whistle", but not "wind sound". The category label can be in the form of 0 or 1, 0 means "no" and 1 means "yes".
[0063] In unsupervised data, audio data does not carry any category labels.
[0064] The first original model and the second original model both include a feature extraction network and a classifier. The network structure of the feature extraction network of the first original model and the second original model is the same, and the initialization parameters of the feature extraction network are the same. The classifier can be composed of a fully connected layer, including output units corresponding to each sound event category, and each output unit performs a binary classification task of its corresponding sound event category during prediction. Exemplarily, the output unit of any sound event category can output a value between 0 and 1, which represents the probability that the audio data belongs to the sound event category.
[0065] The classifier of the first original model contains output units of all original categories; the classifier of the second original model contains output units of all original categories and all new categories. The output units of all original categories in the classifier of the first original model and the output units of all original categories in the classifier of the second original model have the same initialization parameters. The original categories contain unknown categories, so the classifiers of the first original model and the second original model are both equipped with output units corresponding to unknown categories.
[0066] The audio features of the audio data are feature data that can be processed by the deep neural network (the first original model and the second original model), and the audio features are extracted from the audio data. For details, please see the relevant description below.
[0067] The first output category is the output result of the classifier of the first original model; the second output category is the output result of the classifier of the second original model.
[0068] After the audio features of the audio data in the specified data set are input into the first original model, the first original model can process the audio features so as to output the first predicted category through each output unit of the classifier. In the training stage, for any audio data carrying a category label, only the first predicted category output by the output unit corresponding to the category label can be focused on, and the output results of other output units can be ignored or not output directly; for audio data that does not carry a category label (unsupervised data), only the first predicted category output by the output unit corresponding to the unknown category can be focused on, and the output results of other output units can be ignored or not output directly.
[0069] After the audio features of the audio data in the specified data set are input into the second original model, the second original model can process the audio features, so as to output the second predicted category through each output unit of the classifier. In the training stage, for any audio data carrying a category label, only the second predicted category output by the output unit corresponding to the category label can be paid attention to, and the output results of other output units can be ignored or not output directly; for audio data that does not carry a category label (unsupervised data), only the second predicted category output by the output unit corresponding to the unknown category can be paid attention to, and the output results of other output units can be ignored or not output directly.
[0070] S2: Determine a prediction loss according to the first predicted category, the second predicted category, the category label of the original category data, and the category label of the new category data, and adjust the parameters of the second original model based on the prediction loss.
[0071] After determining the first prediction category and the second prediction category for the audio features of each audio data, the prediction loss of the second original model during category prediction is calculated according to the first prediction category and the second prediction category, as well as the category label of the original category data and the category label of the new category data, and the parameters of the second original model are adjusted according to the prediction loss. Here, the adjusted parameters may include parameters of the feature extraction network and the classifier of the second original model.
[0072] S3: Return to step S1, and after multiple iterations, use the converged second original model as the sound event classification model.
[0073] After adjusting the parameters of the second original model, the process can return to step S1 to start a new round of training. In this process, since the parameters of the first original model do not change, theoretically, the first prediction category output by the first original model for each audio feature does not change. Therefore, at the beginning of the second round, only the audio features of the audio data in the specified data set can be input into the second original model to reduce the amount of calculation and save computing resources.
[0074] Through repeated iterations, when the training round reaches a preset round threshold, or when the prediction loss tends to be stable, it can be considered that the second original model has converged. At this time, the converged second original model can be used as a sound event classification model.
[0075] Through the above measures, the prediction loss is calculated by the first prediction result output by the first original model, which can constrain the prediction difference between the new model (second original model) and the old model (first original model) on the original category, and the introduction of unsupervised data can constrain the prediction difference between the new model (second original model) and the old model (first original model) on the unknown category data, maintain the consistency of the first original model and the second original model, alleviate the problem of the new model forgetting the original category knowledge, and maintain the classification performance of the new model for the original category. This process does not require the parameters of the feature extraction network to be fixed, so that the trained feature extraction network can be adjusted and optimized according to the new category data to ensure that its feature representation ability meets the classification requirements.
[0076] In one embodiment, see Figure 3 , is a flow chart of a method for extracting audio features provided in an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 310 to 330.
[0077] Step 310: Perform fast Fourier transform on the audio data in the specified data set to obtain a transformation result.
[0078] Step 320: Perform Mel filter bank processing on the transformation result to obtain a processing result.
[0079] Step 330: Take the logarithm of the processing result and perform discrete cosine transform on the logarithm to obtain the audio features of the audio data.
[0080] The audio data in the specified data set is time domain data, and can be processed into audio features that can be processed by a deep neural network through the preprocessing process of fast Fourier transform (FFT), Mel filter bank processing, logarithm and discrete cosine transform by executing the process from step 310 to step 330. Exemplarily, the audio features can be in the form of any one of vectors, feature maps, etc.
[0081] In one embodiment, see Figure 4 , is a flow chart of a method for generating an original model provided in an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps 410 to 440.
[0082] Step 410: Input the audio features of multiple audio data in the original data set into the deep learning model to obtain a third predicted category output by the deep learning model; wherein the original data set includes original category data; and the deep learning model includes a feature extraction network and a classifier.
[0083] The classifier of the deep learning model includes output units corresponding to all sound event categories of all audio data in the original dataset, as well as output units corresponding to unknown categories.
[0084] In the manner described above, audio features are extracted from each audio data in the original data set, and then the audio features are input into the deep learning model. The audio features are processed by the feature extraction network and classifier of the deep learning model to obtain the output results of each output unit of the classifier as the third prediction category.
[0085] During the training phase, for any audio data carrying a category label, we can only focus on the third predicted category output by the output unit corresponding to the category label, and the output results of other output units can be ignored or not output directly.
[0086] Step 420: Determine a prediction loss based on the third predicted category and the category label of the original category data, and adjust the parameters of the deep learning model based on the prediction loss.
[0087] After obtaining the third predicted category corresponding to each audio data, the difference between the third predicted category and the category label can be evaluated through a preset loss function to obtain the prediction loss, and then the parameters of the feature extraction network and classifier in the deep learning model can be adjusted with the prediction loss.
[0088] Step 430: Repeat the above process, and after multiple iterations, use the converged deep learning model as the first original model.
[0089] Repeat the training process from step 410 to step 420. After repeated iterations, when the training round reaches the round threshold, or when the prediction loss tends to be stable, it can be considered that the deep learning model has converged. At this time, the deep learning model is used as the first original model.
[0090] Step 440: Based on the first original model, generate a second original model including a new category output unit.
[0091] After obtaining the first original model, the structure and parameters of the first original model can be copied, and the output unit of the new category to be identified is added to the copied model to obtain the second original model. In this case, the second original model has the same feature extraction network as the first original model and has the same output unit corresponding to the original category.
[0092] In one embodiment, original category data of each category can be uniformly and randomly extracted from the original data set to form an original data subset. In this case, the number of audio data of each original category in the original data subset is similar.
[0093] In addition, preset new category data and preset unsupervised data can be obtained, and the original data subset, new category data and unsupervised data are mixed to obtain a specified data set.
[0094] Given the limited amount of strongly labeled data available for model training, introducing unlabeled unsupervised data into a specified dataset can maintain the prediction consistency of the first original model and the second original model on unknown categories. Specifically, the smaller the difference in predictions between the first original model and the second original model for audio data of unknown categories, the more the second original model, as a new model, retains knowledge of the original category. Therefore, the introduction of unsupervised data can improve the generalization performance of the second original model during training.
[0095] In addition, by retaining the original category data in a balanced sampling manner to construct a subset of the original data, it is possible to learn new category knowledge while retaining the original category knowledge of the second original model. At the same time, the balanced number of original category data helps the wall model to predict the deviation of the original category. The original data of each category can be learned fairly, thereby generating a reliable and fair sound event classification model.
[0096] In one embodiment, see Figure 5 , is a flow chart of a method for calculating predicted loss provided in an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps 510 to 540.
[0097] Step 510: Determine a first sub-loss according to the first prediction result and the second prediction result of the original category data.
[0098] The first sub-loss can be obtained by calculating the difference between the first prediction result and the second prediction result of the original category data.
[0099] In one embodiment, a relative entropy loss (Kullback-Leibler divergence) may be calculated based on the first prediction result and the second prediction result of the original category data as the first sub-loss.
[0100] Step 520: Determine a second sub-loss based on the first prediction result and the second prediction result of the unsupervised data.
[0101] The first prediction result of the unsupervised data is the output result of the first original model in the output unit corresponding to the unknown category; the second prediction result of the unsupervised data is the output result of the second original model in the output unit corresponding to the unknown category. By calculating the difference between the two, the second sub-loss can be obtained.
[0102] In one embodiment, a mean square error loss may be calculated based on the first prediction result and the second prediction result of the unsupervised data as the second sub-loss.
[0103] In this case, the second sub-loss will pay special attention to the unsupervised data with large prediction differences between the first original model and the second original model. That is to say, the second sub-loss will be small only when the prediction differences between the first original model and the second original model for each unsupervised data are relatively small. With this as supervision, the prediction differences between the first original model and the second original model for the unsupervised data can be narrowed, thereby achieving the retention of the original category knowledge by the second original model.
[0104] Step 530: Determine a third sub-loss according to the second prediction result of the original category data, the category label of the original category data, the second prediction result of the new category data, and the category label of the new category data.
[0105] The third sub-loss can be obtained by calculating the difference between the second prediction result and the category label of the original category data, and the difference between the second prediction result and the category label of the new category data.
[0106] In one embodiment, the first cross entropy loss may be calculated based on the second prediction result and the category label of the original category data. The second cross entropy loss may be calculated based on the second prediction result and the category label of the new category data.
[0107] After obtaining the first cross entropy loss and the second cross entropy loss, the third sub-loss may be determined. Exemplarily, the third sub-loss may be obtained by adding the first cross entropy loss to the second cross entropy loss. Step 540: Determine the prediction loss according to the first sub-loss, the second sub-loss, and the third sub-loss.
[0108] After obtaining the first sub-loss, the second sub-loss and the third sub-loss, the three sub-losses may be weighted and summed to obtain the prediction loss. Here, the weight of each sub-loss may be configured as required. Exemplarily, the weights of each sub-loss are the same.
[0109] Through the above measures, the prediction loss of the second original model can be effectively evaluated with the help of the first prediction result output by the first original model. Figure 6 , is a schematic diagram of a training method for a sound event classification model provided in an embodiment of the present application, such as Figure 6As shown, the audio data obtains audio features after preprocessing, and the audio feature distribution can be input into the first original model and the second original model. The output results of the first original model include the original category data results and the unsupervised data output results; the output results of the second original model include the original category data results, the unsupervised data output results and the new category output results. The original category output results in the first prediction result and the original category output results in the second prediction result can calculate the Kullback-Leibler divergence as the first sub-loss. The unsupervised data output results in the first prediction result and the unsupervised data output results in the second prediction result can calculate the mean square error loss as the second sub-loss. The original category output results and the new category data results in the second prediction result can be combined with the category label to calculate the cross entropy loss as the third sub-loss. The prediction loss of the second original model is then determined by the first sub-loss, the second sub-loss and the third sub-loss.
[0110] In one embodiment, after the sound event classification model is trained, the sound event classification model can be used to perform sound event classification on the audio data to be classified.
[0111] The audio data to be classified is the audio data that needs to be classified.
[0112] By preprocessing the audio data to be classified, the corresponding audio features can be obtained. The preprocessing method is described above and will not be repeated here.
[0113] After the audio features are input into the trained sound event classification model, the sound event category information output by the sound event classification model can be obtained.
[0114] Through this measure, the sound event classification model can be used to effectively classify the sound events in the audio data.
[0115] Figure 7 is a block diagram of a training device for a sound event classification model according to an embodiment of the present invention. Figure 7 As shown, the device may include:
[0116] The prediction module 710 is used to execute step S1, wherein S1: audio features of multiple audio data in a specified data set are input into a first original model and a second original model respectively to obtain a first prediction category output by the first original model and a second prediction category output by the second original model; wherein the specified data set includes original category data, new category data and unsupervised data; the first original model and the second original model have a feature extraction network with the same structure, the classifier of the second original model has an output unit of a new category compared to the classifier of the first original model, and the second original model and the first original model have the same initialization parameters;
[0117] The calculation module 720 is used to execute step S2, S2: determine the prediction loss according to the first prediction category, the second prediction category, the category label of the original category data and the category label of the new category data, and adjust the parameters of the second original model based on the prediction loss;
[0118] The iteration module 730 is used to execute step S3. S3: return to step S1, and after multiple iterations, use the converged second original model as the sound event classification model.
[0119] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the training method of the above-mentioned sound event classification model, and will not be repeated here.
[0120] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0121] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0122] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.
Claims
1. A method for training a sound event classification model, characterized in that: include: S1: Input the audio features of multiple audio data in the specified data set into the first original model and the second original model respectively, and obtain the first predicted category output by the first original model and the second predicted category output by the second original model; wherein the specified data set includes original category data, new category data and unsupervised data; the first original model and the second original model have the same feature extraction network structure, the classifier of the second original model adds an output unit of the new category compared with the classifier of the first original model, and the second original model and the first original model have the same initialization parameters; S2: determining a prediction loss according to the first prediction category, the second prediction category, the category label of the original category data, and the category label of the new category data, and adjusting the parameters of the second original model based on the prediction loss; S3: Return to step S1, and after multiple iterations, use the converged second original model as the sound event classification model.
2. The method according to claim 1, characterized in that Before step S1, the method further includes: Performing a fast Fourier transform on the audio data in the specified data set to obtain a transformation result; Performing Mel filter bank processing on the transformation result to obtain a processing result; The logarithm of the processing result is taken, and discrete cosine transform is performed on the logarithm to obtain the audio features of the audio data.
3. The method according to claim 1, characterized in that Before step S1, the method further includes: Inputting audio features of multiple audio data in the original data set into a deep learning model to obtain a third predicted category output by the deep learning model; wherein the original data set includes original category data; and the deep learning model includes a feature extraction network and a classifier; Determining a prediction loss according to the third predicted category and the category label of the original category data, and adjusting parameters of the deep learning model based on the prediction loss; Repeat the above process for multiple iterations, and use the converged deep learning model as the first original model; Based on the first original model, a second original model including a new category output unit is generated.
4. The method according to claim 3, characterized in that The specified data set is constructed in the following way: Uniformly and randomly extracting original category data of each category from the original data set to form an original data subset; The original data subset, the preset new category data and the preset unsupervised data constitute the specified data set.
5. The method according to claim 1, characterized in that The determining of the prediction loss according to the first predicted category, the second predicted category, the category label of the original category data, and the category label of the new category data includes: Determine a first sub-loss according to the first prediction result and the second prediction result of the original category data; Determine a second sub-loss according to the first prediction result and the second prediction result of the unsupervised data; Determine a third sub-loss according to the second prediction result of the original category data, the category label of the original category data, the second prediction result of the new category data, and the category label of the new category data; The predicted loss is determined according to the first sub-loss, the second sub-loss and the third sub-loss.
6. The method according to claim 5, characterized in that The determining of the first sub-loss according to the first prediction result and the second prediction result of the original category data comprises: Calculating a relative entropy loss as a first sub-loss according to the first prediction result and the second prediction result of the original category data; The determining the second sub-loss according to the first prediction result and the second prediction result of the unsupervised data comprises: According to the first prediction result and the second prediction result of the unsupervised data, the mean square error loss is calculated as the second sub-loss; The determining the third sub-loss according to the second prediction result of the original category data, the category label of the original category data, the second prediction result of the new category data, and the category label of the new category data includes: Calculating a first cross entropy loss according to the second prediction result and the category label of the original category data; Calculating a second cross entropy loss according to the second prediction result and the category label of the new category data; A third sub-loss is determined according to the first cross entropy loss and the second cross entropy loss.
7. The method according to claim 1, characterized in that The method further comprises: The audio data to be classified is preprocessed to obtain corresponding audio features, and the audio features are input into the sound event classification model to obtain the sound event category information output by the sound event classification model.
8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the training method of the sound event classification model described in any one of claims 1-7.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the training method of the sound event classification model described in any one of claims 1-7.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the training method of the sound event classification model described in any one of claims 1 to 7 is implemented.