Classification model generation method, audio classification method, device, medium and equipment

Abstract

The present disclosure relates to a classification model generation method, an audio classification method, an apparatus, a medium and an equipment. The audio classification model comprises an audio representation extraction module and a classifier. The method comprises: obtaining a new-class audio sample and a reference class of the new-class audio sample; extracting a first audio representation of the new-class audio sample by using each pre-training model in at least one pre-training model; performing model training by taking the new-class audio sample as the input of the audio representation extraction module, taking a spliced audio representation obtained based on each first audio representation and a second audio representation output by the audio representation extraction module as the input of the classifier, and taking the reference class as the target output of the classifier, so as to obtain the audio classification model. Thus, the classification effect of the rare audio classification task can be improved. The model structure of the classification model does not need to be consistent with the model structure of the pre-training model, and the classification model structure most suitable for the rare audio can be adopted, so as to further improve the classification effect.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, specifically to a classification model generation method, an audio classification method, an apparatus, a medium, and a device. Background Technology

[0002] Audio classification refers to the analysis and understanding of an audio segment to arrive at a predefined audio category. Common categories include speech, singing, and instrumental music, while less common categories include wildlife calls and thunder. Developing an audio classification model typically requires massive amounts of training data. However, collecting data on less common event categories is extremely difficult. Therefore, training sets for uncommon sound events are usually small, and audio classification models trained directly using such data generally have poor classification performance. Summary of the Invention

[0003] This section is provided to briefly introduce the concepts, which will be described in detail in the Detailed Description section later. This section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, this disclosure provides a method for generating an audio classification model, the audio classification model including an audio representation extraction module and a classifier, the method comprising:

[0005] Obtain the new type of audio sample and the reference category of the new type of audio sample;

[0006] The first audio representation of the new class of audio samples is extracted using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples;

[0007] The audio classification model is obtained by using the new type of audio samples as input to the audio representation extraction module, using the spliced ​​audio representation obtained based on each first audio representation and the second audio representation output by the audio representation extraction module as input to the classifier, and using the reference category as the target output of the classifier.

[0008] Secondly, this disclosure provides an audio classification method, including:

[0009] Obtain the audio data to be categorized;

[0010] The audio data to be classified is input into a pre-trained audio classification model to obtain the target category of the audio data to be classified, wherein the audio classification model is generated according to the audio classification model generation method provided in the first aspect of this disclosure.

[0011] Thirdly, this disclosure provides an audio classification model generation apparatus, wherein the audio classification model includes an audio representation extraction module and a classifier, and the apparatus includes:

[0012] The first acquisition module is used to acquire new type audio samples and reference categories of the new type audio samples;

[0013] An extraction module is used to extract a first audio representation of the new class audio sample using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples.

[0014] The model generation module is used to train the model by taking the new class of audio samples as input to the audio representation extraction module, taking the spliced ​​audio representation obtained based on each first audio representation and the second audio representation output by the audio representation extraction module as input to the classifier, and taking the reference class as the target output of the classifier, so as to obtain the audio classification model.

[0015] Fourthly, this disclosure provides an audio classification device, comprising:

[0016] The second acquisition module is used to acquire the audio data to be classified.

[0017] An audio classification module is used to input the audio data to be classified into a pre-trained audio classification model to obtain the target category of the audio data to be classified, wherein the audio classification model is generated according to the audio classification model generation method provided in the first aspect of this disclosure.

[0018] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the audio classification model generation method provided in the first aspect of this disclosure or the steps of the audio classification method provided in the second aspect of this disclosure.

[0019] Sixthly, this disclosure provides an electronic device, comprising:

[0020] A storage device having at least one computer program stored thereon;

[0021] At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the audio classification model generation method provided in the first aspect of this disclosure or the steps of the audio classification method provided in the second aspect of this disclosure.

[0022] In the above technical solution, the audio representation extracted from the pre-trained model trained on base class audio samples is used as prior information and concatenated into the training process of the audio classification model for classifying new audio classes. This improves the classification performance of uncommon audio classification tasks. Furthermore, since the audio classification model is not fine-tuned from the pre-trained model, its structure does not need to be consistent with the pre-trained model. Therefore, the audio classification model can adopt the most suitable classification model structure for uncommon audio, further improving the classification performance of uncommon audio classification tasks.

[0023] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0024] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0025] Figure 1 This is a flowchart illustrating an audio classification model generation method according to an exemplary embodiment.

[0026] Figure 2A This is a schematic diagram illustrating an audio classification model generation process according to an exemplary embodiment.

[0027] Figure 2B This is a schematic diagram illustrating an audio classification model generation process according to another exemplary embodiment.

[0028] Figure 3 This is a flowchart illustrating a method for determining spliced ​​audio representations according to an exemplary embodiment.

[0029] Figure 4 This is a flowchart illustrating an audio classification method according to an exemplary embodiment.

[0030] Figure 5 This is a block diagram illustrating an audio classification model generation apparatus according to an exemplary embodiment.

[0031] Figure 6 This is a block diagram illustrating an audio classification device according to an exemplary embodiment.

[0032] Figure 7 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0033] As discussed in the background section, training sets for uncommon sound events are typically small, leading to poor classification performance for audio classification models trained directly on such data. Current methods often employ transfer learning to address this issue. The mainstream approach involves pre-training the model using large datasets of common audio categories (e.g., speech, singing, instrumental music) and then fine-tuning the pre-trained model using training data from uncommon sound events. However, this results in a final audio classification model with a structure consistent with the pre-trained model. This structure may not be optimal for classifying uncommon audio. Furthermore, the pre-trained model is usually a large, general model trained on a large dataset, which can easily lead to overfitting when transferred to small datasets of uncommon audio. Consequently, it's difficult to guarantee good classification performance for uncommon audio.

[0034] In view of this, the present disclosure provides a classification model generation method, an audio classification method, an apparatus, a medium, and a device.

[0035] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0036] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0037] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0038] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0039] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0040] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0041] All actions involving the acquisition of signals, information, or data in this disclosure are carried out in accordance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0042] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0043] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0044] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0045] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0046] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0047] Figure 1This is a flowchart illustrating an audio classification model generation method according to an exemplary embodiment. For example... Figure 1 As shown, the audio classification model generation method may include the following S101 to S103.

[0048] In S101, obtain the new class audio sample and the reference class of the new class audio sample.

[0049] In this disclosure, the novel audio sample can be audio data of uncommon audio categories (such as wildlife calls, thunder, etc.). For example, the novel audio sample can be acquired by recording. The reference category for the novel audio sample can be, for example, wildlife calls, thunder, etc. All actions involving the acquisition of signals, information, or data in this disclosure are performed in accordance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0050] like Figure 2A and Figure 2B As shown, the audio classification model described above may include an audio representation extraction module and a classifier. The audio representation extraction module extracts audio representations from the input audio (e.g., new class audio samples, audio data to be classified); the classifier, connected to the audio representation extraction module, predicts the category of the input audio based on the audio representations input into the classifier, thus obtaining the predicted category of the input audio.

[0051] Among them, such as Figure 2A and Figure 2B As shown, during the training phase of the audio classification model, the audio representation input to the classifier is a concatenated audio representation (hereinafter referred to as the concatenated audio representation) obtained based on each first audio representation and the second audio representation output by the audio representation extraction module. During the application phase of the audio classification model, the audio representation input to the classifier is the audio representation extracted from the audio data to be classified by the audio representation extraction module, i.e., the output of the audio representation extraction module.

[0052] In S102, each of the at least one pre-trained models extracts the first audio representation of the new class of audio samples.

[0053] In this disclosure, the pre-trained model is a model trained on base class audio samples for audio classification. The base class audio samples are audio data of common (i.e., speech, singing, instrumental music, etc.) audio categories, which can be obtained from a large-scale dataset of common audio categories. For example, a large-scale dataset of common audio categories can be an open-source dataset, such as the open-source AudioSet dataset with thousands of hours of data, containing more than 500 audio categories. These large-scale datasets are characterized by their inclusion of common audio categories such as speech, singing, and instrumental music. The pre-trained model can also include an audio representation extraction module and a classifier. The structure of the audio representation extraction module in the pre-trained model can be the same as or different from that in the audio classification model described above. Similarly, the structure of the classifier in the pre-trained model can be the same as or different from that in the audio classification model described above; this disclosure does not impose any limitations. Specifically, the audio representation extraction module of the pre-trained model can be used to extract the first audio representation of new class audio samples.

[0054] In one implementation, such as Figure 2A As shown, a first audio representation of a new class of audio samples can be extracted using a pre-trained model.

[0055] In another implementation, each of the multiple pre-trained models can be used to extract a first audio representation of the new audio class sample, resulting in multiple first audio representations, where the structures of the multiple pre-trained models are different from each other. In this way, the audio representations extracted by each of the multiple structurally different pre-trained models can be used as prior information and concatenated into the training process of the audio classification model used for classifying the new audio class, thereby ensuring classification performance for uncommon audio classification tasks.

[0056] For example, such as Figure 2B As shown, the first audio representation of the new class of audio samples is extracted using pre-trained model A and pre-trained model B, respectively, to obtain the first audio representation A1 and the first audio representation B1.

[0057] You can either train a pre-trained model based on a dataset of general audio categories, or directly download an open-source general audio classification model as a pre-trained model. These open-source general audio classification models are pre-trained based on datasets of general audio categories.

[0058] In S103, an audio classification model is obtained by using new class audio samples as input to the audio representation extraction module, using the spliced ​​audio representation obtained based on each first audio representation and the second audio representation output by the audio representation extraction module as input to the classifier, and using the reference class as the target output of the classifier.

[0059] In this disclosure, during the training of the audio classification model, the cross-entropy loss function and the Adaptive Moment Estimation (Adam) optimizer can be used to update the model parameters. For example, the model is trained for 50 epochs using the cross-entropy loss function and the Adam optimizer (one epoch means that each new class of audio sample in the training set is trained once). The cross-entropy loss function calculates the error by measuring the difference between the reference class and the predicted class output by the classifier; the Adam optimizer is a gradient-based optimization algorithm with fast convergence speed, high computational efficiency, and low memory consumption.

[0060] In the above technical solution, the audio representation extracted from the pre-trained model trained on base class audio samples is used as prior information and concatenated into the training process of the audio classification model for classifying new audio classes. This improves the classification performance of uncommon audio classification tasks. Furthermore, since the audio classification model is not fine-tuned from the pre-trained model, its structure does not need to be consistent with the pre-trained model. Therefore, the audio classification model can adopt the most suitable classification model structure for uncommon audio, further improving the classification performance of uncommon audio classification tasks.

[0061] like Figure 2A and Figure 2B As shown, the audio representation extraction module in the audio classification model can include a cascaded feature extractor and encoder. The feature extractor is used to extract the audio features of the input audio, and the encoder is used to encode the audio features to obtain the audio representation of the input audio.

[0062] Among them, audio features can be, for example, log-mel filter bank features, Mel-scale frequency cepstral coefficients (MFCC) features, constant Q transform (CQT) features, fundamental frequency features, etc.

[0063] For example, the encoder of an audio classification model can be a neural network model, which may include cascaded convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The CNNs may consist of four sequentially connected CNN blocks, and the RNN is connected to the last of the four sequentially connected CNN blocks. Each CNN block includes sequentially connected convolutional layers, batch normalization layers, and pooling layers. For example, the kernel size of the convolutional layers is 3x3, the number of channels in the four sequentially connected CNN blocks are 64, 128, 256, and 512 respectively, and the pooling size of the pooling layers is 2x2. The RNN may be, for example, a unidirectional long short-term memory (LSTM) neural network or a bidirectional long short-term memory (Bi-LSTM) neural network, with 256 neurons.

[0064] The encoder of the audio classification model (specifically, the recurrent neural network mentioned above) outputs a matrix of size [bs, T, emb1], which is the second audio representation. Here, bs is the batch size; T is the time dimension (duration) of the second audio representation, which is positively correlated with the duration of the new class audio sample; and emb1 is the dimension of the audio representation vector of the new class audio sample at each time step. For example, emb1 is 256.

[0065] The classifier of the audio classification model can include two fully connected layers. The first fully connected layer has 256 neurons, and the second layer has the number of neurons corresponding to the number of categories of uncommon audio.

[0066] The following section details the method for determining the spliced ​​audio representation. Specifically, it can be achieved through... Figure 3 The S301 and S302 shown are used to obtain the spliced ​​audio representation.

[0067] In S301, for each first audio representation, a representation transformation process is performed on the first audio representation so that the time dimension of the first audio representation obtained after the representation transformation process is consistent with the time dimension of the second audio representation.

[0068] In this disclosure, the audio representation extraction of the pre-trained model is consistent with the configuration of the pre-trained model itself, including features, parameters, etc. The first audio representation extracted by the pre-trained model is a matrix of size [bs, t, emb2]. Here, t is the time dimension of the first audio representation, which is positively correlated with the duration of the new class of audio samples; emb2 is the dimension of the audio representation vector of the new class of audio samples at each time step. Since the feature types and parameters extracted by the pre-trained model may differ from those of the aforementioned audio classification model, t and T may not be consistent, and emb2 is also related to the configuration of the pre-trained model, and its dimension may also be inconsistent with emb1. In order to concatenate the first audio representations extracted by each pre-trained model with the second audio representation, it is necessary to perform representation transformation on the first audio representations extracted by each pre-trained model separately, so that the time dimension of the first audio representation obtained after the representation transformation is consistent with the time dimension of the second audio representation, that is, converting the first audio representations extracted by each pre-trained model into a matrix of size [bs, t, emb2].

[0069] In S302, the first audio representation and the second audio representation obtained after all representation transformation processes are spliced ​​together to obtain the spliced ​​audio representation.

[0070] In one implementation, the above-mentioned at least one pre-trained model is one. In this case, the first audio representation extracted by the one pre-trained model can be subjected to representation transformation processing. Then, the first audio representation obtained after the representation transformation processing is concatenated with the second audio representation to obtain the concatenated audio representation.

[0071] In another implementation, there are multiple pre-trained models. In this case, the first audio representations extracted by each first pre-trained model can be processed by representation transformation. Then, the first audio representations and second audio representations obtained after all representation transformation processes are spliced ​​together to obtain spliced ​​audio representations.

[0072] For example, such as Figure 2B As shown, pre-trained model A and pre-trained model B are used to extract the first audio representation of the new type of audio representation, respectively, to obtain the first audio representation A1 and the first audio representation B1. At this time, the first audio representation A1 and the first audio representation B1 can be transformed to obtain the first audio representation A1 and the first audio representation B1 after transformation. Then, the first audio representation A1, the first audio representation B1 after transformation, and the second audio representation can be concatenated in sequence to obtain the concatenated audio representation.

[0073] The following is a detailed description of the specific implementation method for the representation conversion processing of the first audio representation in S301 above. Specifically, it can be achieved through the following steps (1) to (3):

[0074] Step (1): Perform feature averaging on the first audio representation in the time dimension.

[0075] Specifically, the first audio representation of size [bs,t,emb2] can be processed by averaging the features along the t dimension (i.e., calculating the average feature value along the t dimension) to obtain the audio representation of size [bs,emb2], which is the first audio representation obtained after feature averaging.

[0076] Step (2): Perform standard normalization on the first audio representation obtained after feature averaging.

[0077] Standard normalization of the first audio representation obtained after feature averaging means normalizing each element in the first audio representation (vector) obtained after feature averaging to a value with a mean of 0 and a variance of 1.

[0078] Step (3): Based on the time dimension of the second audio representation, extend the first audio representation obtained after standard normalization.

[0079] Specifically, the first audio representation obtained after standard normalization can be copied T times in the time dimension, that is, the first audio representation of size [bs,emb2] obtained after standard normalization can be copied T times in the time dimension to obtain a matrix of size [bs,T,emb2], which is the first audio representation obtained after representation transformation.

[0080] In addition, this disclosure also provides an audio classification method, such as Figure 4 As shown, the audio classification can include S401 and S402.

[0081] In S401, the audio data to be classified is obtained.

[0082] In S402, the audio data to be classified is input into a pre-trained audio classification model to obtain the target category of the audio data to be classified.

[0083] The audio classification model is generated according to the audio classification model generation method provided in this disclosure.

[0084] Figure 5 This is a block diagram illustrating an audio classification model generation apparatus according to an exemplary embodiment, wherein the audio classification model includes an audio representation extraction module and a classifier. Figure 5 As shown, the device 500 includes:

[0085] The first acquisition module 501 is used to acquire a new type of audio sample and a reference category of the new type of audio sample;

[0086] Extraction module 502 is used to extract the first audio representation of the new class audio sample using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples;

[0087] The model generation module 503 is used to train the model by taking the new class of audio samples as input to the audio representation extraction module, taking the spliced ​​audio representation obtained based on each first audio representation and the second audio representation output by the audio representation extraction module as input to the classifier, and taking the reference class as the target output of the classifier, so as to obtain the audio classification model.

[0088] In the above technical solution, the audio representation extracted from the pre-trained model trained on base class audio samples is used as prior information and concatenated into the training process of the audio classification model for classifying new audio classes. This improves the classification performance of uncommon audio classification tasks. Furthermore, since the audio classification model is not fine-tuned from the pre-trained model, its structure does not need to be consistent with the pre-trained model. Therefore, the audio classification model can adopt the most suitable classification model structure for uncommon audio, further improving the classification performance of uncommon audio classification tasks.

[0089] Optionally, the model generation module 503 includes:

[0090] The representation conversion processing submodule is used to perform representation conversion processing on each of the first audio representations so that the time dimension of the first audio representation obtained after the representation conversion processing is consistent with the time dimension of the second audio representation.

[0091] The splicing submodule splices the first audio representation and the second audio representation obtained after all the representation conversion processes to obtain the spliced ​​audio representation.

[0092] Optionally, the representation transformation processing submodule includes:

[0093] The feature averaging submodule is used to perform feature averaging on the first audio representation in the time dimension;

[0094] The standard normalization processing submodule is used to perform standard normalization processing on the first audio representation obtained after feature averaging.

[0095] An extension submodule is used to extend the first audio representation obtained after standard normalization processing according to the time dimension of the second audio representation.

[0096] Optionally, the extended submodule is used to copy the first audio representation obtained after standard normalization processing T times in the time dimension, where T is the time dimension of the second audio representation.

[0097] Optionally, there may be multiple pre-trained models.

[0098] Figure 6 This is a block diagram illustrating an audio classification device according to an exemplary embodiment. Figure 6 As shown, the device 600 includes:

[0099] The second acquisition module 601 is used to acquire audio data to be classified.

[0100] The audio classification module 602 is used to input the audio data to be classified into a pre-trained audio classification model to obtain the target category of the audio data to be classified, wherein the audio classification model is generated according to the audio classification model generation method provided in this disclosure.

[0101] This disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the audio classification model generation method or the steps of the audio classification method provided in this disclosure.

[0102] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 700 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0103] like Figure 7As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0104] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0105] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0106] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0107] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0108] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0109] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a new class of audio samples and a reference class of the new class of audio samples, wherein the audio classification model includes an audio representation extraction module and a classifier; extract a first audio representation of the new class of audio samples using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples; and train the model by using the new class of audio samples as input to the audio representation extraction module, using a concatenated audio representation obtained based on each first audio representation and a second audio representation output by the audio representation extraction module as input to the classifier, and using the reference class as the target output of the classifier, to obtain the audio classification model.

[0110] Alternatively, the aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire audio data to be classified; input the audio data to be classified into a pre-trained audio classification model to obtain the target category of the audio data to be classified, wherein the audio classification model is generated according to the audio classification model generation method provided in this disclosure.

[0111] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0113] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules do not necessarily limit the module itself; for example, the first acquisition module can also be described as "a module for acquiring new type audio samples and a reference category of the new type audio samples".

[0114] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] According to one or more embodiments of this disclosure, Example 1 provides an audio classification model generation method, the audio classification model including an audio representation extraction module and a classifier, the method comprising: obtaining a new class of audio samples and a reference class of the new class of audio samples; extracting a first audio representation of the new class of audio samples using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples; training the model by using the new class of audio samples as input to the audio representation extraction module, using a concatenated audio representation obtained based on each first audio representation and a second audio representation output by the audio representation extraction module as input to the classifier, and using the reference class as the target output of the classifier, to obtain the audio classification model.

[0117] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein the spliced ​​audio representation is obtained by: performing a representation transformation process on each first audio representation so that the time dimension of the first audio representation obtained after the representation transformation process is consistent with the time dimension of the second audio representation; splicing together all the first audio representations and second audio representations obtained after the representation transformation processes to obtain the spliced ​​audio representation.

[0118] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, wherein the representation transformation processing of the first audio representation includes: performing feature averaging processing on the first audio representation in the time dimension; performing standard normalization processing on the first audio representation obtained after feature averaging processing; and expanding the first audio representation obtained after standard normalization processing according to the time dimension of the second audio representation.

[0119] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, wherein extending the first audio representation obtained after standard normalization processing according to the time dimension of the second audio representation includes: copying the first audio representation obtained after standard normalization processing T times in the time dimension, where T is the time dimension of the second audio representation.

[0120] According to one or more embodiments of this disclosure, Example 5 provides a method of any one of Examples 1-4, wherein the pre-trained models are multiple.

[0121] According to one or more embodiments of this disclosure, Example 6 provides an audio classification method, including: acquiring audio data to be classified; inputting the audio data to be classified into a pre-trained audio classification model to obtain a target category of the audio data to be classified, wherein the audio classification model is generated according to the audio classification model generation method of any one of Examples 1-5.

[0122] According to one or more embodiments of this disclosure, Example 7 provides an audio classification model generation apparatus. The audio classification model includes an audio representation extraction module and a classifier. The apparatus includes: a first acquisition module for acquiring new class audio samples and a reference class of the new class audio samples; an extraction module for extracting a first audio representation of the new class audio samples using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples; and a model generation module for training the model by using the new class audio samples as input to the audio representation extraction module, using a concatenated audio representation obtained based on each first audio representation and a second audio representation output by the audio representation extraction module as input to the classifier, and using the reference class as the target output of the classifier, to obtain the audio classification model.

[0123] According to one or more embodiments of this disclosure, Example 8 provides the apparatus of Example 7, wherein the model generation module includes: a representation conversion processing submodule, configured to perform representation conversion processing on each of the first audio representations, such that the time dimension of the first audio representation obtained after the representation conversion processing is consistent with the time dimension of the second audio representation; and a splicing submodule, configured to splice all the first audio representations and the second audio representations obtained after the representation conversion processing to obtain the spliced ​​audio representation.

[0124] According to one or more embodiments of this disclosure, Example 9 provides the apparatus of Example 8, wherein the representation conversion processing submodule includes: a feature averaging processing submodule for performing feature averaging processing on the first audio representation in the time dimension; a standard normalization processing submodule for performing standard normalization processing on the first audio representation obtained after feature averaging processing; and an expansion submodule for expanding the first audio representation obtained after standard normalization processing according to the time dimension of the second audio representation.

[0125] According to one or more embodiments of this disclosure, Example 10 provides the apparatus of Example 9, wherein the extended submodule is used to copy a first audio representation obtained after standard normalization processing T times in the time dimension, where T is the time dimension of the second audio representation.

[0126] According to one or more embodiments of this disclosure, Example 11 provides an apparatus of any one of Examples 7-10, wherein the pre-trained models are multiple.

[0127] According to one or more embodiments of this disclosure, Example 12 provides an audio classification apparatus, comprising: a second acquisition module for acquiring audio data to be classified; and an audio classification module for inputting the audio data to be classified into a pre-trained audio classification model to obtain a target category of the audio data to be classified, wherein the audio classification model is generated according to the audio classification model generation method of any one of Examples 1-5.

[0128] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-6.

[0129] According to one or more embodiments of this disclosure, Example 14 provides an electronic device including: a storage device having at least one computer program stored thereon; and at least one processing device for executing the at least one computer program in the storage device to implement the steps of the method described in any one of Examples 1-6.

[0130] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0131] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0132] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for generating an audio classification model, wherein the audio classification model includes an audio representation extraction module and a classifier, characterized in that, The method includes: Obtain the new type of audio sample and the reference category of the new type of audio sample; The first audio representation of the new class of audio samples is extracted using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples; The audio classification model is obtained by using the new type of audio samples as input to the audio representation extraction module, using the spliced ​​audio representation obtained based on each first audio representation and the second audio representation output by the audio representation extraction module as input to the classifier, and using the reference category as the target output of the classifier. The spliced ​​audio representation is obtained in the following way: For each of the first audio representations, a representation transformation process is performed on the first audio representation so that the time dimension of the first audio representation obtained after the representation transformation process is consistent with the time dimension of the second audio representation. The first audio representation and the second audio representation obtained after all the representation transformation processes are spliced ​​together to obtain the spliced ​​audio representation. The representation transformation process for the first audio representation includes: The first audio representation is subjected to feature averaging in the time dimension; The first audio representation obtained after feature averaging is then subjected to standard normalization. Based on the time dimension of the second audio representation, the first audio representation obtained after standard normalization is extended.

2. The method according to claim 1, characterized in that, The step of expanding the first audio representation obtained after standard normalization based on the time dimension of the second audio representation includes: The first audio representation obtained after standard normalization is copied T times in the time dimension, where T is the time dimension of the second audio representation.

3. The method according to claim 1 or 2, characterized in that, There are multiple pre-trained models.

4. An audio classification method, characterized in that, include: Obtain the audio data to be categorized; The audio data to be classified is input into a pre-trained audio classification model to obtain the target category of the audio data to be classified, wherein the audio classification model is generated by the audio classification model generation method according to any one of claims 1-3.

5. An audio classification model generation apparatus, wherein the audio classification model includes an audio representation extraction module and a classifier, characterized in that, The device includes: The first acquisition module is used to acquire new type audio samples and reference categories of the new type audio samples; An extraction module is used to extract a first audio representation of the new class audio sample using each of at least one pre-trained model, wherein the pre-trained model is a model for audio classification trained based on base class audio samples. The model generation module is used to train the model by taking the new class of audio samples as input to the audio representation extraction module, taking the spliced ​​audio representation obtained based on each first audio representation and the second audio representation output by the audio representation extraction module as input to the classifier, and taking the reference class as the target output of the classifier, so as to obtain the audio classification model. The spliced ​​audio representation is obtained in the following way: For each of the first audio representations, a representation transformation process is performed on the first audio representation so that the time dimension of the first audio representation obtained after the representation transformation process is consistent with the time dimension of the second audio representation. The first audio representation and the second audio representation obtained after all the representation transformation processes are spliced ​​together to obtain the spliced ​​audio representation. The representation transformation process for the first audio representation includes: The first audio representation is subjected to feature averaging in the time dimension; The first audio representation obtained after feature averaging is then subjected to standard normalization. Based on the time dimension of the second audio representation, the first audio representation obtained after standard normalization is extended.

6. An audio classification device, characterized in that, include: The second acquisition module is used to acquire the audio data to be classified. An audio classification module is used to input the audio data to be classified into a pre-trained audio classification model to obtain the target category of the audio data to be classified, wherein the audio classification model is generated by the audio classification model generation method according to any one of claims 1-3.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method described in any one of claims 1-4.

8. An electronic device, characterized in that, include: A storage device having at least one computer program stored thereon; At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1-4.

Images