Zero-shot audio classification model training method, zero-shot audio classification method

Abstract

The application discloses a zero-shot audio classification model training method, comprising the following steps: obtaining a sound attribute description corresponding to a sound category to which a sample audio belongs; determining an attribute description text corresponding to the sample audio according to the sound attribute description; and performing zero-shot audio classification model training according to the similarity between the sample audio and the attribute description text. According to the application, the attribute description text corresponding to the sample audio is determined according to the sound attribute description, so that the description text in the application focuses on diversified sound attributes, and the sound attributes can better represent the intrinsic characteristics of the sound, thereby improving the recognition accuracy of the zero-shot audio classification model obtained by training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a zero-shot audio classification model training method, a zero-shot audio classification method, an electronic device, and a storage medium. Background Technology

[0002] Supervised learning holds great promise in many fields. However, it is limited to predefined categories included during training. In audio classification, current datasets only contain everyday sounds and rarely include domain-specific, rare sounds. Therefore, supervised audio classification requires time-consuming and laborious annotation of unseen sounds, hindering its real-world application.

[0003] To overcome the reliance on meticulously labeled training data, zero-shot audio classification has gained significant attention in recent years. Zero-shot audio classification aims to identify and classify sound categories never encountered during model training. Most research focuses on learning the correlation between audio features and category semantic features. Related techniques use bilinear models to compute audio-category similarity between a given audio file and its category embedding. Text labels are categories and definitions from Wikipedia; for example, the text label for the crow category is "sounds associated with medium to large birds like the Corvus". Synonyms, semantic extensions, and onomatopoeia are added as supplementary information in related techniques. However, labels and supplementary information are often insufficient to distinguish sounds, especially similar ones. Because they define the object of the sound rather than auditory attributes, they fail to reflect differences between approximate categories (such as machine guns and pistols). Summary of the Invention

[0004] This application provides a zero-shot audio classification model training method, a zero-shot audio classification method, an electronic device, and a storage medium to at least solve one of the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a zero-shot audio classification model training method, including:

[0006] Obtain the sound attribute description corresponding to the sound category of the sample audio;

[0007] Determine the attribute description text corresponding to the sample audio based on the sound attribute description;

[0008] A zero-shot audio classification model is trained based on the similarity between the sample audio and the attribute description text.

[0009] In some embodiments, obtaining the sound attribute description corresponding to the sound category to which the sample audio belongs includes:

[0010] Obtain the sound attributes corresponding to the sound category of the sample audio;

[0011] A pre-defined language model is used to generate descriptive information corresponding to the sound attributes as the sound attribute description.

[0012] In some embodiments, the sound attribute description includes descriptive information for multiple attributes.

[0013] Determining the attribute description text corresponding to the sample audio based on the sound attribute description includes:

[0014] Select descriptions of some attributes from the descriptions of multiple attributes;

[0015] The attribute description text corresponding to the sample audio is determined based on the description information of the aforementioned partial attributes.

[0016] In some embodiments, training a zero-shot audio classification model based on the similarity between the sample audio and the attribute description text includes:

[0017] The sample audio is subjected to audio encoding processing to obtain a sample audio embedding vector;

[0018] The attribute description text is processed by text encoding to obtain the sample text embedding vector;

[0019] A zero-shot audio classification model is trained based on the similarity between the sample audio embedding vector and the sample text embedding vector.

[0020] In some embodiments, training a zero-shot audio classification model based on the similarity between the sample audio embedding vector and the sample text embedding vector includes:

[0021] Obtain the audio fully connected projection layer embedding corresponding to the sample audio embedding vector;

[0022] Obtain the text fully connected projection layer embedding corresponding to the sample text embedding vector;

[0023] Zero-shot audio classification model training is performed based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding.

[0024] In some embodiments, the zero-shot audio classification model includes a text encoder and an audio encoder.

[0025] Training a zero-shot audio classification model based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding includes:

[0026] The audio encoder is trained based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding.

[0027] In some embodiments, sound attributes include at least one of sound category, sound frequency, timbre, onomatopoeia, metaphor, time, emotion, intensity, and texture.

[0028] Secondly, this application also provides a zero-shot audio classification method, comprising: classifying audio using a zero-shot audio classification model trained by the method described in any embodiment of this application.

[0029] Thirdly, this application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any embodiment of this application.

[0030] Fourthly, this application also provides a storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements the steps of the method described in any embodiment of this application.

[0031] In this embodiment, the attribute description text corresponding to the sample audio is determined based on the sound attribute description, so that the description text in this application focuses on diverse sound attributes, and the sound attributes can better represent the intrinsic features of the sound, thereby improving the recognition accuracy of the zero-shot audio classification model trained. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating an embodiment of the zero-shot audio classification model training method of this application;

[0034] Figure 2 A flowchart of another embodiment of the zero-shot audio classification model training method of this application;

[0035] Figure 3 A flowchart of another embodiment of the zero-shot audio classification model training method of this application;

[0036] Figure 4 A flowchart of another embodiment of the zero-shot audio classification model training method of this application;

[0037] Figure 5 A flowchart of another embodiment of the zero-shot audio classification model training method of this application;

[0038] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0039] Figure 7 This is an architecture diagram of an embodiment of the zero-shot audio classification model training method of this application;

[0040] Figure 8 This is a comparative diagram showing the performance improvement after adding sound attributes to this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0042] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0043] like Figure 1 As shown, embodiments of this application provide a zero-shot audio classification model training method, including:

[0044] S10. Obtain the sound attribute description corresponding to the sound category to which the sample audio belongs.

[0045] For example, this application uses two datasets, VGGSound and AudioSet, where sample audio samples are managed according to sound categories. The sample audio samples in this application are obtained from these two datasets. Furthermore, this application predetermines the sound attributes corresponding to the sample audio samples for different sound categories (e.g., sound attributes include at least one of: sound class, sound frequency, timbre, onomatopoeia, simile, time, emotion, intensity, and texture), and the corresponding sound attribute descriptions are descriptive texts corresponding to the sound attributes. Taking the sound of "machine gun fire" as an example, for "timbre," the corresponding sound attribute description is: the timbre of the sound, that is, the unique quality or feature of the sound; timbre is a rapid, continuous burst.

[0046] S20. Determine the attribute description text corresponding to the sample audio based on the sound attribute description. For example, generate the attribute description text corresponding to the sample audio based on the sound attribute description of at least one sound attribute. For instance, generate the corresponding attribute description text based on the sound attribute descriptions corresponding to "timbre" and "time".

[0047] S30. Train a zero-shot audio classification model based on the similarity between the sample audio and the attribute description text. For example, the zero-shot audio classification model is trained based on the loss between the similarity between the sample audio and the attribute description text and the true similarity.

[0048] In this embodiment, the attribute description text corresponding to the sample audio is determined based on the sound attribute description, so that the description text in this application focuses on diverse sound attributes, and the sound attributes can better represent the intrinsic features of the sound, thereby improving the recognition accuracy of the zero-shot audio classification model trained.

[0049] like Figure 2 The diagram shown is a flowchart of another embodiment of the zero-shot audio classification model training method of this application. In this embodiment, the sound attribute description corresponding to the sound category to which the sample audio belongs is obtained, including:

[0050] S11. Obtain the sound attributes corresponding to the sound category to which the sample audio belongs.

[0051] For example, the sound attributes corresponding to a sound category can be predetermined. This can be achieved by using a large language model (ChatGPT), by human analysis and summarization, or by a combination of both. This application does not impose any specific limitations on this.

[0052] S12. Generate descriptive information corresponding to the sound attributes using a preset language model as the sound attribute description. For example, the preset language model can be a large language model such as ChatGPT. In this embodiment, ChatGPT is used to automatically generate descriptive information (i.e., sound attribute descriptions) focusing on sound attributes for all sound categories.

[0053] This application proposes a series of sound attributes and utilizes the domain knowledge of large language models (e.g., ChatGPT) to generate detailed sound attribute descriptions for each sound category. This provides accurate attribute description information without the need for manual annotation, thus improving the efficiency of generating description information.

[0054] like Figure 3 The diagram shown is a flowchart of another embodiment of the zero-shot audio classification model training method of this application. In this embodiment, the sound attribute description includes descriptive information of multiple attributes. Determining the attribute description text corresponding to the sample audio based on the sound attribute description includes:

[0055] S21. Select some attribute descriptions from multiple attribute descriptions.

[0056] S22. Determine the attribute description text corresponding to the sample audio based on the description information of some attributes.

[0057] In this embodiment, zero-shot audio classification model training is performed using description information of some attributes from the determined description information of multiple attributes. This is because the inventors discovered during the implementation of this application that different sound categories are applicable to different sound attributes. If all attribute description information is simply used for training, it will affect the classification accuracy of the obtained classification model.

[0058] like Figure 4 The diagram shown is a flowchart of another embodiment of the zero-shot audio classification model training method of this application. In this embodiment, the zero-shot audio classification model is trained based on the similarity between the sample audio and the attribute description text, including:

[0059] S31. Perform audio encoding processing on the sample audio to obtain the sample audio embedding vector.

[0060] For example, an audio encoder is used to encode the sample audio to obtain the corresponding sample audio embedding vector.

[0061] S32. Perform text encoding on the attribute description text to obtain the sample text embedding vector.

[0062] For example, a text encoder is used to perform text encoding on the attribute description text to obtain the sample text embedding vector.

[0063] S33. Train a zero-shot audio classification model based on the similarity between the sample audio embedding vector and the sample text embedding vector.

[0064] For example, for a pair of audio clips And category label c, the sound attribute of category label c Commented by ChatGPT. The audio encoder will... This is converted into an embedding vector 'a'. During training, attributes are randomly selected to form descriptive text, which is then encoded accordingly by a text encoder.

[0065]

[0066] Where s is the number of sampled attributes.

[0067] like Figure 5 The diagram shown is a flowchart of another embodiment of the zero-shot audio classification model training method of this application. In this embodiment, the zero-shot audio classification model is trained based on the similarity between the sample audio embedding vector and the sample text embedding vector, including:

[0068] S331. Obtain the audio fully connected projection layer embedding corresponding to the sample audio embedding vector;

[0069] S332. Obtain the text fully connected projection layer embedding corresponding to the sample text embedding vector;

[0070] S333. Train a zero-shot audio classification model based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding.

[0071] For example, a zero-shot audio classification model includes a text encoder and an audio encoder. Training the zero-shot audio classification model based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding includes: training the audio encoder based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding.

[0072] In this embodiment, the similarity function sim(·,·) is used to train the model to minimize the distance between class a and its corresponding class t, while excluding classes a and t from other classes. During inference, all attributes are used to calculate the similarity:

[0073]

[0074]

[0075] Here, Z is the set of zero-sample classes used for testing, which does not intersect with the training class set. Compared with the framework in related technologies, the method in this application makes two modifications:

[0076] Similarity function: Baseline models in related technologies calculate audio-category similarity using a bilinear model:

[0077] s=(W′a)′t (4)

[0078] This application extends it to the cosine similarity between fully connected projection layer embeddings:

[0079]

[0080] In this embodiment, a series of acoustic attributes are manually defined, such as frequency, timbre, onomatopoeia, etc. A large language model, such as ChatGPT, describes a sound category from these aspects. During training, a portion of the attribute list is randomly selected to form the corresponding description. During testing, all attributes are used. Audio data and descriptive text are each encoded to obtain features a and t for two modalities. After mapping, the similarity between audio and text is calculated. During training, the loss function calculates the difference between the similarity and the training label (i.e., whether they match) to obtain the loss value for training. During testing, this similarity is directly used to determine which category the audio belongs to that was not seen in training.

[0081] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0082] In some embodiments, this application also provides a zero-shot audio classification method, comprising: classifying audio using a zero-shot audio classification model trained by the method described in any embodiment of this application.

[0083] In some embodiments, this application provides a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the zero-shot audio classification model training methods described above.

[0084] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described zero-shot audio classification model training methods.

[0085] In some embodiments, this application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a zero-shot audio classification model training method.

[0086] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a zero-shot audio classification model training method according to another embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0087] One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0088] The device for performing the zero-shot audio classification model training method may further include: an input device 630 and an output device 640.

[0089] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0090] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the zero-shot audio classification model training method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the zero-shot audio classification model training method in the above-described method embodiments.

[0091] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created during the use of the zero-shot audio classification model training device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories may be connected to the zero-shot audio classification model training device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0092] Input device 630 can receive input digital or character information, and generate signals related to user settings and function control of the zero-shot audio classification model training device. Output device 640 may include a display device such as a screen.

[0093] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they execute the zero-shot audio classification model training method in any of the above method embodiments.

[0094] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0095] To make the technical solution and effects of this application clearer, the research, implementation, and experimental demonstration of this application are presented as follows:

[0096] Zero-shot audio classification aims to identify and classify sound categories never encountered during model training. This application proposes a zero-shot audio classification method utilizing automatically generated sound attribute descriptions. A series of sound attributes are proposed, and detailed attribute descriptions are generated for each category using domain knowledge from ChatGPT. Unlike previous works that primarily relied on category labels or simple descriptions, this application's descriptions focus on diverse sound attributes, capturing features of different categories. Furthermore, this application employs a contrastive learning approach to enhance the effectiveness of zero-shot learning from text labels. We validate the effectiveness of our method on VGGSound and AudioSet. Results show a significant improvement in zero-shot classification accuracy. Ablation experiments demonstrate a consistent performance improvement regardless of model architecture.

[0097] 1. Introduction

[0098] Supervised learning holds great promise in many fields. However, it is limited to predefined categories included during training. In audio classification, current datasets only contain commonly heard sounds and rarely include domain-specific, rare sounds. Therefore, supervised audio classification requires time-consuming and labor-intensive annotation of unseen sounds, hindering its real-world application.

[0099] To overcome the reliance on meticulously labeled training data, zero-shot audio classification has gained significant attention in recent years. Most studies focus on learning the correlation between audio features and category semantic features. Related techniques use bilinear models to compute audio-category similarity for a given audio embedding and category embedding. Text labels are categories and definitions from Wikipedia; for example, the text label for the crow category is "sounds associated with medium to large birds like the Corvus". Related techniques also incorporate synonyms and semantic extensions as supplementary information. However, labels and supplementary information are often insufficient to distinguish sounds, especially similar ones. Because they define the sound-producing object rather than auditory attributes, they fail to reflect differences between approximate categories (such as machine guns and pistols). Therefore, these methods do not extensively utilize perceptual auditory knowledge to describe sound categories.

[0100] Auditory attributes are more representative of the intrinsic characteristics of sound and can serve as primary markers for sound description. Some studies have explored using sound attributes as supervisory signals for classification, while using attributes as inputs or training targets. In this application, the advantages of semantic and attribute supervision are integrated by learning the correspondence between audio and its description (focusing on sound attributes). This application defines a set of attributes applicable to general sound categories. Then, ChatGPT is used to automatically generate attribute-focused descriptions for all categories. This application leverages the capabilities of large-scale language models (LLMs) to provide accurate attribute descriptions without manual annotation. Furthermore, inspired by the success of self-supervised learning, we apply a more advanced contrastive learning paradigm to zero-point classification, thereby improving the consistency between audio and text labels.

[0101] In summary, this application makes three contributions:

[0102] This application defines a set of attributes for general sound and uses ChatGPT to automatically generate attribute-oriented descriptions.

[0103] This application uses an improved contrastive learning paradigm to learn better audio-text correspondences.

[0104] The results show that zero-shot audio classification performance is significantly improved by utilizing the audio-related domain knowledge in LLM mined from massive text data.

[0105]

[0106] Table 1. Taking machine gun firing as an example, ten properties are defined to describe the sound. To distinguish the "description" provided by AudioSet from the property-oriented description we propose, we use "definition" to refer to the former.

[0107] 2. ChatGPT's description of sound attributes

[0108] To describe the characteristics of sound categories, it is necessary to define a set of sound attributes. While related techniques define some binary attributes, such as duration and materiality, these attributes only cover a small portion of sound features and are, to some extent, only applicable to the collision sound datasets involved. Therefore, we attempt to determine general sound attributes. We ask ChatGPT which attributes can be used to describe sound categories. In addition to the summarized attributes, we manually added three attributes: 1) Onomatopoeia: Onomatopoeia is added in this application to supplement sounds that are difficult to describe with attributes such as pitch and intensity; 2) Timbre: An attribute that distinguishes different sounds, especially sounds with similar pitch; 3) Similar sounds. In zero-shot audio classification, the model can be extensively trained on seen categories. Therefore, information about the similarity between an unseen category and a seen category is very valuable when calculating the probability of an unseen category.

[0109] Based on these attributes, we first ask ChatGPT to generate descriptions for several sound categories. Then, using these as a small set of instances, we guide ChatGPT to generate descriptions for the remaining categories. This context-based learning approach fully leverages the extensive acoustic knowledge learned by large language models. Finally, we generate descriptions for each sound category with nine attributes, as shown in Table 1. For AudioSet, we also use descriptions provided by Wikipedia as additional attributes.

[0110] 3. Zero-shot audio classification through audio-text comparison learning

[0111] 3.1. Comparative Learning Frameworks

[0112] The zero-shot audio classification method in this application is similar to related technologies, with only some modifications. The overall framework is as follows: Figure 7 As shown. For a pair of audio segments And category label c, the sound attribute of category label c Commented by ChatGPT. The audio encoder will... This is converted into an embedding vector 'a'. During training, attributes are randomly selected to form descriptive text, which is then encoded accordingly by a text encoder.

[0113]

[0114] Here, s is the number of sampled attributes. The model is trained using the similarity function sim(·,·) to minimize the distance between a and its corresponding class t, while excluding a and t from other classes. During inference, all attributes are used to calculate similarity:

[0115]

[0116]

[0117] Here, Z is the set of zero-sample classes used for testing, which does not intersect with the training class set. Compared with the framework in related technologies, the method in this application makes two modifications:

[0118] Similarity function: Baseline models in related technologies calculate audio-category similarity using a bilinear model:

[0119] s=(W′a)′t (4)

[0120] This application extends it to the cosine similarity between fully connected projection layer embeddings:

[0121]

[0122] Figure 7 This is a contrastive learning framework for zero-shot audio classification. The text is a description of the category sound attributes generated by ChatGPT. During training, attributes are randomly selected to form the description, while all attributes are used during inference.

[0123] Loss Function: In related techniques, the model is trained using a weighted maximum marginal ranking loss. Given the success of the InfoNCE loss in supervised contrastive learning (Sup-Con), we adopt it in our improved learning framework. For a batch of B samples, the loss is calculated as follows:

[0124]

[0125] Where i, j, k ∈ {1...B}, and τ is the temperature. For the label l i For an audio sample i, descriptions of samples with the same label are considered positive, while descriptions of other samples are considered negative. InfoNCE loss can be used to train a model to distinguish between positive and negative descriptions.

[0126] In summary, the main methods of this application include two types: Baseline and SupCon. In Baseline, bilinear similarity is utilized with a weighted maximum marginal ranking loss. This method is similar to related techniques, using all attributes as text labels. SupCon combines cosine similarity with InfoNCE loss; in this method, attributes are randomly selected during training.

[0127] 3.2. Model Architecture

[0128] To encode detailed attribute descriptions into discriminative embeddings, we use SentenceBERT as the text encoder. We validate the effectiveness of our method on two commonly used audio encoder architectures: PANNs, CNN14, and AST. The pre-trained text encoder was frozen, while the audio encoder was trained from scratch.

[0129] 4. Experimental Setup

[0130] Datasets: While previous zero-shot audio classification work largely used ESC50, we found the results to be inconsistent due to limited data volume. Therefore, we used two large datasets: VGGSound and AudioSet. VGGSound contains over 200,000 segments covering 309 categories. For AudioSet, we selected only single-label audio samples from an imbalanced set, including 21 abstract classes such as "animals" and "music." We also excluded categories with fewer than 100 samples. For categories with more than 1500 samples, we randomly selected 1500 samples to achieve a balanced distribution. Finally, 115k samples and 285 categories were selected for AudioSet.

[0131] Dataset partitioning: We split the dataset into 5 disjoint folds (e.g., 5 disjoint subsets of data) based on the categories. We then train the model on 4 of these folds and evaluate it on the remaining 1 fold. To ensure a balanced distribution of data across the folds, we sort the categories in descending order of sample count. Starting with 5 empty folds, we repeat this process until all categories are assigned: we select the next 5 unassigned categories in sorted order, shuffle them, and assign them to the 5 folds. During the splitting process, each fold of VGGSound generates approximately 35k samples, and each fold of AudioSet generates approximately 23k samples.

[0132] Hyperparameters: Based on relevant technologies, this application employs a two-stage training model: 1) training the audio encoder from scratch; 2) performing baseline training or SupCon audio-text alignment training. In the first stage, the model is trained using an SGD optimizer with an initial learning rate of 0.1 and a cosine learning rate scheduler, with a batch size of 64. In the second stage, we use a learning rate of 1×10⁻⁶. -4 Adam optimizer with a batch size of 256.

[0133] 5. Results and Analysis

[0134] Comparison with baseline models: Table 2 lists the results on the VGGSound and AudioSet datasets. We provide the average accuracy on 3 random seeds. Standard deviations are omitted due to their small size. Note that the "definitions" in Table 1 are not available for VGGSound. Therefore, for text labels, we use only the category as the baseline for VGGSound, and both the category and definition as the baseline for AudioSet. The results demonstrate that supervision using our defined sound attribute descriptions is effective. Regardless of the text labels and datasets used, the SupCo method consistently outperforms the baseline methods. The comparison validates the effectiveness of our improved contrastive learning paradigm.

[0135] Table 2. Zero-shot audio classification performance using the CNN14 encoder architecture. The "definitions" here refer to the Wikipedia description and apply only to AudioSet.

[0136]

[0137] Table 3. Average of VGGSound's zero-shot audio classification performance across 5 tradeoffs using different audio / text encoder architectures.

[0138]

[0139] Model Architecture: To investigate the performance of our method under different model architectures, we modified the audio / text encoder architecture, and the results are shown in Table 3. Although the performance of AST tiny224 is worse than CNN14, the supervision of attribute description does improve upon the text labeling of classes, indicating that our method is feasible. It exhibits strong stability against the audio backbone. Furthermore, performance was further improved by replacing mpnet with the MiniLM text encoder, which performs better in text matching. Our method leverages an advanced text encoder to achieve better zero-shot audio classification performance.

[0140] Attribute Sampling Strategy: As described in Section 3, we randomly sample attributes to form text labels. The class name is always included in our default sampling strategy, which we call "With-class". We compare this with two other strategies: a deterministic strategy and a random strategy, to study the impact of the sampling strategy. The deterministic strategy means we select all attributes, while the random strategy means all attributes are randomly sampled. The results in Table 4 show the superiority of our default strategy. The class label is an important and clean attribute, while the descriptions of other attributes may contain noise. Therefore, the randomness of attributes other than the class should also be incorporated into the training process.

[0141] Table 4. Average zero-shot classification performance of VGGSound using different attribute sampling strategies at 5 folds.

[0142]

[0143] Case Study: To analyze the improvements brought about by attribute descriptions, we selected several cases from different trade-offs and presented them in... Figure 8 The numbers in the shaded and white boxes (corresponding to italics) represent the number of correctly classified and misclassified samples, respectively. These categories are easily misclassified because they have similar sound features or their labels are semantically close. Additional attributes describe unique sound features to distinguish similar categories. For example, onomatopoeia are used to mimic the sounds of a dog growling and barking to highlight the difference between the two.

[0144] Figure 2 Performance improvements after adding attributes. Shaded and white bars represent true classification and misclassification, respectively. Corresponding attribute descriptions are provided: ono. = onomatopoeia, simi. = imitation, emo. = emotion.

[0145] 6. Conclusion

[0146] This application proposes a zero-shot audio classification method using sound attribute descriptions. This method leverages the rich knowledge in ChatGPT, combining the advantages of semantic and attribute supervision. We define a set of sound attributes and use ChatGPT to automatically describe the sound category based on each attribute. Furthermore, an improved contrastive learning paradigm is proposed to enhance the model's ability to learn effectively from text labels. Significant improvements in zero-shot sampling accuracy on VGGSound and AudioSet demonstrate the high effectiveness of our method. Ablation studies on encoder architectures show continuous improvement over baselines, validating that our method is architecture-independent.

[0147] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0148] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0149] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0150] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0151] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0152] (5) Other electronic devices with data interaction functions.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a zero-shot audio classification model, wherein the zero-shot audio classification model includes a text encoder and an audio encoder, the method comprising: Obtain the sound attribute description corresponding to the sound category of the sample audio; Determine the attribute description text corresponding to the sample audio based on the sound attribute description; A zero-shot audio classification model is trained based on the similarity between the sample audio and the attribute description text. The training of the zero-shot audio classification model based on the similarity between the sample audio and the attribute description text includes: The sample audio is subjected to audio encoding processing to obtain the sample audio embedding vector; The attribute description text is processed by text encoding to obtain the sample text embedding vector; A zero-shot audio classification model is trained based on the similarity between the sample audio embedding vector and the sample text embedding vector. The training of the zero-shot audio classification model based on the similarity between the sample audio embedding vector and the sample text embedding vector includes: Obtain the audio fully connected projection layer embedding corresponding to the sample audio embedding vector; Obtain the text fully connected projection layer embedding corresponding to the sample text embedding vector; The audio encoder is trained based on the cosine similarity between the audio fully connected projection layer embedding and the text fully connected projection layer embedding.

2. The method according to claim 1, characterized in that, Obtain the sound attribute description corresponding to the sound category of the sample audio, including: Obtain the sound attributes corresponding to the sound category of the sample audio; A pre-defined language model is used to generate descriptive information corresponding to the sound attributes as the sound attribute description.

3. The method according to claim 1, characterized in that, The sound attribute description includes descriptive information for multiple attributes. Determining the attribute description text corresponding to the sample audio based on the sound attribute description includes: Select descriptions of some attributes from the descriptions of multiple attributes; The attribute description text corresponding to the sample audio is determined based on the description information of the aforementioned partial attributes.

4. The method according to any one of claims 1-3, characterized in that, Sound attributes include at least one of the following: sound category, sound frequency, timbre, onomatopoeia, metaphor, time, emotion, intensity, and texture.

5. A zero-shot audio classification method, comprising: The zero-shot audio classification model trained using any one of claims 1-4 is used to classify audio.

6. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-5.

7. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

Images