Automatic audio labeling system and method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an automatic audio labeling system and method, electronic equipment and a storage medium, and the automatic audio labeling method comprises the steps: obtaining an audio signal, and extracting an audio representation from the audio signal based on an EAT model; aligning the audio representation and the text embedding corresponding to the audio representation based on the linear projection layer, and performing down-sampling to obtain a first text embedding; generating text description at least based on the large language model according to the aligned and down-sampled audio representation and text embedding; and the similarity between each text description and the audio signal is calculated based on the strategy calculation module, and the most matched text description is selected as the output of the automatic audio labeling system.

Description

Technical Field

[0001] The present invention belongs to the technical field of large-scale language models, and in particular relates to an automatic audio annotation system, method, electronic device and storage medium. Background Art

[0002] Among related technologies, Automatic Audio Captioning (AAC) is a challenging multimodal task that aims to generate natural results from audio data. Unlike traditional audio understanding tasks such as Audio Tagging (AT), AAC requires the system to not only understand the content of the audio clip, but also unify the text and sound patterns to ultimately generate a coherent and fluent description.

[0003] In AAC tasks, the sequence-to-sequence (seq2seq) architecture is widely used, in which the audio encoder extracts sound representations and the language model uses these representations to generate annotations through autoregression. Traditional methods usually rely on supervised models (such as PANNs (Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition)). Recognition, a large-scale pre-trained audio neural network for audio pattern recognition) for audio feature extraction. Recently, self-supervised pre-training models such as BEATs (Audio pre-training with acoustictokenizers) have been integrated into AAC systems, significantly improving system performance.

[0004] In the AAC task, the lack of high-quality audio and text paired datasets is a huge challenge, which highlights the importance of effective data augmentation techniques to improve model performance. In automatic speech recognition (ASR), the n-best re-scoring strategy is widely used to reduce the word error rate (WER), that is, training a language model to score and select the most accurate result from the n-best decoding candidate list. Summary of the invention

[0005] Embodiments of the present invention provide an automatic audio annotation system, method, electronic device and storage medium, which are used to solve at least one of the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides an automatic audio tagging system, comprising: an EAT model, used to extract an audio representation from an audio signal; a linear projection layer, used to align and downsample the audio representation and a text embedding corresponding to the audio representation to obtain a first text embedding; a large language model, used to generate a text description based on at least the aligned and downsampled audio representation and the text embedding; and a strategy calculation module, used to calculate the similarity between each of the text descriptions and the audio signal and select the most matching text description as the output of the automatic audio tagging system.

[0007] In a second aspect, an embodiment of the present invention provides an automatic audio tagging method, comprising: obtaining an audio signal, extracting the audio representation from the audio signal based on the EAT model; aligning the audio representation and the text embedding corresponding to the audio representation based on the linear projection layer and downsampling to obtain a first text embedding; generating a text description based on the aligned and downsampled audio representation and text embedding at least based on the large language model; calculating the similarity between each of the text descriptions and the audio signal based on the strategy calculation module and selecting the most matching text description as the output of the automatic audio tagging system.

[0008] In a third aspect, an embodiment of the present invention further provides a computer program product, wherein the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the automatic audio tagging method of any embodiment of the present invention.

[0009] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in the second aspect.

[0010] In a fifth aspect, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the second aspect.

[0011] In the method of the embodiment of the present application, AAC is enhanced by LLM enhanced parsing and CLAP-Refine. We use the EAT encoder to extract audio representations, which are then downsampled and aligned with text embeddings through a linear projection layer. Further, decoding is performed by LLM Vicuna, and fine-tuning is limited to the projector and LoRA (low-rank adapter) adapter to improve training efficiency. The SLAM-AAC of this application also applies text enhancement technology to increase the annotation diversity of audio clips in Clotho, thereby improving the versatility of the model. In addition, we also propose CLAP-Refine, a plug-and-play decoding strategy that strengthens annotation selection in post-processing. This strategy uses multiple beam search outputs as candidates and selects the final annotation based on the highest similarity score with the input audio calculated by the CLAP model. Experiments show that on the AudioCaps and Clotho datasets, the SLAM-AAC of the embodiment of the present application outperforms existing AAC models, and the reduction study verifies the contribution of each module in improving the overall performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0013] Figure 1 A flowchart of an automatic audio tagging method provided by an embodiment of the present invention; Figure 2 An overview of the SLAM-AAC system provided by one embodiment of the present invention; Figure 3 An example of back-translated text in the Clotho dataset provided by an embodiment of the present invention; Figure 4 The figure shows the performance comparison of the AAC model provided by an embodiment of the present invention in terms of Clotho and AudioCaps evaluation scores; Figure 5 The figure shows a reduction test of Clotho and AudioCaps evaluation scores by SLAM-AAC provided by an embodiment of the present invention; Figure 6 A comparison of different text decoding strategies provided by an embodiment of the present invention is shown; Figure 7 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] AAC is a challenging multimodal task that aims to generate natural text descriptions from input audio data (mainly referring to sound events such as ambient sounds and background sound effects). Unlike traditional audio understanding tasks (such as audio tagging), AAC requires not only that the system understand the content of the audio clip, but also that the text and audio information be accurately aligned to ultimately generate a fluent and coherent text description. In recent years, with the rapid development of audio pre-training models and large language models (LLMs), the AAC field has made significant progress, which provides new possibilities for further improving audio understanding and text reasoning capabilities.

[0016] At present, academia and industry have proposed a variety of models and technologies to complete AAC-related tasks. For example, the EnCLAP model, the WavCaps model, and the AAC system based on fine-grained audio representation extraction, text embedding supervision and LLM. In addition, there are AAC systems trained with automatic speech recognition (ASR). Most of these systems rely on the combination of audio encoders and text decoders. For example, EnCLAP and WavCaps both use the supervised model HTS-AT as the audio encoder and use the BART language model for text decoding. It is worth noting that WavCaps has achieved significant performance improvements through pre-training on large-scale weakly labeled datasets. Some research teams use the self-supervised model BEATs for audio feature extraction. The latter uses the more advanced Vicuna large language model instead of the traditional BART model for text decoding, demonstrating stronger text generation capabilities.

[0017] Although the existing technologies have made significant progress in AAC tasks, there are still many defects. First, the current systems perform poorly in terms of consistency between the generated audio subtitles and the input audio, especially the details in the audio are easily overlooked. Most systems still use the earlier supervised model HTS-AT for audio feature extraction, and although self-supervised models (such as BEATs) can extract more fine-grained audio representations, they also face the problem of insufficient processing of complex audio data. In addition, the latest self-supervised audio models such as EAT and ATST have not been fully applied, and these models have the potential to further improve the sophistication of audio feature extraction. Secondly, most of these AAC systems lack effective post-processing mechanisms, and usually directly use a single beam search result or top-p sampling result as the final generated subtitles, resulting in unstable scores for the subtitles generated by the system. In terms of text generation, except for the AAC system of a certain university, most existing systems still use smaller-scale language models for text decoding, which limits the reasoning ability and fluency of the model in the text generation process. In addition, these technologies have limited robustness and it is difficult to achieve high scores on multiple evaluation benchmarks (such as AudioCaps and Clotho) at the same time. The main reason lies in the limitations of data enhancement methods. Most systems rely on collecting more audio-text pairs to improve the generalization ability of the model. However, a team used a hybrid enhancement strategy based on ChatGPT in its AAC system. By automatically expanding and enhancing existing audio-text pairs, it solved the problem of data scarcity to a certain extent, but there is still room for optimization overall, and no other data enhancement methods were explored.

[0018] In summary, although the current AAC technology has made some progress, there is still significant room for improvement in audio detail extraction, text generation capabilities, and system robustness. Based on this, our (X-LANCE) team members proposed an automatic audio annotation system that combines text retelling enhancement with audio-text alignment post-processing, aiming to address many defects in existing technologies and further improve the performance of AAC systems.

[0019] Currently in the industry, solutions to these defects are generally concentrated in several aspects. First, in terms of audio representation extraction, many practitioners tend to choose self-supervised audio encoders with better performance, such as the BEATs model adopted by a certain team. This model has certain advantages in extracting fine-grained audio representations. However, this year I proposed the EAT (EfficientAudio Transformer) model, which is an efficient audio self-supervised learning model that learns audio features through masked language modeling (MLM) as a self-supervised task. Compared with BEATs, EAT performs better in extracting fine-grained audio representations. Although the model has not yet been widely used, it has shown significant performance improvements in automatic audio labeling tasks.

[0020] In terms of text decoding, past AAC systems usually use smaller language models for text generation. Although these models can generate subtitles of a certain quality, they have limitations in reasoning ability and generation fluency. To this end, we use a large language model (LLM) as a text decoder and further introduce the Vicuna model, an LLM that excels in reasoning ability. By combining a low-rank adapter (LoRA), we are able to significantly reduce training costs while ensuring efficient fine-tuning, thereby improving the quality and diversity of subtitle generation.

[0021] In addition, many AAC systems usually use the beam search method when generating subtitles, directly selecting the decoding result with the highest score as the final output. However, this method cannot ensure the highest match between the generated subtitles and the audio in all cases. I proposed an innovative audio-text alignment post-processing algorithm, namely the CLAP-Refine strategy. By using the CLAP model (Contrastive Language-Audio Pre-training model), the algorithm can calculate the similarity between the audio and the generated subtitles, and re-rank multiple generated candidate subtitles based on this. The CLAP model is trained through contrastive learning, using a text-audio dual encoder to process text and audio data separately, and project them into the same feature space, so that the match between them can be evaluated by calculating the cosine similarity of audio and text. By sorting the similarity of multiple beam search results, we can select the subtitle that best matches the input audio as the final output, which greatly improves the accuracy of the system's generated subtitles.

[0022] To further enhance the robustness of the system, many teams have adopted data augmentation methods, aiming to improve the generalization ability of the model by diversifying the training data. However, most of the current enhancement methods are limited to simple audio masking or data augmentation. Only the Carnegie Mellon University team proposed an audio-text hybrid enhancement strategy based on ChatGPT. In contrast, I introduced a text retelling enhancement method based on the back-translation technology in machine translation. By translating the English annotations of the audio into Chinese and then translating them back to English, we can generate enhanced text with the same meaning as the original annotations but different words, thereby greatly increasing the language diversity and vocabulary in the dataset. This method is not only applicable to multilingual audio annotation tasks, but also significantly improves the generalization ability of the model in low-data scenarios.

[0023] In the actual implementation process, as shown in the flowchart, we first use the frozen EAT model to extract fine-grained representations of the audio, and then align and downsample these representations with the text embedding through a linear projection layer. In the text generation stage, based on these aligned representations, a large language model fine-tuned with LoRA is used to generate text descriptions. During the reasoning process, the system generates multiple candidate subtitles through multiple beam searches, and uses the CLAP-Refine strategy to calculate the similarity between each candidate subtitle and the audio, and finally selects the most matching subtitle as the system output. In addition, during the model training process, we not only used the original dataset, but also added a dataset enhanced by text restatement to further improve the generalization performance of the system.

[0024] During the development of the embodiments of this application, we have conducted in-depth discussions on multiple alternatives. In terms of text restatement enhancement, in addition to the back-translation-based enhancement method that was finally selected, we also explored the use of large language models (LLMs) for text restatement. This method uses LLM to generate restatement texts with different styles according to different prompts. The advantage is that the diversity of text generation is high, and texts with different styles and expressions can be flexibly generated according to needs. However, the disadvantages of LLM restatement are also obvious: first, the cost of calling LLMAPI is high, especially when large-scale text enhancement is required, the cost is more significant. Secondly, in order to obtain the expected restatement text, the user must spend extra effort to design high-quality prompts. Therefore, although the LLM restatement enhancement method has certain potential, considering the convenience and cost issues at this stage, we finally chose a more efficient and low-cost back-translation enhancement method. However, with the further maturity and popularization of LLM technology, we will continue to explore enhancement methods based on LLM restatement, and I believe that this will bring more innovation opportunities in the future. In the design of the audio-text alignment post-processing algorithm, we initially tried to use the n-best method to screen text candidates. In this scheme, we retained the top four search results with the highest probability through beam search (for example, when the beam is 4), and post-processed them as candidate texts. The advantage of this method is that multiple text candidates can be obtained by performing only one beam search, thus saving computing resources and time. However, the experimental results show that the similarity between these candidate texts is high, and the effect of using CLAP-Refine to screen them is not significant, and the score improvement of generated subtitles is relatively limited. Therefore, in the end, we still used different beam searches to generate text candidates with greater diversity, which not only enriched the diversity of candidate texts, but also better played the advantages of CLAP-Refine in post-processing.

[0025] In terms of effect, the AAC system proposed in this scheme has achieved significant performance improvement on the AudioCaps and Clotho datasets, especially on the AAC standard evaluation indicators, including METEOR, CIDEr, SPICE, SPIDEr, SPIDEr-FL and FENSE, all of which have achieved the best results. Through ablation experiments, we verified the key role of text restatement enhancement and audio-text alignment post-processing in improving model performance. In further research on the audio-text alignment post-processing algorithm, we found that after obtaining the ranking set based on the similarity score sorting, the higher-ranked text candidate set showed better results, proving the effectiveness of CLAP-Refine in optimizing beam search output. However, when we compare these results with the theoretical optimal value (i.e., the decoded text with the highest FENSE score), there is still a certain gap. This shows that the theoretical optimal decoding results may be distributed in different beam searches, suggesting that we still have considerable potential for improvement in post-processing and selection strategies based on the current AAC model. In the future, we will further study how to gradually narrow this gap and get closer to the theoretical optimal performance by optimizing the post-processing method.

[0026] In a specific embodiment, the embodiment of the present application provides an automatic audio tagging system, including: an EAT model, used to extract an audio representation from an audio signal; a linear projection layer, used to align and downsample the audio representation and a text embedding corresponding to the audio representation to obtain a first text embedding; a large language model, used to generate a text description based on at least the aligned and downsampled audio representation and the text embedding; and a strategy calculation module, used to calculate the similarity between each of the text descriptions and the audio signal and select the most matching text description as the output of the automatic audio tagging system.

[0027] In some optional embodiments, the large language model further includes a large language model tokenizer for processing the text prompt and the real subtitle to obtain a second text embedding and a length of the real subtitle.

[0028] In some optional embodiments, the automatic audio tagging system further includes a low-rank adapter for fine-tuning the large language model.

[0029] In some optional embodiments, only the linear projection layer and the low-rank adapter are trainable, and the rest of the system remains frozen.

[0030] In some optional embodiments, the strategy calculation module is CLAP-Refine, which projects the text description and the audio signal into a shared space through comparative pre-training to calculate text similarity, evaluates the similarity between the candidate annotations generated by multi-beam search decoding and the audio signal, and finally selects the annotation with the highest score as the output of the automatic audio annotation system.

[0031] Please refer to Figure 1 , which shows a flowchart of an embodiment of an automatic audio annotation method of the present application.

[0032] like Figure 1 As shown, in step 101, an audio signal is acquired, and the audio representation is extracted from the audio signal based on the EAT model; In step 102, the audio representation and the text embedding corresponding to the audio representation are aligned and downsampled based on the linear projection layer to obtain a first text embedding; In step 103, a text description is generated based on at least the large language model according to the aligned and downsampled audio representation and the text embedding; In step 104, the similarity between each of the text descriptions and the audio signal is calculated based on the strategy calculation module, and the most matching text description is selected as the output of the automatic audio tagging system.

[0033] In some optional embodiments, generating a text description based on the aligned and downsampled audio representation and the text embedding at least based on the large language model includes: during training, concatenating the first text embedding, the second text embedding, and the length of the real subtitle using the large language model fine-tuned by the low-rank adapter to obtain a first joint embedding, and training the large language model based on the joint embedding and cross-entropy loss; during inference, concatenating the first text embedding and the second text embedding to obtain a second joint embedding for inputting into the large language model to generate a text description.

[0034] In the method of the embodiment of the present application, AAC is enhanced by LLM enhanced parsing and CLAP-Refine. We use the EAT encoder to extract audio representations, which are then downsampled and aligned with text embeddings through a linear projection layer. Further, decoding is performed by LLM Vicuna, and fine-tuning is limited to the projector and LoRA (low-rank adapter) adapter to improve training efficiency. The SLAM-AAC of this application also applies text enhancement technology to increase the annotation diversity of audio clips in Clotho, thereby improving the versatility of the model. In addition, we also propose CLAP-Refine, a plug-and-play decoding strategy that strengthens annotation selection in post-processing. This strategy uses multiple beam search outputs as candidates and selects the final annotation based on the highest similarity score with the input audio calculated by the CLAP model. Experiments show that on the AudioCaps and Clotho datasets, the SLAM-AAC of the embodiment of the present application outperforms existing AAC models, and the reduction study verifies the contribution of each module in improving the overall performance of the model.

[0035] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the specified step order, and this application has no limitation on this.

[0036] The solution of the present application is described below through a specific embodiment so that those skilled in the art can better understand the solution of the present application.

[0037] In the embodiment of the present application, the SLAM-AAC model we proposed uses the efficient audio Transformer architecture EAT (Efficient Audio Transformer) (a self-supervised pre-training model that has achieved state-of-the-art performance in audio tagging tasks) as an audio encoder to extract more refined audio representations, where SLAM-AAC is a subproject of SLAM-LLM, where SLAM stands for speech, language, audio, and music. The project is open source and available at https: / / github.com / X-LANCE / SLAM-LLM. In order to improve computational efficiency and improve the consistency between audio and text embeddings, we use a lightweight linear layer to downsample the 50Hz audio representation to approximately 10Hz. In terms of text decoding, the emergence of LLMs has proven that large language models have superior understanding and reasoning capabilities compared to small models, and can generate smoother and more natural text for AAC tasks. Therefore, we use the large language model Vicuna as a text decoder, which can process aligned audio and text representations and generate corresponding results. To further improve training efficiency, SLAM-AAC integrates the LoRA (Low Rank Adapter) method for parameter-efficient fine-tuning (PEFT) of LLM, while freezing EAT and only training the alignment layer.

[0038] In SLAM-AAC, we employ audio and text augmentation techniques during model training. For audio augmentation, we use SpecAugment to proportionally mask the audio spectrogram in both time and frequency dimensions to enhance the robustness of the model. In previous AAC approaches, text augmentation involves synonym replacement using WordNet, or modifying words with low TF-IDF scores to preserve the information content of key words. Here, we introduce sentence-level augmentation, where instead of word-level replacement, text is generated for each audio annotation via a back-translation approach. This approach extends the Clotho training set, which has limited data compared to other AAC datasets such as AudioCaps and WavCaps, thereby increasing the diversity and complexity of SLAM-AAC pre-training data.

[0039] We introduce a plug-and-play CLAP-Refine strategy to enhance text decoding. CLAP projects text and audio features into a shared space through contrastive pre-training, which allows the calculation of text-audio similarity. By evaluating the similarity of candidate annotations generated by different beam search decodings with the input audio, our proposed CLAP-Refine selects the annotation with the highest score as output. This approach enables SLAM-AAC to effectively utilize different beam search results to provide the best match for the input audio. SLAM-AAC achieves state-of-the-art performance on all AAC metrics in both AudioCaps and Clotho evaluations.

[0040] Figure 2 Figure 1 shows an overview of the SLAM-AAC system. We use frozen EAT to extract fine-grained audio representations, which are then downsampled and aligned with text embeddings by a linear projector. The LLM for decoding generates text based on these concatenated representations and uses LoRA (Low Rank Adapter) for efficient fine-tuning. During inference, multiple candidate annotations are generated through various beam searches, and the CLAP-Refine strategy is used to select the result with the highest audio alignment as the final output. Here, B n Represents the candidate headers generated using the n-bundle size when decoding.

[0041] SLAM-AAC Network Architecture According to the existing AAC model, SLAM-AAC adopts a sequence-to-sequence framework, such as Figure 2 shown.

[0042] In terms of audio coding, SLAM-AAC uses the Transformer-based model EAT as an acoustic feature extractor. EAT is an efficient audio Transformer architecture that uses masked language modeling as a pre-task within the self-distillation framework. It has significantly improved performance over supervised models such as PANNs and self-supervised models such as BEATs, especially in audio classification tasks such as AS-2M and AS20K. In our experiments, we use the EAT-base model (EAT-base) pre-trained and fine-tuned on the AudioSet dataset. 2EAT resamples the input waveform to 16kHz and converts it to 128-dimensional Mel-frequency bands using a 25ms Hanning window and 10ms shift. These Mel-spectrograms are then converted to 2D patch embeddings through a CNN encoder and then passed through a 12-layer ViT-B model for feature extraction. The resulting audio representation EAT a The frequency is about 50Hz. To facilitate the alignment of audio-text patterns and reduce the length of feature sequences, we use a lightweight 2-layer linear projection to downsample the audio tags E by 5 times. a Converted to E with a frequency of about 10Hz A .

[0043] To enhance text generation, our method adopts LLM Vicuna as the decoder. Figure 2 As shown, SLAM-AAC uses Vicuna’s tokenizer to process text prompts (such as “Describe the audio you hear”) and groundtruth captions, and uses a 32K vocabulary to generate the corresponding text embedding E P and E T During training, the text annotations are incorporated using teacher-forcing, and the joint embedding word E required by the decoder J It is formed by concatenating these embedded words as follows (where during training: training, during inference: reasoning): Let |ET| denote the ground truth of the length of the text embedding, and let E T,t represents the tth token of ET. The training objective of our system is the cross entropy loss, which is defined as To improve training efficiency, we fine-tune LLM with LoRA (Low Rank Adapter). Therefore, only the linear projections and LoRA (Low Rank Adapter) adapters are trainable, while the rest of the model remains frozen.

[0044] Paraphrasing Augmentation in AAC In AAC datasets, audio annotations are usually made by a limited number of annotators, which leads to a consistent annotation style and limited vocabulary, thus hindering the overall diversity of the dataset. Specifically, the Clotho dataset, which is part of the SLAM-AAC training data, has audio clips of various lengths. However, compared with other datasets such as AudioCaps and WavCaps, the size of the Clotho training set is relatively small, with only 3,839 audio clips, each paired with five annotation annotations. To alleviate this limitation, we innovatively introduced a text augmentation method based on the back-translation method to expand the audio-text pairing in Clotho. This method aims to enrich the dataset with diverse annotations and enhance the versatility of the model.

[0045] Figure 3 An example of back-translated text from the Clotho dataset is shown.

[0046] Figure 3 An example of the back-translation method applied to Clotho is provided. In the experiment, we used the Google Translate API to translate each English annotation into Chinese and then back to English, generating five additional annotations for each audio clip. As a result, the total vocabulary of the annotations increased from 7454 words to 10453 words. Compared with previous word-level text augmentation techniques, this paraphrase technique can perform a wider range of sentence reorganizations while preserving the original semantics.

[0047] CLAP-Refine for Text Decoding Traditional AAC systems usually use beam search or kernel sampling for text decoding. However, beam search mainly focuses on maximizing the score of the text decoder during the decoding process. The alignment between the generated text and the audio embedding is usually ignored. On the other hand, kernel sampling leads to unstable output, which usually requires multiple decoding attempts and extensive post-processing to achieve optimal results.

[0048] To address these limitations, we introduce an innovative plug-and-play strategy called CLAP-Refine to enhance the decoding results by effectively utilizing multi-beam search as a post-processing step. CLAP constructs an implicit audio-text multimodal semantic space through contrastive learning. It uses a dual encoder to process text and audio data independently and generate the corresponding audio representation C in a shared representation space. A and text representation C TThe model can measure the alignment of text-audio pairs using cosine similarity, which is defined as follows: During reasoning, SLAM-AAC first generates the highest probability sentences with different beam sizes for the same input audio, resulting in a set of candidate annotations B1, B2, ..., B n ,like Figure 2 As shown in Figure 2. The CLAP-Refine strategy then uses the CLAP model to compute similarity scores between each candidate annotation and the input audio. These scores are used to re-rank the candidate captions and refine the output results, prioritizing those that are closest to the audio. The annotations with the highest scores are selected as the final output. In this work, we used the CLAP model with the audio encoder HTS-AT and the text encoder RoBERTa.

[0049] Dataset SLAM-AAC is pre-trained using four key AAC datasets: Clotho, AudioCaps, WavCaps, and MACS. For Clotho, we use version 2.1, which contains audio clips lasting 15 to 30 seconds with captions ranging from 8 to 20 words. The dataset includes 3839 training audio examples, 1045 validation audio examples, and 1045 evaluation audio examples, each paired with 5 annotations. AudioCaps contains more than 50,000 10-second audio clips from AudioSet. It is divided into a training set (49,274 clips, one annotation per clip), a validation set (494 clips, five annotations per clip), and a test set (957 clips, five annotations per clip). WavCaps contains 403,050 audio clips from AudioSet, BBC Sound Effects, FreeSound, and SoundBible. MACS consists of 3930 10-second audio files, each with 2 to 5 annotations, recorded in three acoustic scenes (airport, public square, and park) in the TAU Urban Acoustic Scenes 2019 dataset.

[0050] In our experiments, the pre-training data includes the training sets from Clotho, AudioCaps, and MACS, as well as the entire WavCaps dataset. In addition, as shown before, we also augment the Clotho training set using the proposed paraphrase method.

[0051] Evaluation Metrics To evaluate the quality of the generated audio annotations, we used several standard AAC metrics: METEOR, CIDEr, SPICE, SPIDEr, SPIDEr-FL, and FENSE. METEOR takes into account single-word precision, recall, synonyms, and stemming. CIDEr measures the consensus between the generated and reference texts using TF-IDF weighted n-grams. SPICE compares the semantic graphs of the generated and reference titles. SPIDEr linearly combines CIDEr and SPICE for a balanced grammatical and semantic evaluation. SPIDEr-FL extends this with a fluency error detector from FENSE, a BERT-based model trained on audio annotations that penalizes a sentence’s SPIDEr score if the probability of a fluency error exceeds 90%. FENSE combines the semantic similarity from Sentence-BERT with this fluency error detection for a comprehensive evaluation. Higher values ​​for all metrics indicate better performance.

[0052] Training and Inference Details Our model was initially trained on the pre-training dataset detailed in the previous examples, with a batch size of 16, a peak learning rate of 1e-4, and 100,000 updates. The learning rate was controlled by a linear decay schedule, with a warm-up phase of 1,000 iterations before the linear decay. The model was then fine-tuned for 10 epochs on the Clotho and AudioCaps training sets, respectively, with a batch size of 4 and a peak learning rate of 8e-6. During training, the LoRA adapter was integrated into Vicuna to refine the q and v projection layers in the Transformer block. The model was validated every 500 updates, and checkpoints were saved based on the lowest validation loss. The pre-training and fine-tuning processes took about 26 hours and 5 hours, respectively, on an NVIDIA A800 GPU.

[0053] The CLAP model used for CLAP-Refine decoding is trained on AudioCaps, Clotho, and WavCaps with a batch size of 128 and a peak learning rate of 5e-5 for 15 epochs. The training uses a cosine annealing schedule with 2 warm-up phases, and the model is saved at the minimum of the validation loss.

[0054] During inference, SLAM-AAC uses beam search with beam sizes ranging from 2 to 8, generating decoding candidates within each beam size based on the highest probability. The final output annotation is then determined using CLAP-Refine, which selects the candidate annotation with the highest similarity score to the input audio.

[0055] Experimental Results We compared the performance of our SLAM-AAC model on Clotho and AudioCaps with the best existing AAC models. Both EnCLAP and WavCaps use the supervised model HTS-AT as the audio encoder and BART as the text decoder, with WavCaps benefiting from pre-training on a large-scale, weakly labeled dataset. Wu et al. and Tang et al. used BEATs for audio feature extraction, and Tang et al. further used a large language model Vicuna instead of BART for text decoding. Wu et al. integrated ChatGPT hybrid enhancements and fine-tuned their model on Clotho to validate its performance, while Tang et al. integrated the Whisper speech encoder and additional speech data for pre-training and validated their model on AAC and ASR tasks.

[0056] Figure 4 The figure shows the performance comparison of AAC models on CLOGO and AUDIOCAPS evaluation scores. Pre-training datasets: AudioCaps (AC), Clotho (CL), WavCaps (WC), MACS (MA), LibriSpeech (LS), and GigaSpeech (GS). Metrics: METEOR (MT), CIDEr (CD), SPICE (SC), SPIDEr (SD), SPIDEr-FL (SF), and FENSE (FS). MT, CD, SC, SD, SF, FS: METEOR CIDEr SPICESPIDEr SPIDEr-FL FENSE, are the names of evaluation metrics, AC, CL, WC, MA, LS, GS: AudioCaps, Clotho, WavCaps, MACS, LibriSpeech, and GigaSpeech, are the names of English datasets, and Ours: Our model. CLC and CLP represent the Clotho training set augmented using the ChatGPT hybrid method and our back-translation method, respectively. Superscript a indicates that for open source models, we evaluate them on metrics not reported in the original paper using our evaluation splits. Superscript b indicates that the best performance of WavCaps on two datasets is selected for comparison (the model architecture may be different).

[0057] like Figure 4As shown in Figure 2, SLAM-AAC achieves state-of-the-art performance in all tasks. On the AudioCaps dataset, SLAM-AAC achieves significant improvements (≥1%) in all metrics compared to existing AAC systems, indicating that the generated annotations are more consistent with human annotations in terms of semantic and lexical similarity.

[0058] Ablation studies We conduct a comprehensive subtraction study to verify the effectiveness of each component in SLAM-AAC.

[0059] Network Architecture We studied the impact of different audio encoders and the necessity of fine-tuning LLM. Specifically, we replaced the commonly used audio encoder BEATs with EAT, which showed superior performance in audio tagging tasks. As shown in Table 3, this replacement brought significant model performance improvements, especially in the AudioCaps evaluation. These findings show that the choice of audio encoder is still a key bottleneck in AAC systems, and more powerful encoders help extract fine-grained audio features and improve the model's overall understanding of audio. In addition, our ablation study on LoRA (Low Rank Adapter) highlights the importance of efficient LLM fine-tuning. It turns out that the inclusion of the LoRA (Low Rank Adapter) adapter is crucial for adjusting text and audio patterns, which can effectively utilize the pre-trained knowledge of LLM to complete AAC tasks.

[0060] Paraphrasing Augmentation Figure 5 Figure 2 shows the subtraction study of SLAM-AAC on Clotho and AudioCaps evaluation scores. The subtracted parts are marked by underline. Clotho Refers to the restatement enhancement performed on the Clotho training set during model pre-training, that is, the Clotho dataset using restatement enhancement. Among them, MT, CD, SC, SD, SF, FS: METEOR CIDEr SPICE SPIDErSPIDEr-FL FENSE, are the names of evaluation indicators, w / o: removed, LoRA: low-rank adapter, Beam search: beam search, EAT, BEATs, CLAP-Refine, SLAM-AAC: model / method names.

[0061] Figure 5The results of our ablation study on paraphrase augmentation are presented. Incorporating text augmentation based on a back-translation approach into the Clotho training dataset during pre-training significantly improves the model's performance on the Clotho and AudioCaps benchmarks. Surprisingly, using augmented Clotho data significantly improves the AudioCaps evaluation scores by more than 1% on the CIDEr, SPIDEr, and SPIDEr-FL metrics. These findings demonstrate that text augmentation effectively enriches data vocabulary and diversity, thereby improving the robustness and generalizability of the model, especially when trained with limited audio-text paired data.

[0062] Text Decoding Strategies Figure 6 Comparison of audio monitoring decoding strategies is shown. The gray row is the oracle result (theoretically optimal performance) from multiple beam searches. Our results achieve the highest rearrangement score in Clap-refine, which is marked in bold.

[0063] Figure 6 Different text decoding strategies are compared. The results show that kernel sampling (temperature 0.5, top-p 0.95) outputs low CIDEr and SPIDEr scores, indicating a lack of consistency with human annotations, and therefore this approach is not well suited for the AAC task. For beam search, the generated annotation bundle sizes range from 2 to 8, denoted as B2, B3, ..., B8. The best performance is achieved when the bundle size is 4 (B4), which is also the baseline for comparison. The CLAP-Refine strategy assigns an audio-text similarity score to each bundle result and re-ranks them based on these scores, ultimately forming 7 ranked sets, with the highest ranked set representing the final output annotation. For clarity, we show the results of the first, third, fifth, and seventh ranked sets. As Figure 6 As shown in Figure 2, the higher-ranked sets consistently outperform the others, indicating that CLAP-Refine is very effective in refining the beam search output. However, there is still a large gap compared to the theoretical optimal decoding result (i.e., the decoded text with the highest FENSE score among all beam results, called the oracle). This suggests that the optimal decoding result may span different beam searches, indicating that there is considerable potential for improving post-processing and selection strategies based on current AAC models.

[0064] V. Conclusion and Future Work In this work, we propose SLAM-AAC, which enhances AAC through LLM enhanced parsing and CLAP-Refine. We use the EAT encoder to extract audio representations, which are then downsampled and aligned with text embeddings through a linear projection layer. Decoding is performed by LLM Vicuna, and fine-tuning is limited to the linear projection layer (projector) and the LoRA (low rank adapter) adapter to improve training efficiency. The SLAM-AAC of the embodiment of the present application applies text enhancement technology to increase the annotation diversity of audio clips in Clotho, thereby improving the versatility of the model. In addition, we also propose CLAP-Refine, a plug-and-play decoding strategy that strengthens annotation selection in post-processing. This strategy uses multiple beam search outputs as candidates and selects the final annotation based on the highest similarity score with the input audio calculated by the CLAP model. Experiments show that SLAM-AAC outperforms existing AAC models on the AudioCaps and Clotho datasets, and the reduction study verifies the contribution of each component in improving the overall performance of the model.

[0065] Future work will explore other paraphrase enhancement methods such as LLM-based paraphrasing and study the impact of different CLAP-based models on the CLAP-Refine strategy, aiming to approach or even reach oracle-level decoding performance.

[0066] In some other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, the computer executable instructions can execute the automatic audio tagging method in any of the above method embodiments, and is used in an automatic audio tagging system; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Acquire an audio signal, and extract the audio representation from the audio signal based on the EAT model; Aligning the audio representation and the text embedding corresponding to the audio representation based on the linear projection layer and downsampling to obtain a first text embedding; generating a text description based on the aligned and downsampled audio representation and the text embedding at least based on the large language model; Based on the strategy calculation module, the similarity between each of the text descriptions and the audio signal is calculated and the most matching text description is selected as the output of the automatic audio tagging system.

[0067] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the automatic audio tagging method and system, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the automatic audio tagging method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0068] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned automatic audio tagging methods.

[0069] Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the device includes: one or more processors 710 and a memory 720, Figure 7 A processor 710 is used as an example. The device of the automatic audio tagging method and system may also include: an input device 730 and an output device 740. The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The example of connecting via a bus is taken. The memory 720 is the above-mentioned non-volatile computer-readable storage medium. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the automatic audio annotation method of the above-mentioned method embodiment is implemented. The input device 730 can receive input digital or character information, and generate key signal input related to user settings and function control of the automatic audio annotation device. The output device 740 may include display devices such as display screens.

[0070] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0071] As an implementation mode, the electronic device is applied to an automatic audio tagging system, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: Acquire an audio signal, and extract the audio representation from the audio signal based on the EAT model; Aligning the audio representation and the text embedding corresponding to the audio representation based on the linear projection layer and downsampling to obtain a first text embedding; generating a text description based on the aligned and downsampled audio representation and the text embedding at least based on the large language model; Based on the strategy calculation module, the similarity between each of the text descriptions and the audio signal is calculated and the most matching text description is selected as the output of the automatic audio tagging system.

[0072] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0073] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0074] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0075] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0076] (5) Other electronic devices with data interaction functions.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0078] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automatic audio annotation system, comprising: EAT model, used to extract audio representation from audio signals; A linear projection layer, configured to align the audio representation and a text embedding corresponding to the audio representation and downsample the audio representation to obtain a first text embedding; a large language model for generating a text description based at least on the aligned and downsampled audio representations and the text embeddings; The strategy calculation module is used to calculate the similarity between each of the text descriptions and the audio signal and select the most matching text description as the output of the automatic audio tagging system.

2. The system according to claim 1, wherein: The large language model also includes a large language model tokenizer, which is used to process the text prompt and the real subtitle to obtain a second text embedding and the length of the real subtitle.

3. The system according to claim 2, wherein: The automatic audio tagging system also includes a low-rank adapter for fine-tuning the large language model.

4. The system according to claim 3, wherein: Only the linear projection layer and the low-rank adapter are trainable, the rest of the system remains frozen.

5. The system according to claim 4, wherein: The strategy calculation module is CLAP-Refine, which projects the text description and the audio signal into a shared space through comparative pre-training to calculate the text similarity, evaluates the similarity between the candidate annotations generated by multi-beam search decoding and the audio signal, and finally selects the annotation with the highest score as the output of the automatic audio annotation system.

6. An automatic audio annotation method, used in the automatic audio annotation system according to any one of claims 1 to 5, comprising: Acquire an audio signal, and extract the audio representation from the audio signal based on the EAT model; Aligning the audio representation and the text embedding corresponding to the audio representation based on the linear projection layer and downsampling to obtain a first text embedding; generating a text description based on the aligned and downsampled audio representation and the text embedding at least based on the large language model; Based on the strategy calculation module, the similarity between each of the text descriptions and the audio signal is calculated and the most matching text description is selected as the output of the automatic audio tagging system.

7. The method according to claim 6, wherein: Generating a text description based on the aligned and downsampled audio representation and the text embedding at least based on the large language model comprises: During training, the first text embedding, the second text embedding and the length of the real subtitle are concatenated using the large language model fine-tuned by the low-rank adapter to obtain a first joint embedding, and the large language model is trained based on the joint embedding and cross entropy loss; During inference, the first text embedding and the second text embedding are concatenated to obtain a second joint embedding, which is used to input the large language model to generate a text description.

8. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in claim 6 or 7.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 6 or 7 are implemented.

Images