Three-stage label re-labeling method and device based on universal audio language multi-modal large model

Abstract

The invention relates to a three-stage label re-labeling method and device based on a universal audio language multi-mode large model. The method comprises the following steps: firstly, extracting a model through audio features, and generating three-section structured description containing overall description, voice and music features from processed audio; embedding the audio content into a semantic enhanced prompt template, inputting a tag prediction language model, and generating a prediction tag matched with the audio content by combining a tag quantity control mechanism of corpus statistical characteristics; then, a multi-modal label alignment model is used, the prediction label is mapped to an original label category of the data set through three-layer matching, and candidate labels are obtained; and finally, the similarity between the candidate tags and the audio is calculated through cosine similarity, and the tags of which the scores are not lower than a threshold value are reserved as results. The method does not depend on the correctness of the original label, even if the original label is poor in quality and incomplete, a high-quality re-labeling result can still be output, and the inherent defects of an existing method are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a three-stage label re-labeling method and device based on a general audio language multi-modal large model. BACKGROUND

[0002] AudioSet is a widely used benchmark in the audio research community and has significantly advanced various audio-related tasks. However, the persistent issues of label accuracy and completeness remain a key bottleneck limiting the performance of downstream applications.

[0003] Audio signal processing is a highly interdisciplinary field that includes a range of core tasks such as speech recognition, sound event detection, medical sound analysis, and emotion recognition, all of which aim to extract discriminative representations from raw audio signals. In recent years, the rapid development of deep learning has made it the dominant paradigm in this field. AudioSet, a large audio dataset released by Google, has significantly advanced the development of audio understanding systems. It includes approximately 2 million 10-second segments (over 2000 hours) spanning 527 sound event classes, including human voices, source-obscured sounds, animal sounds, thing sounds, music, natural sounds, and background sounds.

[0004] To this end, several hierarchical optimization strategies have been proposed in the field. Tomasz et al. introduced a two-stage method - parent expansion labels, parent expansion, and child masking labels - to address semantic redundancy and hierarchical conflicts, thereby improving label balance. Building on this, Ludovic et al. proposed hierarchical label propagation, a systematic approach to inferring missing parent-child labels using the AudioSet ontology. While these methods have achieved certain improvements, they are inherently dependent on the correctness of the original labels and their performance is significantly limited by label noise and incompleteness. SUMMARY

[0005] Therefore, it is necessary to provide a three-stage label re-labeling method and device based on a general audio language multi-modal large model that can accurately re-label without relying on original labels.

[0006] A three-stage label re-labeling method based on a general audio language multi-modal large model, the method comprising: obtaining an audio dataset, the audio dataset including a plurality of audio data with original labels, and preprocessing all audio data in the audio dataset; using an audio feature extraction model to generate a three-part structured description including audio overall description, speech features, and music features based on the processed audio data; inputting the three-part structured description into a label prediction language model, in which the three-part structured description is embedded into a semantic-enhanced prompt template customized for label generation, and a label quantity control mechanism preset based on corpus statistical characteristics is used to generate audio prediction labels matched with the audio content; using a multi-modal label alignment model, mapping the audio prediction labels to original label categories of the audio data set through a three-layer matching mechanism, and taking the aligned audio prediction labels as candidate labels; calculating the similarity score between the candidate labels and the corresponding audio data through cosine similarity, and taking the aligned audio prediction labels with a similarity score higher than a preset threshold as the final audio re-labeling results.

[0007] In one embodiment, the audio data set is an AudioSet data set, and the original label is an AudioSet 527 class standard label.

[0008] In one embodiment, the audio feature extraction model uses a Qwen-Audio large model.

[0009] In one embodiment, the label prediction language model uses a Mistral large model.

[0010] In one embodiment, the multi-modal label alignment model uses a DeepSeek-R1 large model.

[0011] In one embodiment, the preprocessing of all audio data in the audio data set includes: using an audio processing library to convert the audio data into floating-point waveform data, wherein if it is multi-channel, the channel data is weighted averaged or a single channel is selected to obtain single-channel waveform; resampling the waveform data according to a preset sampling rate, performing short-time Fourier transform on the resampled waveform data to obtain time-frequency spectrum representation, filtering through a predefined Mel filter bank to generate Mel spectrum features; performing amplitude compression and normalization processing on the Mel spectrum features to obtain processed audio data.

[0012] In one embodiment, when generating a three-part structured description from the processed audio data using a Qwen-Audio large model, a multi-round prompting strategy driven by an AudioSet hierarchical classification topology is used, and three rounds of prompting are performed through a "model.chat" interface, including:

[0013] inputting a first round of prompt templates into the Qwen-Audio large model, and outputting the overall audio description to complete the first round of description; If the first round of description contains human voice, a second round of prompt templates is inputted into the Qwen-Audio large model, and the vocal feature is outputted; if there is no human voice, this round is skipped, and the vocal information is marked as "none";

[0014] If the first round of description contains music, a third round of prompt templates is inputted into the Qwen-Audio large model, and the music feature is outputted; if there is no music, this round is skipped, and the music content is marked as "none".

[0015] In one embodiment, the DeepSeek-R1 large model is used to map the audio prediction label to the original label category of the audio data set through a three-layer matching mechanism, including: accurately matching the audio prediction label with the AudioSet527 class label and synonym matching to obtain a plurality of matching labels; performing similarity calculation based on the CLAP model on the plurality of matching labels and the audio file respectively; discarding labels with similarity calculation results below a threshold value, and taking labels with similarity calculation results above the threshold value as the mapping result of the audio prediction label.

[0016] In one embodiment, the similarity score between the candidate label and the corresponding audio data is calculated by cosine similarity, using the following formula:

[0017] In the above formula, and represent the audio data and the corresponding candidate label respectively, represent their similarity scores, and and are the audio encoder and the text encoder respectively.

[0018] The application also provides a three-stage label re-labeling device based on a general audio language multi-modal large model, which comprises: an original audio data acquisition module for acquiring an audio data set, the audio data set comprising a plurality of audio data with original labels, and pre-processing all audio data in the audio data set; an audio feature extraction module for generating a three-section structured description containing audio overall description, speech feature and music feature from the processed audio data by using an audio feature extraction model; an audio label prediction module configured to input the three-section structured description into a label prediction language model, embed the three-section structured description into a semantic enhanced prompt template customized for label generation in the label prediction language model, and generate audio prediction labels matched with the audio content through a label quantity control mechanism preset based on corpus statistical characteristics; an audio prediction label alignment module configured to map the audio prediction labels to original label categories of the audio data set through a three-layer matching mechanism by using a multi-modal label alignment model, and take the aligned audio prediction labels as candidate labels; an audio relabeling result obtaining module configured to calculate similarity scores between the candidate labels and corresponding audio data through cosine similarity, and take the aligned audio prediction labels with similarity scores not lower than a preset threshold as final audio relabeling results.

[0019] A computer device including a memory and a processor, the memory storing a computer program, and the processor implementing the specific steps of the three-stage label relabeling method based on a general audio language multi-modal large model when executing the computer program.

[0020] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the specific steps of the three-stage label relabeling method based on a general audio language multi-modal large model.

[0021] The three-stage label relabeling method and device based on a general audio language multi-modal large model, by using an audio feature extraction model, generating a three-section structured description including audio overall description, speech features and music features according to processed audio data, inputting the three-section structured description into a label prediction language model, embedding the three-section structured description into a semantic enhanced prompt template customized for label generation in the label prediction language model, generating audio prediction labels matched with the audio content through a label quantity control mechanism preset based on corpus statistical characteristics, using a multi-modal label alignment model to map the audio prediction labels to original label categories of the audio data set through a three-layer matching mechanism, taking the aligned audio prediction labels as candidate labels, and finally calculating similarity scores between the candidate labels and corresponding audio data through cosine similarity, taking the aligned audio prediction labels with similarity scores not lower than a preset threshold as final audio relabeling results. This method completely breaks the dependence on the correctness of the original labels, and can still output high-quality relabeling results in the scene where the original labels are poor and incomplete, solving the inherent defects of existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1This is a flowchart illustrating a three-stage tag re-labeling method based on a large multimodal model of a general audio language in one embodiment. Figure 2 This is a schematic diagram of a three-stage label re-annotation framework in one embodiment; Figure 3 This is a structural block diagram of a three-stage tag re-annotation device based on a large multimodal model of a general audio language in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] Existing audio dataset label optimization methods still inherently rely on the correctness of the original labels, and their performance is significantly limited by label noise and incompleteness. In this embodiment, as... Figure 1 As shown, a three-stage label re-annotation method based on a general audio language multimodal large model is provided, which specifically includes the following steps: Step S100: Obtain the audio dataset, which includes multiple audio data with original labels, and preprocess all the audio data in the audio dataset.

[0025] Step S110: Using an audio feature extraction model, a three-part structured description containing an overall audio description, speech features, and music features is generated based on the processed audio data.

[0026] Step S120: Input the three-segment structured description into the tag prediction language model. In the tag prediction language model, embed the three-segment structured description into a semantically enhanced prompt template customized for tag generation. Then, through a tag quantity control mechanism based on corpus statistical features, generate audio prediction tags that match the audio content.

[0027] Step S130: Using a multimodal label alignment model, the audio prediction labels are mapped to the original label categories of the audio dataset through a three-layer matching mechanism, and the aligned audio prediction labels are used as candidate labels.

[0028] Step S140: Calculate the similarity score between the candidate label and the corresponding audio data using cosine similarity, and use the aligned audio prediction label with a similarity score not lower than a preset threshold as the final audio re-annotation result.

[0029] This application proposes an automated label re-annotation framework that leverages a state-of-the-art audio language foundation model and incorporates a cue chain strategy. Specifically, the audio feature extraction model first extracts rich semantic information from each audio segment by generating a structured three-part text description containing the main content, the presence of human voices, and the presence of music. Then, this text summary is processed by a label prediction language model, which utilizes structured cues to generate semantically accurate and context-based label predictions. Finally, the label prediction language model aligns the predicted labels with the originally defined label categories in the audio dataset through exact matching and synonym recognition, thereby achieving consistent label mapping and semantic normalization.

[0030] In step S100, if the audio dataset is the AudioSet dataset, then the original labels are the standard labels of the AudioSet527 class. In fact, this method can optimize labels for any audio dataset; however, this method uses the AudioSet dataset as an example for illustration.

[0031] This method addresses the fact that in deep learning, especially in large-scale model applications (LLM, VLM, ALM), a single prompt often cannot cover all the needs of complex tasks, and large models may sometimes exhibit thought leaps or erroneous reasoning. A cue chain is a structured cue strategy that decomposes complex tasks into consecutive sub-tasks, thereby enabling intermediate reasoning, stable output, and context-aware decision-making, further improving accuracy, controllability, and interpretability. Inspired by these advantages, a three-stage label re-annotation framework consistent with the cue chain paradigm is designed.

[0032] Specifically, the so-called three-stage iterative process involves breaking down the tag re-labeling task into three sub-tasks. In each sub-task, a general audio language model is used to generate the target content for that stage. Furthermore, the prompts for the next general audio language model are dynamically generated based on the output of the previous task, thereby more effectively utilizing the generated content from each stage and improving the accuracy, completeness, and fine-grainedness of tag re-labeling.

[0033] In this embodiment, the audio feature extraction model adopts the Qwen-Audio large model.

[0034] In this embodiment, the label prediction language model adopts the Mistral large model.

[0035] In this embodiment, the multimodal label alignment model adopts the DeepSeek-R1 large model.

[0036] It should be noted here that the audio feature extraction model, label prediction language model, and multimodal label alignment model mentioned above can also use other large models with corresponding functions. However, the Qwen-Audio large model, Mistral large model, and DeepSeek-R1 large model mentioned above are the optimal combinations in each stage after being verified by different model combinations. Therefore, these three relatively new and high-performance models were used to complete the corresponding steps.

[0037] Preferably, the audio feature extraction model, label prediction language model, and multimodal label alignment model employ the Qwen-Audio large model, the Mistral large model, and DeepSeek-R1, respectively. Qwen-Audio is a multimodal large model for general audio-language understanding. This model uses a multi-level pre-training paradigm, first learning cross-modal representations using large-scale audio-text pairs, and then enhancing the model's task adaptability through instruction fine-tuning. Qwen-Audio supports various downstream tasks such as speech recognition, audio question answering, and event detection, achieving leading performance on multiple public benchmark datasets, demonstrating its superiority in multimodal reasoning and audio semantic understanding. Mistral is a class of open-source large-scale language models developed by Mistral AI, designed to provide efficient and scalable natural language processing capabilities. This model, based on an improved decoder architecture, combines grouped query attention and sliding window attention, maintaining competitive performance while reducing computational and memory overhead. Experiments show that Mistral achieves or surpasses the performance of similarly sized closed-source models on various natural language reasoning and question-answering benchmarks, highlighting its potential in efficient reasoning and large-scale application scenarios. DeepSeek-R1 is an open-source large-scale language model designed for complex reasoning tasks, employing a policy-optimized training paradigm to enhance multi-step reasoning capabilities. Specifically, DeepSeek-R1 combines supervised fine-tuning, reinforcement learning based on human feedback, and multi-round self-game optimization, enabling the model to explicitly generate reasoning chains and maintain high consistency. Experiments demonstrate that this model significantly outperforms baseline models of similar scale on tasks such as mathematical reasoning, logical reasoning, question answering, and code generation, providing a new paradigm for researching large-scale reasoning-enhanced models.

[0038] The following sections will use a three-stage label relabeling framework built with the Qwen-Audio large model, the Mistral large model, and DeepSeek-R1 to perform label relabeling on the AudioSet dataset, in order to better illustrate this method.

[0039] like Figure 2As shown, starting with Qwen-Audio, it performs semantic decomposition by generating structured text summaries from the raw audio, covering core aspects such as main content, vocal presence, and musical elements. Next, Mistral utilizes these structured summaries from cue templates to generate semantically accurate and context-relevant candidate labels. Finally, DeepSeek-R1 uses precise synonym-aware matching to align the predicted labels with 527 AudioSet categories, ensuring consistency and classification consistency.

[0040] Figure 2 The left side of the diagram presents the overall concept of a multi-turn cue chain, where the output of each model is integrated into a fixed instruction template to construct the next cue, progressively guiding the label generation process. The right side details the implementation process, sequentially integrating Qwen-Audio for fine-grained semantic parsing, Mistral for initial label prediction, DeepSeek-R1 for classification alignment, and CLAP for semantic similarity filtering, ultimately generating a high-quality relabeled label set, AudioSet-R.

[0041] In this embodiment, before inputting all audio data in the audio dataset into the three-stage label re-annotation framework, preprocessing is performed, including converting the audio data into floating-point waveform data using an audio processing library. If it is multi-channel, the data of each channel is weighted and averaged or a single channel is selected to obtain a single-channel waveform. The waveform data is resampled according to a preset sampling rate. The resampled waveform data is subjected to short-time Fourier transform to obtain a time-frequency spectrum representation. Mel spectrum features are generated by filtering through a predefined Mel filter bank. The amplitude compression and normalization processing of the Mel spectrum features are performed to obtain the processed audio data.

[0042] Specifically, firstly, audio data is read using an audio processing library (such as torchaudio) and converted into floating-point waveform data. When the audio data is multi-channel, a weighted average of the data from each channel can be performed, or a single channel can be selected to obtain a single-channel waveform. Next, the waveform data is resampled to adjust the sampling rate to a predetermined rate (e.g., 16kHz) to ensure consistency in subsequent feature extraction. Then, a short-time Fourier transform is performed on the resampled waveform data to obtain a time-spectrum representation, and this time-spectrum is filtered using a predefined Mel filter bank to generate Mel spectral features. Finally, amplitude compression and normalization are performed on the Mel spectral features to obtain a normalized feature tensor for use in subsequent models.

[0043] In one embodiment, the Qwen-Audio model can also be used to preprocess the audio data.

[0044] In this embodiment, accurate audio tag prediction relies on obtaining fine-grained, high-quality semantic descriptions to capture the core content of each audio segment. These descriptions serve as the foundational input for downstream tag prediction, enhancing semantic interpretability and consistency. In step S110, based on the effectiveness of Qwen-Audio in audio understanding and cross-modal generation, Qwen-Audio is adopted as the core component of the semantic extraction module in this embodiment.

[0045] In this embodiment, when generating a three-segment structured description based on the processed audio data using the Qwen-Audio large model, a multi-round prompting strategy driven by the AudioSet hierarchical classification topology is adopted. The three rounds of prompting are executed through the "model.chat" interface, including: inputting a first-round prompting template into the Qwen-Audio large model and outputting the overall audio description to complete the first round of description; if the first round of description contains human voice, then inputting a second-round prompting template into the Qwen-Audio large model and outputting the vocal features; if there is no human voice, then skipping this round and marking the vocal information as "none". If the first round of description contains music, then inputting a third-round prompting template into the Qwen-Audio large model and outputting the music features; if there is no music, then skipping this round and marking the music content as "none".

[0046] Specifically, "model.chat" was used, and a multi-turn cueing strategy was designed to extract structured semantic representations from the raw audio input. The cue templates were carefully designed to standardize the semantic scope and format of each response. This three-turn cueing mechanism was driven by the hierarchical classification topology of the AudioSet.

[0047] Furthermore, the first round of prompting generates a concise overall audio description (≤50 words). The second round (if there is voice) requires identifying relevant features of the speaker, such as emotional state, gender, and speech. The third round (if there is music) instructs the model to specify the genre and instrument involved. The output of all rounds is aggregated into a standardized, three-part text summary, including: (1) main content, (2) vocal information, and (3) musical content. This strategy enables Qwen-Audio to reliably extract semantically rich and structurally consistent descriptions from complex auditory scenes, thus providing high-quality input to the label prediction module.

[0048] Furthermore, when examining the limitations of direct audio tagging, relying solely on high-level semantic descriptions without explicit control often leads to over-tagting or inconsistent tag quality. Additionally, analysis of the AudioSet corpus reveals a relatively low average number of effective tags per segment (2.39 during training and 2.55 during evaluation), suggesting the need for constraints and contextual precision in tag generation.

[0049] When examining the limitations of direct audio tagging, it was observed that relying solely on high-level semantic descriptions without explicit control often leads to over-tagging or inconsistent tag quality. Furthermore, analysis of the AudioSet corpus revealed a relatively low average number of effective tags per segment (2.39 during training and 2.55 during evaluation), suggesting the need for constraints and contextual precision in tag generation.

[0050] In step S120, a context-aware tag generation strategy was designed based on the Mistral 7B large-scale language model developed by Mistral AI. This strategy takes a structured three-part description provided by Qwen-Audio as input and embeds it into a semantically enhanced prompt template specifically customized for tag generation. By analyzing corpus statistical features, a tag quantity control mechanism was introduced into the prompt design, enabling the model to maintain a balance between semantic coverage and output conciseness. Through this strategy, the Mistral model achieved logically consistent and semantically robust guided inference prediction. This model combines detailed contextual input with guiding constraints, effectively improving tag validity control while ensuring computational efficiency—fully demonstrating its practical value in scalable automated audio re-annotation tasks.

[0051] However, analysis shows that many of the labels predicted in Stage 2, while semantically valid, do not conform to the standardized AudioSet ontology. These free-form labels differ in expression, granularity, and wording, leading to semantic inconsistencies and hindering downstream integration. To address this issue, a label alignment strategy is designed in this method, using DeepSeek R1, a multimodal LLM. The goal is to map non-standard tags to 527 AudioSet categories through a three-layer matching mechanism: (1) exact matching (direct correspondence), (2) fuzzy matching (allowing lexical variations), and (3) synonym matching using semantic resources.

[0052] In step S130, a multimodal label alignment model is used to map audio predicted labels to the original label categories of the audio dataset through a three-layer matching mechanism. This includes: performing exact matching and synonym matching between the audio predicted labels and AudioSet527 class labels to obtain multiple matching labels. Then, similarity calculations based on the CLAP model are performed between each of the multiple matching labels and the audio file. Labels with similarity calculation results below a threshold are discarded, and labels with similarity calculation results above the threshold are used as the mapping results for the audio predicted labels.

[0053] Specifically, the audio prediction labels that are aligned after the above-mentioned precise matching, fuzzy matching, and synonym matcher are used as candidate labels, while other labels that are not aligned are deleted.

[0054] While the alignment strategy ensures structural consistency, some mapping tags lack semantic relevance to the corresponding audio. This indicates a need for post-alignment filtering based on content tag consistency. To address this issue, in step S140, semantic alignment is evaluated based on the candidate tags and their corresponding original audio data.

[0055] In this embodiment, CLAP is used to evaluate semantic alignment. For each candidate tag, its CLAP similarity score with the corresponding audio data is calculated, and only those exceeding a predefined threshold are retained. This filtering step improves tag reliability by removing predictions with low confidence or semantic inconsistency.

[0056] Specifically, the similarity score between candidate tags and their corresponding audio data is calculated using cosine similarity, using the following formula:

[0057] In the above formula, and These represent the audio data and the corresponding candidate tags, respectively. This represents their similarity score, and and These are the audio encoder and the text encoder, respectively.

[0058] In this embodiment, the two strategies in steps S130 and S140—namely, ontology alignment using DeepSeek-R1 and semantic filtering using CLAP—effectively address the issues of inconsistent labels and weak semantic foundation. Combined with the earlier stages (using Qwen-Audio for description and Mistral for prediction), these components together constitute a robust and modular re-annotation process.

[0059] The aforementioned three-stage label re-annotation method based on a general audio language multimodal large model proposes an automated label re-annotation framework. This framework leverages a state-of-the-art audio language foundational model and incorporates a cue chain strategy. Specifically, the Qwen-Audio multimodal model first extracts rich semantic information from each audio segment by generating a structured three-part text description containing the main content, vocal presence, and musical presence. This text summary is then processed by the Mistral language model, which utilizes structured cues to generate semantically accurate and context-based label predictions. Finally, the DeepSeek R1 model aligns the predicted labels with 527 predefined AudioSet categories through exact matching and synonym recognition, achieving consistent label mapping and semantic normalization. This multi-model chain framework provides a scalable and automated solution for audio annotation. It enhances the completeness and semantic accuracy of labels while effectively addressing the subjectivity, noise, and inconsistencies often encountered in manual labeling or single-pass cue methods.

[0060] It should be understood that, although Figure 1 Perform the steps in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps can be performed. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0061] In one embodiment, such as Figure 3 As shown, a three-stage label re-labeling device based on a general audio language multimodal large model is provided, including: a raw audio data acquisition module 200, an audio feature extraction module 210, an audio label prediction module 220, an audio prediction label alignment module 230, and an audio re-labeling result acquisition module 240, wherein: The raw audio data acquisition module 200 is used to acquire an audio dataset, which includes multiple audio data with original labels, and to preprocess all the audio data in the audio dataset. The audio feature extraction module 210 is used to generate a three-segment structured description containing an overall audio description, speech features, and music features based on the processed audio data using an audio feature extraction model. The audio tag prediction module 220 is used to input the three-segment structured description into the tag prediction language model. In the tag prediction language model, the three-segment structured description is embedded into a semantically enhanced prompt template customized for tag generation. Then, through a tag quantity control mechanism based on corpus statistical features, audio prediction tags matching the audio content are generated. The audio prediction label alignment module 230 is used to map the audio prediction label to the original label category of the audio dataset through a three-layer matching mechanism using a multimodal label alignment model, and to use the aligned audio prediction label as a candidate label. The audio re-annotation result acquisition module 240 is used to calculate the similarity score between the candidate label and the corresponding audio data through cosine similarity, and to take the aligned audio prediction label with the similarity score not lower than a preset threshold as the final audio re-annotation result.

[0062] Specific limitations regarding the three-stage tag re-annotation device based on a general audio language multimodal large model can be found in the limitations of the three-stage tag re-annotation method based on a general audio language multimodal large model mentioned above, and will not be repeated here. Each module in the aforementioned three-stage tag re-annotation device based on a general audio language multimodal large model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0063] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a three-stage tag re-annotation method based on a general audio language multimodal large model. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.

[0064] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Obtain an audio dataset, which includes multiple audio data with original labels, and preprocess all audio data in the audio dataset; Using an audio feature extraction model, a three-part structured description containing an overall audio description, speech features, and music features is generated based on the processed audio data. The three-segment structured description is input into the tag prediction language model. In the tag prediction language model, the three-segment structured description is embedded into a semantically enhanced prompt template customized for tag generation. Then, through a tag quantity control mechanism based on corpus statistical features, audio prediction tags matching the audio content are generated. Using a multimodal label alignment model, the audio predicted labels are mapped to the original label categories of the audio dataset through a three-layer matching mechanism, and the aligned audio predicted labels are used as candidate labels. The similarity score between the candidate label and the corresponding audio data is calculated using cosine similarity. The aligned audio prediction label with a similarity score not lower than a preset threshold is taken as the final audio re-annotation result.

[0066] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain an audio dataset, which includes multiple audio data with original labels, and preprocess all audio data in the audio dataset; Using an audio feature extraction model, a three-part structured description containing an overall audio description, speech features, and music features is generated based on the processed audio data. The three-segment structured description is input into the tag prediction language model. In the tag prediction language model, the three-segment structured description is embedded into a semantically enhanced prompt template customized for tag generation. Then, through a tag quantity control mechanism based on corpus statistical features, audio prediction tags matching the audio content are generated. Using a multimodal label alignment model, the audio predicted labels are mapped to the original label categories of the audio dataset through a three-layer matching mechanism, and the aligned audio predicted labels are used as candidate labels. The similarity score between the candidate label and the corresponding audio data is calculated using cosine similarity. The aligned audio prediction label with a similarity score not lower than a preset threshold is taken as the final audio re-annotation result.

[0067] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A three-stage label re-labeling method based on a general audio language multi-modal large model, characterized in that, The method comprises: obtaining an audio data set comprising a plurality of audio data with original labels, preprocessing all audio data in the audio data set; using an audio feature extraction model to generate a three-part structured description comprising audio overall description, speech features and music features from the preprocessed audio data; inputting the three-part structured description into a label prediction language model, embedding the three-part structured description into a semantic enhanced prompt template customized for label generation in the label prediction language model, and generating audio predicted labels matching the audio content through a label quantity control mechanism based on corpus statistical characteristics; using a multi-modal label alignment model to map the audio predicted labels to the original label categories of the audio data set through a three-layer matching mechanism, and taking the aligned audio predicted labels as candidate labels; calculating the similarity score between the candidate labels and the corresponding audio data through cosine similarity, and taking the aligned audio predicted labels with a similarity score not lower than a preset threshold as the final audio re-labeling result.

2. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 1, characterized in that, The audio data set is an AudioSet data set, and the original label is an AudioSet 527 class standard label.

3. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 2, characterized in that, The audio feature extraction model uses a Qwen-Audio large model.

4. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 2 or 3, characterized in that, The label prediction language model uses a Mistral large model.

5. The three-stage label re-labeling method based on the general audio language multimodal large model according to any one of claims 2-4, characterized in that, The multi-modal label alignment model uses a DeepSeek-R1 large model.

6. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 5, characterized in that, The preprocessing of all audio data in the audio data set comprises: using an audio processing library to convert the audio data into floating point waveform data, wherein if it is multi-channel, the channel data is weighted averaged or single channel is selected to obtain single channel waveform; resampling the waveform data according to a preset sampling rate, performing short-time Fourier transform on the resampled waveform data to obtain time-frequency spectrum representation, filtering through a predefined Mel filter bank to generate Mel spectrum features; performing amplitude compression and normalization processing on the Mel spectrum features to obtain preprocessed audio data.

7. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 6, characterized in that, When generating a three-part structured description from the preprocessed audio data using a Qwen-Audio large model, a multi-round prompting strategy driven by an AudioSet hierarchical classification topology is adopted, and three rounds of prompting are performed through a "model.chat" interface, including: inputting a first round of prompt template into the Qwen-Audio large model to output the audio overall description to complete the first round of description; if there is human voice in the first round of description, input a second round of prompt template into the Qwen-Audio large model to output the vocal features; if there is no human voice, skip this round and mark the vocal information as "none"; if there is music in the first round of description, input a third round of prompt template into the Qwen-Audio large model to output the music features; if there is no music, skip this round and mark the music content as "none".

8. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 7, characterized in that, Using a DeepSeek-R1 large model, the audio predicted labels are mapped to the original label categories of the audio data set through a three-layer matching mechanism, including: The audio prediction label is accurately matched with the AudioSet 527 category label and synonym matching is performed to obtain a plurality of matched labels; The plurality of matched labels are respectively subjected to similarity calculation based on the CLAP model with respect to the audio file; Labels with similarity calculation results lower than a threshold value are discarded, and labels with similarity calculation results higher than the threshold value are taken as mapping results of the audio prediction label.

9. The three-stage label re-labeling method based on a general audio language multimodal large model according to claim 7, characterized in that, A similarity score between the candidate label and corresponding audio data is calculated by cosine similarity, and the following formula is used: In the above formulae, and represent audio data and corresponding candidate labels, respectively, represent their similarity scores, while and are an audio encoder and a text encoder, respectively.

10. A three-stage label re-labeling device based on a general audio language multi-modal large model, characterized in that, The device comprises: An original audio data acquisition module configured to acquire an audio data set, wherein the audio data set comprises a plurality of audio data with original labels, and all audio data in the audio data set is preprocessed; An audio feature extraction module configured to generate a three-part structured description comprising audio overall description, speech features and music features from the processed audio data by using an audio feature extraction model; An audio label prediction module configured to input the three-part structured description into a label prediction language model, embed the three-part structured description into a semantic enhanced prompt template customized for label generation in the label prediction language model, and generate an audio prediction label matched with the audio content by a label quantity control mechanism based on corpus statistical characteristics; An audio prediction label alignment module configured to map the audio prediction label to the original label category of the audio data set by a three-layer matching mechanism by using a multi-modal label alignment model, and take the aligned audio prediction label as a candidate label; An audio re-labeling result obtaining module configured to calculate a similarity score between the candidate label and corresponding audio data by cosine similarity, and take the aligned audio prediction label with a similarity score not lower than a preset threshold value as a final audio re-labeling result.

Images