An audio text correlation evaluation method based on a multi-expert model, an audio retrieval system, a storage medium and a program product

Abstract

The application belongs to the technical field of audio processing, and a plurality of pre-trained audio text expert models are introduced to construct a multi-expert audio text semantic representation. Meanwhile, the consistency relationship between audio and text at the overall semantic level and the semantic inconsistency between audio and text that is perceptually significant to humans are modeled through a semantic alignment branch and a semantic mismatch branch respectively, and a multi-branch correlation score fusion mechanism is used to output an audio text correlation prediction result that is highly consistent with human subjective evaluation. This method can simultaneously consider semantic consistency and perceptual differences, effectively achieving more accurate and more human subjective evaluation standard-compliant audio text correlation evaluation. Meanwhile, based on the above method, the application constructs an audio retrieval system that evaluates and sorts the correlation between a text query and audio samples in an audio library to realize an audio retrieval function oriented to natural language description.

Description

Technical Field

[0001] This invention belongs to the field of audio processing technology, specifically relating to an audio text relevance evaluation method based on a multi-expert model, an audio retrieval system, a storage medium, and a program product. Background Technology

[0002] The goal of audio-text relevance assessment is to automatically evaluate the semantic relevance between a given audio signal and its text description. This task has wide applications in multimodal learning and cross-modal retrieval. However, due to the significant differences between the semantic representations of audio and text, the dynamic changes in information within audio signals, and the diverse ways in which text describes audio, extracting text and audio features and effectively matching them remains a technical challenge.

[0003] Existing research has made some progress in evaluating audio-text relevance. Many methods rely primarily on the semantic similarity between audio and text representations, using objective metrics such as CLAPScore and FAD to measure the semantic relevance of audio and text. For example, deep learning-based models typically learn the embedding spaces of audio and text to calculate their similarity and then evaluate relevance using objective metrics. However, while these methods and metrics improve the effectiveness of audio-text matching to some extent, they only focus on the consistency of semantic features between audio and text, ignoring perceptual differences and subtle mismatches, and often have low correlation with human subjective evaluation. For instance, in some cases, audio and text may be highly correlated in the semantic feature space, but due to subtle differences in human perception, semantic mismatches still exist, which significantly impact human judgment.

[0004] To meet the needs of real-world scenarios, recent research proposes developing an automated evaluation method that is highly correlated with human subjective assessment. This method considers both semantic consistency and perceptual differences between audio and text, thereby achieving a relevance score that closely mirrors human subjective judgment. This requires not only a system with strong semantic understanding capabilities to accurately capture the matching degree between audio and text, but also the ability to identify subtle differences and semantic mismatches that humans often focus on during evaluation. This results in more reliable and intuitive assessments that mimic human judgment in actual user interactions. Summary of the Invention

[0005] To address the issues of semantic alignment and mismatch with human perception in existing audio-text relevance assessment methods, this invention provides an audio-text relevance assessment method, audio retrieval system, storage medium, and program product based on a multi-expert model, achieving more accurate audio-text relevance assessment that better conforms to human subjective evaluation standards.

[0006] The objective of this invention is achieved as follows:

[0007] An audio-text relevance assessment method based on a multi-expert model includes the following steps:

[0008] The target audio signal and its corresponding text description are obtained. Multiple pre-trained Contrastive Language-Audio Pre-training (CLAP) expert models are used to extract features from the audio signal and the text description respectively, so as to obtain the audio embedding features and text embedding features corresponding to each expert model.

[0009] For each expert model, semantic alignment evaluation is performed based on the corresponding audio embedding features and text embedding features to obtain the first relevance score;

[0010] For each expert model, a semantic mismatch assessment is performed based on the corresponding audio embedding features and text embedding features to obtain a second relevance score. The semantic mismatch assessment is performed by constructing and analyzing the difference features between audio and text to show the modeling semantic inconsistency.

[0011] The first and second relevance scores of each expert model are combined to obtain the comprehensive score of the expert model.

[0012] The final audio-text relevance prediction score is obtained by combining the scores from multiple expert models.

[0013] Further, the first relevance score is an audio text relevance score based on semantic consistency, including the following steps:

[0014] Calculate the cosine similarity between the audio embedding features and the text embedding features output by each expert model;

[0015] The cosine similarity is linearly scaled and mapped to a preset relevance scoring range to obtain an audio text relevance score based on semantic consistency.

[0016] Furthermore, the audio text relevance score based on semantic consistency... ,

[0017]

[0018] in, This represents the cosine similarity between audio semantic features and text semantic features under the k-th expert model, where α is a scaling factor used to control the range of values ​​for the semantic alignment score.

[0019] Furthermore, the second relevance score is an audio-text relevance score based on semantic mismatch, which includes the following steps:

[0020] Based on the audio embedding features and text embedding features output by each expert model, a semantic mismatch feature is constructed to characterize the semantic relationship between the two. The semantic mismatch feature integrates the similarity information and the difference information between the two.

[0021] The semantic mismatch features are input into a multilayer perceptron network for nonlinear mapping to obtain higher-order mismatch features.

[0022] The higher-order mismatch features are mapped using a scoring head function to obtain an audio-text relevance score based on semantic mismatch.

[0023] Furthermore, the semantic mismatch feature ,

[0024]

[0025] in," " indicates a vector concatenation operation, where the first term is the cosine similarity between the text semantic features and the audio semantic features, and the second term is the difference vector between the text semantic features and the audio semantic features;

[0026] The audio-text relevance score based on semantic mismatch ,

[0027]

[0028] in, For the scoring header function, This is a high-order mismatch feature.

[0029] Furthermore, the semantic mismatch evaluation introduces an auxiliary classification task based on discrete relevance levels; the human subjective relevance scores are divided into multiple continuous non-overlapping discrete level categories; the higher-order mismatch features are input into the classifier to predict their true discrete level; a difference loss function for the semantic mismatch branch is constructed based on the difference between the predicted discrete level and the true discrete level, and the multilayer perceptron network and the classifier are jointly optimized.

[0030] The difference loss function ,

[0031]

[0032] in, This represents the classification loss function based on angular interval constraints. Represents the feature center constraint loss function. , These are the weighting coefficients. For true discrete levels, For the predicted discrete levels.

[0033] Furthermore, the final audio-text relevance prediction score ,

[0034]

[0035] in, Indicates the first The audio-text relevance score is obtained by fusing the first relevance score and the second relevance score from the expert models; K represents the total number of expert models.

[0036] To constrain Human subjective relevance score Consistency, constructing the prediction loss function for,

[0037] .

[0038] An audio retrieval system based on a multi-expert model, characterized in that it applies the audio-text relevance evaluation method based on the multi-expert model, and the system includes:

[0039] The text query module is used to receive natural language text queries input by the user;

[0040] The audio feature management module is used to store the semantic feature representations of multiple audio samples in the audio library. The semantic feature representations are audio embedding features pre-extracted by the multiple pre-trained contrastive language-audio pre-trained expert models.

[0041] The audio-text relevance evaluation module is used to perform an audio-text relevance evaluation method based on a multi-expert model based on the text query and the audio embedding features obtained from the audio feature management module, and to calculate the audio-text relevance prediction score between the text query and each audio sample.

[0042] The retrieval, sorting, and output module is used to sort the audio samples according to the relevance prediction score and output the sorted retrieval results.

[0043] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an audio-text relevance assessment method based on a multi-expert model.

[0044] A computer program product includes a computer program that, when executed by a processor, implements steps of an audio-text relevance assessment method based on a multi-expert model.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] To address the problem that existing audio-text relevance assessment methods mainly rely on the semantic similarity between audio and text representations, which fails to reflect significant semantic inconsistencies in human subjective perception, this invention designs a semantic alignment modeling branch and a semantic mismatch modeling branch. These branches explicitly model the semantic consistency and semantic mismatch of audio and text, respectively, enabling a multi-faceted characterization of the relationship between audio and text. This effectively captures audio-text pairs that appear similar at the semantic level but have significant differences in human perception, accurately capturing subtle perceptual differences and making the assessment results more consistent with the rules of human subjective judgment.

[0047] Meanwhile, this invention employs multiple pre-trained expert models to extract and represent semantic features from audio and text. Different expert models can capture audio and text features from different semantic levels and perceptual scales, effectively alleviating the limitations of a single model in semantic understanding and improving the robustness and generalization ability of audio-text relevance assessment. Furthermore, the audio encoder and text encoder are constructed in a modular manner, allowing them to be replaced with a newer CLAP model without changing the overall method structure, thus achieving model upgrades and expansion.

[0048] Furthermore, the method proposed in this invention has a clear structure and a high degree of modularity. It can be used as an independent audio-text relevance assessment method, or it can be further integrated to build an audio retrieval system based on this method, realizing audio relevance assessment and retrieval ranking functions based on text queries. Based on the audio retrieval system, audio signals highly relevant to specific semantic events can be retrieved from an audio database according to the user-input text query, such as audio containing abnormal events like violent behavior or traffic accidents. This provides technical support for application scenarios such as multimedia content review and urban security monitoring. The method described in this invention and the audio retrieval system it constructs can flexibly adapt to different types of audio-text assessment tasks and can be widely applied to practical application scenarios such as audio retrieval, audio generation quality assessment, and multimodal content understanding, possessing high engineering practical value and promising prospects for promotion. Attached Figure Description

[0049] Figure 1 This is the overall technical roadmap of the present invention;

[0050] Figure 2 This is a schematic diagram of the semantic alignment modeling branch in this invention;

[0051] Figure 3 This is a schematic diagram of the semantic alignment modeling branch;

[0052] Figure 4 A schematic diagram of an audio retrieval system based on the present invention is shown. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0054] This invention discloses an audio text relevance evaluation method, an audio retrieval system, a storage medium, and a program product based on a multi-expert model. The method selects the CLAP series model as the expert model, and the specific steps are as follows:

[0055] Step 1: Extraction of semantic features from multi-expert audio text;

[0056] This invention employs multiple pre-trained contrastive language-audio pre-trained models as independent experts to extract semantic embedding features from the same audio signal and text description, thus constructing a multi-expert audio-text semantic representation. Assume the expert set is... , of which An expert Including audio encoders With text encoder .

[0057] Step 1.1: Audio feature extraction;

[0058] First, the original audio signal After undergoing a Short-Time Fourier Transform (STFT), the data is converted into a spectrogram feature, which encompasses all frequency bands defined by the sampling frequency. This spectrogram feature is then filtered using a Mel filter bank to obtain the Mel spectrum. To amplify the regions of the spectrogram that are sensitive to human auditory perception, the Mel spectrum is logarithmically scaled to obtain the Log-Mel spectrum. These features are then input into the audio encoders of various expert models for semantic modeling. For the first... Each expert model outputs a corresponding audio semantic feature vector from its audio encoder. This process can be represented as follows:

[0059]

[0060] in, Indicates the first The audio encoder in an expert model This is the audio semantic embedding vector output by the expert model.

[0061] Step 1.2: Text feature extraction;

[0062] Original input text description After word segmentation or sub-word encoding, the text is input into the text encoder of each expert model for processing. For the first... An expert model, whose text encoder encodes the input text to obtain the corresponding text semantic feature vector, can be represented as follows:

[0063]

[0064] in, Indicates the first The audio encoder in an expert model This is the audio semantic embedding vector output by the expert model.

[0065] Step 2: Semantic alignment branch modeling;

[0066] To measure the degree of consistency between audio semantic features and text semantic features at the overall semantic level, this invention introduces a semantic pair modeling branch, as shown in the appendix. Figure 2 As shown. In the semantic alignment modeling branch, audio and text semantic features output by the same expert model are treated as vector representations located in the same semantic space, and audio semantic embedding vectors from various expert models are used. With text semantic embedding vectors As input, and using cosine similarity as a metric to evaluate the semantic alignment between the two, the calculation can be expressed as follows:

[0067]

[0068] in, Indicates the first The cosine similarity between audio semantic features and text semantic features under an expert model is used to reflect the degree of consistency between the two at the semantic level.

[0069] Since the range of values ​​for cosine similarity is However, audio text relevance assessment tasks typically require outputting positive score intervals with clear semantic meaning. Therefore, in this invention, the cosine similarity is linearly scaled to map it to a preset relevance score interval. The linear scaling relationship can be expressed as:

[0070]

[0071] in, This is a scaling factor used to control the range of values ​​for the semantic alignment score; in this invention, A value of 5 determines the semantic alignment score. fall into Within the scoring range.

[0072] Step 3: Semantic mismatch branch modeling;

[0073] Semantic mismatch branch modeling is used to explicitly characterize semantic inconsistencies between audio and text semantic features that are significant in human perception, thus compensating for the shortcomings of relying solely on semantic similarity measurements. (See attached image) Figure 3 As shown, in the semantic mismatch modeling branch, audio semantic embedding vectors from various expert models are used. With text semantic embedding vectors As input, the semantic mismatch feature is composed of similarity and difference information between audio and text semantic features. The similarity information reflects the overall semantic relationship between the audio and text, while the difference information characterizes the degree of deviation in their semantic expression. The semantic mismatch feature is represented as follows:

[0074]

[0075] in," "" indicates a vector concatenation operation. The first term is the cosine similarity between the text semantic features and the audio semantic features, which is used to provide overall semantic association information. The second term is the difference vector between the text semantic features and the audio semantic features, which is used to characterize the degree of mismatch between the two in semantic representation.

[0076] Subsequently, the aforementioned semantic mismatch features are input into a multilayer perceptron network for nonlinear mapping to learn the complex semantic mismatch relationships between audio and text. This process can be represented as:

[0077]

[0078] in, This represents a multilayer perceptron network used for feature transformation and high-order relation modeling of spliced ​​semantic mismatch features.

[0079] Based on this, through the scoring header function The output of the multilayer perceptron is mapped to obtain the audio-text relevance score based on semantic mismatch modeling, which is represented as:

[0080]

[0081] Meanwhile, to enhance the ability of semantic mismatch branches to distinguish different levels of relevance, this invention introduces an auxiliary classification task based on discrete relevance levels. Specifically, this involves using human subjective relevance rating intervals. The system is divided into multiple continuous, non-overlapping sub-intervals according to preset rules, forming multiple relevance level categories, including: ,common There are several categories. Let the subjective relevance score for primitive humans be... The corresponding discretization level label is denoted as:

[0082]

[0083] The output features of the multilayer perceptron network The data is input into a classifier to predict the relevance level, resulting in a predicted category. ,

[0084]

[0085] Then, based on the difference between the predicted category and the true discretized level label, a difference loss function is constructed for the semantic mismatch branch, which is expressed as:

[0086]

[0087] in, This represents the classification loss function based on angular interval constraints. Represents the feature center constraint loss function. , This refers to the weighting coefficient. In one specific implementation, The value is 1. The value is 0.01; preferably, The range of values ​​is , The range of values ​​is .

[0088] By introducing a difference loss function, the semantic mismatch branch can be guided to learn more discriminative semantic inconsistency features during training, enabling the model to more accurately distinguish different degrees of semantic mismatch in audio text, thereby improving the consistency between the overall audio text relevance assessment results and human subjective judgment.

[0089] Step 4: Fusion of multi-branch correlation scores;

[0090] In this invention, multi-branch relevance score fusion is used to integrate the evaluation results of the semantic alignment modeling branch and the semantic mismatch modeling branch to form the final audio-text relevance prediction score. To fully utilize the complementary information contained in the two types of results, multi-branch relevance score fusion adopts a mean fusion method, the calculation of which can be expressed as follows:

[0091]

[0092] in, Indicates the first The audio-text relevance score is obtained by fusing multiple expert models. Further, the fusion score from multiple expert models is analyzed. The final audio-text relevance prediction score is obtained by averaging the scores from multiple expert models. Its calculation method can be expressed as follows:

[0093]

[0094] To ensure consistency between the relevance prediction results output by the model and human subjective assessment, this invention is based on the final audio text relevance prediction score. Human subjective relevance score Construct the prediction loss function, which is expressed as:

[0095]

[0096] Through the aforementioned multi-branch, multi-expert relevance score fusion mechanism, this invention can fully integrate the information provided by different expert models and different semantic modeling branches, effectively improve the stability and accuracy of audio text relevance assessment results, and ensure that the final prediction score is highly consistent with the human subjective assessment results.

[0097] In this embodiment, the audio encoder and text encoder are implemented based on the Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training (MGA-CLAP) and the Masked Modeling Duo - Contrastive Language-Audio Pre-training (M2D-CLAP), respectively. Furthermore, the audio encoder and text encoder are constructed in a modular manner, allowing them to be replaced with newer models without altering the overall methodological structure, thus enabling model upgrades and expansion.

[0098] The audio retrieval system based on a multi-expert model described in this invention is constructed based on the aforementioned audio-text relevance evaluation method using a multi-expert model. This system retrieves audio samples from an audio database that semantically best match a user-input text query and outputs them in a sorted order according to their relevance. The audio retrieval system includes a text query module, an audio feature management module, an audio-text relevance evaluation module, and a retrieval sorting and output module.

[0099] During system operation, the text query module receives natural language text queries input by the user, preprocesses the queries, and then inputs them into the audio text relevance evaluation module. The audio feature management module stores the semantic feature representations of each audio sample in the audio library. These semantic features can be pre-calculated offline and stored using the aforementioned multi-expert audio text semantic feature extraction method.

[0100] The audio-text relevance assessment module, based on the multi-expert model-based audio-text relevance assessment method described in this invention, evaluates the relevance between the text query and each audio sample in the audio library. Specifically, the module utilizes multi-expert text semantic feature extraction, semantic alignment branch modeling, semantic mismatch branch modeling, and a multi-branch relevance score fusion mechanism to calculate the predicted audio-text relevance score between the text query and each audio sample. The retrieval sorting and result output module sorts the audio samples in the audio library according to the predicted audio-text relevance score and outputs the retrieval results in descending order of relevance. The system can prioritize presenting audio samples with higher relevance scores to the user, thereby realizing the audio retrieval function based on text queries.

[0101] Example 1:

[0102] To verify the effectiveness of the proposed method in audio-text relevance assessment, ablation experiments were conducted on the key components of the method. This embodiment compares and analyzes different feature modeling methods through ablation experiments. The experiments were conducted on the same public dataset and under the same training conditions. By progressively removing different modules and comparing relevance assessment metrics, the contribution of each module to the overall performance was analyzed. Evaluation metrics included Spearman Rank Correlation Coefficient (SRCC), Linear Correlation Coefficient (LCC), Kendall Rank Correlation Coefficient (KTAU), and Mean Squared Error (MSE). SRCC, LCC, and KTAU were used to measure the correlation between the predicted results and human subjective ratings, while MSE was used to measure the prediction error.

[0103] The specific settings are as follows:

[0104] (1) Method of the present invention: This means that the complete method of the present invention includes a multi-expert model, a semantic alignment modeling branch and a semantic mismatch modeling branch.

[0105] (2) Remove MGA-CLAP experts: This means that the MGA-CLAP expert model is removed in the method of the present invention, and only other expert models are retained for evaluation;

[0106] (3) Remove M2D-CLAP experts: This means that the M2D-CLAP expert model is removed in the method of the present invention, and only other expert models are retained for evaluation;

[0107] (4) Remove semantic alignment modeling branch: This means removing the semantic alignment modeling branch and retaining only the semantic mismatch modeling branch;

[0108] (5) Remove semantic mismatch modeling branch: This means removing the semantic mismatch modeling branch and keeping only the semantic alignment modeling branch.

[0109] The experimental results are shown in the table below:

[0110] Table 1: Ablation Experiment Results of the Method of the Present Invention

[0111]

[0112] The experimental results show that: (1) When the MGA-CLAP expert model is removed, all correlation indicators decrease significantly. For example, SRCC decreases from 0.6396 to 0.5670, indicating that the multi-expert model structure can provide complementary semantic representation information, which plays an important role in improving the accuracy of evaluation. (2) When the M2D-CLAP expert model is removed, all indicators also decrease, indicating that different CLAP expert models can characterize the relationship between audio and text from different semantic levels. Multi-expert fusion can improve the robustness and generalization ability of the model. (3) When the semantic alignment modeling branch is removed, SRCC decreases to 0.5431, LCC decreases to 0.5619, and MSE increases to 3.9161, indicating that the semantic alignment modeling branch plays an important role in capturing the overall semantic consistency between audio and text. (4) When the semantic mismatch modeling branch is removed, the model performance also drops significantly. For example, KTAU drops from 0.4635 to 0.4351, while MSE rises to 4.0568, indicating that the semantic mismatch modeling branch can effectively capture the fine-grained differences between audio and text at the human perception level.

[0113] The above experiments demonstrate that multi-expert fusion, semantic alignment modeling, and semantic mismatch modeling can model audio-text relationships from multiple perspectives, thereby significantly improving the consistency between automatic evaluation results and human subjective judgment, and demonstrating technical effects superior to single modeling methods.

[0114] Example 2:

[0115] To further verify the technical effectiveness of the method proposed in this invention compared with existing technologies, a comparative experiment was conducted between the method proposed in this invention and existing methods. The existing method is an audio-text relevance assessment method, which proposes an automatic assessment model for predicting the relevance score between audio and text. This method mainly extracts embedding features based on a single audio encoder and text encoder, and assesses relevance by calculating the semantic similarity between the audio embedding and the text embedding. This method employs a single model structure, does not use a multi-expert model, and does not explicitly model the semantic mismatch relationship between audio and text.

[0116] Under the same dataset and experimental conditions, correlation metrics with human subjective relevance scores were used for evaluation, including SRCC, LCC, KTAU, and MSE. The experimental results are as follows:

[0117] Table 2: Comparative Experimental Results of the Method of the Present Invention and Existing Techniques

[0118]

[0119] Experimental results show that, compared with existing methods, the proposed method achieves significant improvements in all evaluation metrics. For example, SRCC increased from 0.3844 to 0.6396, LCC from 0.3961 to 0.6615, KTAU from 0.2646 to 0.4635, while MSE decreased from 4.8361 to 3.1911. The experimental results demonstrate that by introducing a multi-expert CLAP model structure and combining semantic alignment modeling and semantic mismatch modeling mechanisms, this invention can more comprehensively characterize the semantic relationship between audio and text. It can not only capture the semantic consistency between the two but also identify semantic mismatch information that has a significant impact on human perception, thereby significantly improving the consistency between automatic evaluation results and human subjective ratings.

[0120] In particular, in some preferred embodiments of the present invention, a computer device is also provided, including a memory and a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the audio-text relevance assessment method based on a multi-expert model described in any of the above embodiments.

[0121] In some other preferred embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program / instruction is stored, wherein when the computer program is executed by a processor, it implements the steps of the audio-text relevance evaluation method based on a multi-expert model as described in any of the above embodiments.

[0122] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above embodiments of the audio-text relevance evaluation method based on a multi-expert model, which will not be repeated here.

[0123] Computer-readable storage media encompass a variety of types, including persistent and non-persistent, portable and fixed. These media store information using different technologies, and the content can be machine instructions, data structures, program modules, or other types of data. Some typical examples of computer storage media include: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), various types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory and other storage technologies, optical storage media such as CD-ROM and digital video disc (DVD), magnetic storage devices such as magnetic tape and disks, and other non-transferable media used to store information accessible to computing devices. It is important to note that the computer-readable media described herein do not include temporary storage media, such as modulated data signals and carrier waves.

[0124] Those skilled in the art will further recognize that the operation of the module can be achieved using existing technical protocols or programs, without relying on new computer programs themselves. The units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0125] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the relevance of audio text based on a multi-expert model, characterized in that, Includes the following steps: The target audio signal and its corresponding text description are obtained. Multiple pre-trained contrastive language-audio pre-trained expert models are used to extract features from the audio signal and the text description respectively, so as to obtain the audio embedding features and text embedding features corresponding to each expert model. For each expert model, semantic alignment evaluation is performed based on the corresponding audio embedding features and text embedding features to obtain the first relevance score; For each expert model, a semantic mismatch assessment is performed based on the corresponding audio embedding features and text embedding features to obtain a second relevance score. The semantic mismatch assessment is performed by constructing and analyzing the difference features between audio and text to show the modeling semantic inconsistency. The first and second relevance scores of each expert model are combined to obtain the comprehensive score of the expert model. The final audio-text relevance prediction score is obtained by combining the scores from multiple expert models.

2. The audio-text relevance assessment method based on a multi-expert model according to claim 1, characterized in that, The first relevance score is an audio-text relevance score based on semantic consistency, and includes the following steps: Calculate the cosine similarity between the audio embedding features and the text embedding features output by each expert model; The cosine similarity is linearly scaled and mapped to a preset relevance scoring range to obtain an audio text relevance score based on semantic consistency.

3. The audio-text relevance assessment method based on a multi-expert model according to claim 2, characterized in that, The audio-text relevance score based on semantic consistency , in, This represents the cosine similarity between audio semantic features and text semantic features under the k-th expert model, where α is a scaling factor used to control the range of values ​​for the semantic alignment score.

4. The audio-text relevance assessment method based on a multi-expert model according to claim 1, characterized in that, The second relevance score is an audio-text relevance score based on semantic mismatch, which includes the following steps: Based on the audio embedding features and text embedding features output by each expert model, a semantic mismatch feature is constructed to characterize the semantic relationship between the two. The semantic mismatch feature integrates the similarity information and the difference information between the two. The semantic mismatch features are input into a multilayer perceptron network for nonlinear mapping to obtain higher-order mismatch features. The higher-order mismatch features are mapped using a scoring head function to obtain an audio-text relevance score based on semantic mismatch.

5. The audio-text relevance assessment method based on a multi-expert model according to claim 4, characterized in that, The semantic mismatch features , in," " indicates a vector concatenation operation, where the first term is the cosine similarity between the text semantic features and the audio semantic features, and the second term is the difference vector between the text semantic features and the audio semantic features; The audio-text relevance score based on semantic mismatch , in, For the scoring header function, This is a high-order mismatch feature.

6. The audio-text relevance assessment method based on a multi-expert model according to claim 1 or 5, characterized in that, The semantic mismatch assessment introduces an auxiliary classification task based on discrete relevance levels; human subjective relevance scores are divided into multiple continuous non-overlapping discrete level categories; the higher-order mismatch features are input into the classifier to predict their true discrete level; a difference loss function for the semantic mismatch branch is constructed based on the difference between the predicted discrete level and the true discrete level, and the multilayer perceptron network and the classifier are jointly optimized. The difference loss function , in, This represents the classification loss function based on angular interval constraints. Represents the feature center constraint loss function. , These are the weighting coefficients. For true discrete levels, For the predicted discrete levels.

7. The audio-text relevance assessment method based on a multi-expert model according to claim 1, characterized in that, The final audio-text relevance prediction score , in, Indicates the first The audio-text relevance score is obtained by fusing the first relevance score and the second relevance score from the expert models; K represents the total number of expert models. To constrain Human subjective relevance score Consistency, constructing the prediction loss function for, 。 8. An audio retrieval system based on a multi-expert model, characterized in that, The system employs the audio-text relevance assessment method based on a multi-expert model as described in any one of claims 1 to 7, wherein the system comprises: The text query module is used to receive natural language text queries input by the user; The audio feature management module is used to store the semantic feature representations of multiple audio samples in the audio library. The semantic feature representations are audio embedding features pre-extracted by the multiple pre-trained contrastive language-audio pre-trained expert models. The audio-text relevance evaluation module is used to perform an audio-text relevance evaluation method based on a multi-expert model based on the text query and the audio embedding features obtained from the audio feature management module, and to calculate the audio-text relevance prediction score between the text query and each audio sample. The retrieval, sorting, and output module is used to sort the audio samples according to the relevance prediction score and output the sorted retrieval results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 7.

Images