An intent-driven intelligent model dual-channel fusion audio retrieval method and system

Abstract

The application discloses an intention-driven intelligent model double-channel fusion audio retrieval method and system, and has the characteristics that a query request and audio data to be retrieved are acquired; semantic analysis is performed on the query request, and probability distribution of the query request corresponding to a voice intention, a content intention and a mixed intention is output; initial fusion weight is determined based on the probability distribution, and the following processing is performed on the audio data to be retrieved in parallel: through a voice semantic channel, voice recognition and semantic representation are performed on the audio to be retrieved, and a first retrieval result set is generated; through an audio content channel, acoustic content features and emotional atmosphere labels of the audio to be retrieved are extracted, and a second retrieval result set is generated; retrieval confidence of each channel is calculated respectively, fusion weight of the voice semantic channel and the audio content channel is dynamically updated based on the initial fusion weight and the retrieval confidence; based on the updated fusion weight, the first retrieval result set and the second retrieval result set are fused by using a weighted reciprocal ranking fusion strategy, and preliminary retrieval results are obtained; time sequence consistency constraint is applied to the preliminary retrieval results, time overlapping segments are merged, and the order of adjacent segments is adjusted, and finally, time sequence coherent final retrieval results are output.

Description

Technical Field

[0001] This application relates to the field of audio information processing and intelligent retrieval technology, specifically to an intent-driven intelligent model dual-channel fusion audio retrieval method and system. Background Technology

[0002] Audio retrieval technology has long developed along two independent directions: one is ASR transcription retrieval, which involves converting speech signals into text using speech recognition technology and then using text retrieval technology for keyword searching. This direction is technically mature, with mainstream solutions based on end-to-end ASR models such as Whisper and Conformer for speech transcription, followed by text retrieval using search engines such as Elasticsearch and FAISS. However, ASR transcription retrieval has inherent drawbacks: the transcription process loses non-textual information such as timbre, emotion, and speech rate, and it cannot handle non-speech audio (such as music and ambient sounds). The second direction is content tag retrieval, which involves categorizing and labeling content based on music genres, environmental scenes, etc., but the tag granularity is coarse and disconnected from semantic retrieval.

[0003] The prior art closest to this invention includes the following two categories of solutions:

[0004] Category 1: Domain-Specific Speech Recognition Methods Based on RAG (Ref. CN119296516A). This technical solution deeply integrates the domain knowledge base with the speech recognition system by constructing a hierarchical knowledge vector index structure. This includes classifying domain documents by topic, segmenting paragraphs and sentences, constructing a hierarchical document structure and performing vectorization processing; performing multi-scale time-frequency analysis on real-time audio streams to extract and enhance acoustic features; and performing multi-level retrieval in the knowledge index based on multi-dimensional query vectors to generate relevant knowledge fragments and assist ASR decoding. Its shortcomings are: knowledge retrieval only serves the one-way enhancement of the speech recognition process and does not feed back the ASR results to the retrieval system for joint optimization; and it only focuses on speech semantic understanding, lacking the ability to perform parallel analysis of audio content (such as music type, ambient sound, emotional atmosphere, etc.).

[0005] The second category: Multi-view speech keyword detection and localization methods (refer to CN116453514A). This technical solution uses two sub-models: an end-to-end text keyword detection model and a sample keyword detection model. These models detect keywords from two perspectives: text-to-speech and speech-to-speech, respectively. Finally, the results of the two models are fused for keyword detection and localization. Its shortcomings are: it focuses only on the single task of keyword detection, without addressing broader audio content retrieval; the multi-view fusion is a fixed-weighted fusion, without dynamic adjustment based on query intent; and it also lacks the ability to analyze audio content.

[0006] The existing technology has the following main defects: (1) The retrieval dimension is single and cannot support both voice semantic retrieval and audio content retrieval at the same time; (2) ASR error propagation directly affects the retrieval quality and lacks a joint optimization mechanism between ASR and the retrieval system; (3) It lacks understanding of query intent and cannot dynamically adjust the retrieval strategy according to the user's intent; (4) It is not good at handling multi-speaker scenarios and cannot support refined retrieval needs; (5) The retrieval results are fragmented in time sequence and the continuous listening experience is not good. Summary of the Invention

[0007] The purpose of this invention is to overcome the technical problems of existing audio retrieval technologies, such as single retrieval dimensions, ASR error propagation, lack of query intent understanding, insufficient handling of multi-speaker scenarios, and fragmented temporal results, and to provide an intelligent dual-channel audio retrieval method.

[0008] On the one hand, this invention provides an intent-driven intelligent model dual-channel fusion audio retrieval method, comprising the following steps:

[0009] 1. An intent-driven intelligent model dual-channel fusion audio retrieval method, characterized by comprising the following steps:

[0010] S1. Obtain the query request and the audio data to be retrieved;

[0011] S2. Perform semantic analysis on the query request and output the probability distribution of the query request corresponding to voice intent, content intent and mixed intent respectively;

[0012] S3. Determine the initial fusion weights based on the probability distribution, and perform the following processing in parallel on the audio data to be retrieved: perform speech recognition and semantic representation on the audio to be retrieved through the speech semantic channel to generate a first retrieval result set; extract the acoustic content features and emotional atmosphere tags of the audio to be retrieved through the audio content channel to generate a second retrieval result set;

[0013] S4. Calculate the retrieval confidence of each channel respectively, and dynamically update the fusion weights of the speech semantic channel and the audio content channel based on the initial fusion weights and the retrieval confidence;

[0014] S5. Based on the updated fusion weights, the first search result set and the second search result set are fused using a weighted inverse ranking fusion strategy to obtain preliminary search results;

[0015] S6. Apply temporal consistency constraints to the preliminary search results, and output the final search results with coherent temporal sequence by merging overlapping time segments and adjusting the order of adjacent segments.

[0016] Further, the generation of the first search result set through the voice semantic channel includes:

[0017] We construct an end-to-end speech recognition engine based on a large-scale pre-trained model and jointly optimize it using a multi-task learning framework.

[0018] The loss function of the multi-task learning framework consists of a weighted average of the ASR cross-entropy loss, which ensures transcription accuracy, and the retrieval relevance comparison loss, which improves retrieval relevance. The weights of the ASR cross-entropy loss and the retrieval relevance comparison loss are dynamically adjusted according to the ASR recognition accuracy: when the ASR word error rate is higher than a first threshold, the weight of the ASR cross-entropy loss is increased; when the ASR word error rate is lower than the first threshold, the weight of the retrieval relevance comparison loss is increased.

[0019] The retrieval relevance comparison loss is configured to: bring the transcribed text embedding vector closer to the positive sample document embedding vector in the feature space, while simultaneously increasing the distance between it and the negative sample document embedding vector in the feature space.

[0020] Furthermore, in step S6, applying a temporal consistency constraint to the preliminary search results specifically includes:

[0021] Construct a temporal adjacency graph of the preliminary retrieval results, and use depth-first search to cluster connected components;

[0022] Calculate the overall coherence score for each connected component, which includes a continuity reward, a coverage reward, and a fragmentation penalty.

[0023] The search results are reordered based on the coherence score, and the following constraints are applied: if the temporal overlap of two segments is greater than the second threshold, they are merged; if the time interval between two segments is less than the third threshold and the semantic similarity is greater than the fourth threshold, they are placed adjacent to each other in the sorting; if the time span of a connected component is greater than the fourth threshold and the coherence score is greater than the fifth threshold, the sorting position of the search results within that connected component is improved; and the temporal boundaries of the search results are aligned to speech pauses or acoustic feature abrupt changes.

[0024] Furthermore, the step of generating a second search result set through the audio content channel also includes:

[0025] An encoder-decoder architecture is used to generate audio content summaries, where the encoder reuses a shared backbone network of audio content channels; during training, the accuracy of the summary content and the retrieval relevance of the summary to the original audio are jointly optimized.

[0026] During training, the effectiveness of the summary retrieval is verified: the generated summary is used as the query to retrieve the original audio in the test set. When the value of the retrieval metric is lower than the sixth threshold, the weight of the retrieval relevance loss is increased.

[0027] Furthermore, the intent recognition specifically includes:

[0028] The query text is triple-encoded using an encoder to obtain sparse embedding, dense embedding, and multi-vector representation;

[0029] Extract keyword preference features, user history behavior features, and application context features from the query text;

[0030] The triple encoding result and auxiliary features are input into a lightweight MLP classifier, which outputs the probability distribution.

[0031] Furthermore, the calculation of the retrieval confidence for each channel specifically includes:

[0032] Calculate the concentration entropy value of the distribution of search results;

[0033] Calculate the mean cosine similarity between the query vector and the search result vector;

[0034] Historical click-through rates of search results are obtained based on historical user behavior data;

[0035] The overall confidence score is calculated by weighting the concentration entropy value, mean cosine similarity, and historical click rate.

[0036] On the other hand, this application provides an intent-driven intelligent model dual-channel fusion audio retrieval system, including:

[0037] The query interface module is used to receive query requests input by users, parse and preprocess the queries, and output query feature vectors and raw audio signals.

[0038] The intent recognition module is used to identify the query intent based on the query feature vector and output the probability distribution of the query request corresponding to voice intent, content intent and mixed intent respectively;

[0039] The speech semantic channel module is used to receive the original audio signal, perform speech activity detection, speaker separation and speech recognition transcription, and generate semantic retrieval results;

[0040] The audio content channel module is used to receive the original audio signal, perform multi-task audio feature extraction, audio event detection and emotional atmosphere tag generation, and generate content retrieval results;

[0041] The fusion decision module is connected to the intent recognition module, the speech semantic channel module, and the audio content channel module, respectively. It is used to determine the fusion weight based on the probability distribution and the retrieval confidence of the two channels, perform weighted fusion of the semantic retrieval result and the content retrieval result using a weighted inverse ranking fusion strategy, apply a temporal consistency constraint to the fused result, and output the final retrieval result.

[0042] An index storage module is connected to the speech semantic channel module and the audio content channel module respectively, and is used to store and retrieve semantic vector data, tag index and acoustic feature data;

[0043] The output interface module is used to receive the final search results, format and timestamp them, and then output them to the outside world.

[0044] Furthermore, the voice semantic channel module includes:

[0045] The speech preprocessing unit is used to segment effective speech segments from the audio signal;

[0046] The speaker separation unit has its input end connected to the output end of the speech preprocessing unit and is used to perform speaker clustering on the speech segments.

[0047] The ASR transcription unit, whose input is connected to the output of the speaker separation unit, is used to transcribe the speech of each speaker into text.

[0048] A hybrid embedding unit, whose input is connected to the output of the ASR transcription unit, is used to generate sparse and dense embedding vectors of the text, and to construct a multi-granularity semantic index based on the embedding vectors.

[0049] Furthermore, the audio content channel module includes:

[0050] Shared feature coding units are used to extract the sequence of acoustic feature vectors from the audio.

[0051] The multi-task parallel branch unit, whose input is connected to the output of the shared feature encoding unit, is used to perform music type recognition, ambient sound classification, white noise detection and human sound detection in parallel.

[0052] An emotional atmosphere construction unit, whose input is connected to the output of the shared feature encoding unit, is used to generate emotional space labels and atmosphere scene labels.

[0053] The summary generation unit, whose input is connected to the output of the shared feature encoding unit, is used to generate a retrieval-oriented audio content summary;

[0054] The content retrieval unit, whose input is connected to the output of the multi-task parallel branch unit, the emotional atmosphere construction unit, and the summary generation unit, is used to construct a multi-dimensional content index based on the music type recognition result, environmental sound classification result, emotional atmosphere tag, and audio content summary, and to perform content retrieval and generate content retrieval results.

[0055] Furthermore, the fusion decision module includes:

[0056] The confidence assessment unit, whose input is connected to the speech semantic channel module and the audio content channel module respectively, is used to calculate the comprehensive confidence score of each channel based on the concentration entropy value of the search result distribution, the mean cosine similarity between the query vector and the search result vector, and the historical click rate of the search results.

[0057] The dynamic weight calculation unit, whose input is connected to the output of the confidence evaluation unit and the output of the intent recognition module, is used to update the fusion weights of the speech semantic channel and the audio content channel based on the probability distribution of intent recognition and the comprehensive confidence score.

[0058] The WRRF fusion unit, whose input is connected to the output of the dynamic weight calculation unit, the output of the speech semantic channel module, and the output of the audio content channel module, is used to fuse the semantic retrieval results and the content retrieval results based on the updated fusion weights and using a weighted inverse ranking fusion strategy.

[0059] The timing constraint unit, whose input is connected to the output of the WRRF fusion unit, is used to apply timing consistency constraints to the fused result and output the final retrieval result.

[0060] Compared with the prior art, the present invention has the following significant advantages:

[0061] This application employs a dual-channel parallel architecture consisting of a speech semantic channel and an audio content channel. The speech semantic channel focuses on extracting semantic information, while the audio content channel focuses on extracting acoustic attributes and emotional atmosphere. The two channels complement each other, enabling the system to handle both traditional text keyword retrieval and acoustic scene retrieval, thus expanding the scope of the search.

[0062] This application introduces a query intent recognition module, which automatically determines whether the user prefers voice semantic retrieval, audio content retrieval, or a hybrid retrieval by analyzing the query text, user history, and context. Based on this, the system dynamically adjusts the fusion weights of the two channels, avoiding the problem of excessive suppression of results from a certain channel under a fixed weight strategy, and achieving adaptive matching between the retrieval strategy and the user's intent.

[0063] In the speech-semantic channel, this application employs a multi-task learning framework, using ASR transcription accuracy and retrieval relevance as joint optimization objectives. By introducing a retrieval relevance contrastive loss, the ASR model is guided to generate transcription results that are more conducive to retrieval (such as retaining keywords), effectively alleviating the technical problem that ASR errors directly lead to retrieval failure in traditional two-stage methods.

[0064] To address the fragmentation issue in existing retrieval results, this application introduces temporal consistency constraints during the fusion decision-making stage. By constructing a temporal adjacency graph, calculating a comprehensive coherence score, and applying rules such as overlap merging and proximity connection, temporally continuous and semantically related segments are forcibly merged or arranged adjacently, outputting temporally coherent audio segments and improving the user experience in continuous listening scenarios. Attached Figure Description

[0065] The accompanying drawings are provided to further understand this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof.

[0066] Figure 1 This is a diagram illustrating the overall architecture of the dual-channel audio intelligent retrieval system in this application embodiment.

[0067] Figure 2 This is a flowchart illustrating the processing of the speech semantic channel in an embodiment of this application;

[0068] Figure 3 This is a flowchart illustrating the processing of the audio content channel in an embodiment of this application.

[0069] Figure 4 This is a flowchart of the dual-channel dynamic fusion process in the embodiments of this application;

[0070] Figure 5 This is a flowchart illustrating the timing consistency constraint processing in the embodiments of this application;

[0071] Figure 6 This is a schematic diagram showing the connection relationship between the various modules of the system in the embodiments of this application. Detailed Implementation

[0072] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. However, those skilled in the art will understand that the embodiments described below are some embodiments of the present invention, but not all embodiments, and are only used to illustrate the present invention, and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Where specific conditions are not specified in the embodiments, conventional conditions or conditions recommended by the manufacturer shall be followed. Where the manufacturers of reagents or instruments are not specified, they are all conventional products that can be purchased commercially.

[0073] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0074] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "connected," and "connected" should be interpreted broadly. For example, they can refer to fixed connection, detachable connection, or integral connection; they can refer to mechanical connection or electrical connection; they can refer to direct connection or indirect connection through an intermediate medium; and they can refer to the connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are only for explaining the invention and are not intended to limit it. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0076] Terminology Explanation:

[0077] Before describing the specific embodiments of the present invention in detail, several key custom terms used in this application are first defined to facilitate understanding:

[0078] 1. ASR, Automatic Speech Recognition: refers to the technology of converting human speech signals into corresponding text.

[0079] 2. BART, Bidirectional and Auto-Regressive Transformers: This invention uses it as a decoder for audio summarization.

[0080] 3. BGE-M3, Baidu General Embedding-Multilingual Multi-task Multi-modal: Baidu's general embedding model supports multilingual, multi-task, and multimodal embedding and encoding models. This invention uses it for query encoding and text embedding.

[0081] 4. BM25, Best Match 25: A classic text retrieval ranking algorithm based on a probabilistic retrieval framework, taking into account factors such as word frequency and document length.

[0082] 5. CER, Character Error Rate: An evaluation metric that measures the difference between ASR output and real text.

[0083] 6. DFS, Depth First Search: A graph traversal algorithm, which is used in this invention to cluster connected components of a temporal adjacency graph.

[0084] 7. ECAPA-TDNN, Emphasized Channel Attention, Propagation and Aggregation in Time Delay Neural Network: This invention uses this model to extract speaker embedding vectors.

[0085] 8. MLP, Multi-Layer Perceptron: A feedforward artificial neural network, which is used in this invention as an intent classifier.

[0086] 9. MT, Multi-Task Learning: A machine learning method that simultaneously optimizes the loss functions of multiple related tasks.

[0087] 10. Mel-scale Spectrogram: A spectral representation method based on the characteristics of human hearing, where the frequency axis uses the Mel-scale instead of the nonlinear frequency.

[0088] 11. NDCG, Normalized Discounted Cumulative Gain: A standard evaluation metric for measuring the ranking quality of search results.

[0089] 12. PANNs, Pretrained Audio Neural Networks: This invention uses this model to classify and detect more than 500 types of audio events.

[0090] 13. RAG, Retrieval-Augmented Generation: A technical framework that combines external knowledge retrieval with large language model generation.

[0091] 14. RTTM, Rich Transcription Time Marked: A standard format for storing speaker separation results.

[0092] 15. STFT, Short-Time Fourier Transform: A mathematical transformation that converts a time-domain signal into a time-frequency domain representation.

[0093] 16.T5, Text-to-Text Transfer Transformer: A text-to-text transformation Transformer model that can be used as an alternative to BART for summarization generation.

[0094] 17. VAD, Voice Activity Detection: refers to the technology of identifying speech segments and non-speech segments from audio signals.

[0095] 18. WRRF, Weighted Reciprocal Rank Fusion: The fusion algorithm used in this invention merges multi-way retrieval results by weighting the reciprocal ranking scores.

[0096] 19. Four-dimensional emotional space: The emotional representation system defined in this invention includes four dimensions: Valence, Energy, Tension, and Space.

[0097] 20. Connected Component: A graph theory concept referring to a subset of mutually connected nodes in an undirected graph. In this invention, the connected components of a temporal adjacency graph represent a set of temporally overlapping retrieval results.

[0098] 21. Audio Content Channel: The second processing channel defined in this invention focuses on extracting acoustic content attributes (i.e., what kind of sound) from audio signals, including processes such as audio event detection, emotional atmosphere tagging, summary generation, and content retrieval.

[0099] 22. Speech-Semantic Channel: One of the processing channels defined in this invention, which focuses on extracting speech-semantic information (i.e., what content is being said) from audio signals, including processes such as VAD, speaker separation, ASR transcription, and semantic retrieval.

[0100] Example 1

[0101] See Figures 1 to 5 This embodiment provides an intent-driven intelligent model dual-channel fusion audio retrieval method, which specifically includes the following steps:

[0102] S1. Obtain the query request and the audio data to be retrieved.

[0103] The system receives query requests from users through a query interface. Query requests can be in text format (such as searching for conference speeches about AI technology) or in audio clips (such as humming a melody). Simultaneously, the system retrieves the audio database to be searched from the index storage module.

[0104] S2. Perform semantic analysis on the query request and output the probability distribution of the query request corresponding to voice intent, content intent and mixed intent respectively.

[0105] This step aims to address the technical problem of existing technologies lacking an understanding of user query intent.

[0106] Specifically, the query text is triple-encoded (sparse embedding, dense embedding, and multi-vector representation) using an encoder (in a preferred embodiment of this application, a BGE-M3 encoder may be used). Simultaneously, keyword preference features (such as whether it contains speech-related words or content-related words such as music or sound), user historical behavior features (such as whether the user has historically preferred voice retrieval or content retrieval), and application context features (such as whether the current context is a meeting recording scenario or a music library management scenario) are extracted from the query text.

[0107] The above features are input into a lightweight MLP (Multilayer Perceptron) classifier, which outputs three probability values: P(speech intent), P(content intent), and P(mixed intent). This probability distribution will serve as the basis for subsequent dynamic weight allocation.

[0108] S3. Determine the initial fusion weights based on the probability distribution, and perform the following processing in parallel on the audio data to be retrieved.

[0109] This step constructs the core dual-channel architecture of this application to solve the problem of the single retrieval dimension in existing technologies.

[0110] Based on the intent recognition results, set the initial weights for the two channels:

[0111] (1) When P (voice intent) > 0.7, set to semantic preference mode: w s = 0.9, w c = 0.1;

[0112] (2) When P (content intent) > 0.7, set to content preference mode: w s = 0.1, w c = 0.9;

[0113] (3) Other cases (i.e., mixed intentions) are set to balanced mode: w s = w c = 0.5.

[0114] The threshold of 0.7 mentioned above is an empirical value and can be adjusted according to the actual application scenario. In the hybrid intent mode, the subsequent confidence assessment will dominate the dynamic adjustment of the weights.

[0115] The following processing is performed in parallel on the audio data to be retrieved:

[0116] Channel 1: Voice and Semantic Channel

[0117] This channel is primarily responsible for extracting textual semantic information from the audio.

[0118] Speech preprocessing and speaker separation: Effective speech segments are segmented using endpoint detectors based on deep neural networks (such as Silero-VAD), and voiceprint features are extracted using the ECAPA-TDNN model. Multi-speaker separation is achieved through spectral clustering algorithm.

[0119] ASR Transcription and Joint Optimization: A large-scale pre-trained model (such as Whisper-large-v3) is used as the base ASR model. Unlike existing technologies, this application employs a multi-task learning framework for joint optimization. The loss function is a weighted average of the ASR cross-entropy loss and the retrieval relevance comparison loss (such as InfoNCE loss), with the specific formula as follows:

[0120] (Formula 1),

[0121] Among them, L total Let L be the loss function. asr For ASR cross-entropy loss, L retrieval为 Search relevance comparison loss, λ asrLet λ be the initial weight for the ASR cross-entropy loss, set to 0.7. ret The initial weight for the relevance-based contrast loss is set to 0.3. During training, when the ASR word error rate exceeds 15%, λ is increased. asr To ensure basic recognition capability; when the ASR word error rate is below 15%, increase λ. ret To enhance retrieval relevance, the ASR model is guided to generate more retrieval-friendly text by narrowing the distance between the transcribed text embedding vector and the positive sample document embedding vector, thereby reducing the negative impact of ASR errors on retrieval.

[0122] Hybrid embedding and retrieval: Sparse embeddings (based on word frequency statistics) are generated for the transcribed text. The sparse embedding uses the BM25 algorithm to calculate word weights and introduces ASR confidence weighting optimization: For words with ASR confidence below 0.6, their BM25 weights are reduced proportionally (reduction coefficient = confidence / 0.6) and dense embedding (based on a pre-trained language model) are used to construct a multi-granularity index and generate the first retrieval result set.

[0123] Channel Two: Audio Content Channel

[0124] This channel is primarily responsible for extracting the acoustic properties and emotional atmosphere of the audio.

[0125] Multi-task feature extraction: Extract high-dimensional feature vector sequences of audio using a shared backbone network (such as a Transformer-based network).

[0126] Multi-task parallel analysis: Parallel execution of music genre recognition (genre / emotion / scene / instrument), environmental sound classification (e.g., PANNs model to recognize 500+ environmental sounds), white noise detection, and human sound detection (applause, laughter, etc.).

[0127] Emotional Atmosphere and Summary: A two-layer emotional atmosphere labeling system is constructed. The base layer is a four-dimensional emotional space (pleasure, energy, tension, and spatiality), with each dimension ranging from -1 to 1. The extended layer consists of atmosphere-related scene labels (such as being alone late at night, waking up in the morning, exercising, and focusing on work). Furthermore, an encoder-decoder architecture (such as BART / T5) is used to generate retrieval-oriented audio content summaries. During training, the accuracy and retrieval relevance of the summaries are jointly optimized, with a content accuracy loss weight of 0.8 and a retrieval relevance loss weight of 0.2. During training, the generated summaries are used as the original audio for query retrieval, and the Recall@10 metric is calculated. When Recall@10 is below 0.6, the weight of the retrieval relevance loss is increased.

[0128] Content retrieval: A multi-dimensional index is constructed based on tags, summaries, and acoustic features to generate a second set of search results.

[0129] S4. Calculate the retrieval confidence of each channel respectively, and dynamically update the fusion weights of the speech semantic channel and the audio content channel based on the initial fusion weights and the retrieval confidence.

[0130] To achieve more accurate fusion, this application introduces a real-time confidence assessment mechanism. The confidence assessment formula is as follows: (Formula 2),

[0131] Where α is the result distribution concentration weight coefficient, with a value of 0.4, β is the query-result matching degree weight coefficient, with a value of 0.4, and γ is the historical feedback rating weight coefficient, with a value of 0.2.

[0132] Calculate the overall confidence score for each channel, which can be weighted based on the following dimensions:

[0133] Result distribution concentration: measures whether the search results are concentrated in the high-scoring region (the lower the entropy value, the higher the concentration).

[0134] Query-result matching degree: Calculate the mean cosine similarity between the query vector and the retrieved result vector.

[0135] Historical feedback rating: Weighted based on historical user behavior data (such as click-through rate).

[0136] The dynamic weight update formula can be:

[0137] (Formula 3),

[0138] (Formula 4),

[0139] Where λ is the smoothing coefficient, λ=0.7, used to balance the initial weights and confidence feedback; c s and c c These represent the combined confidence scores for the speech / semantic channel and the audio content channel, respectively. The smoothing coefficient λ is set to 0.7 to ensure the responsiveness of weight adjustments while avoiding excessive weight fluctuations that could lead to unstable search results.

[0140] S5. Based on the updated fusion weights, the first search result set and the second search result set are fused using a weighted inverse ranking fusion strategy to obtain preliminary search results.

[0141] The WRRF algorithm is used to merge and score the two results based on the updated weights. The WRRF algorithm assigns higher weights to the results ranked higher, effectively preserving the highly correlated results of each channel.

[0142] S6. Apply temporal consistency constraints to the preliminary search results, and output the final search results with coherent temporal sequence by merging overlapping time segments and adjusting the order of adjacent segments.

[0143] This step aims to address the issue of fragmented search results.

[0144] Construct a temporal adjacency graph: Extract the time interval of each search result, calculate the temporal overlap between pairs, and construct an undirected graph.

[0145] Connected component clustering: The depth-first search (DFS) algorithm is used to cluster connected components, grouping segments that overlap or are adjacent in time into a group.

[0146] Consistency scoring and rule application:

[0147] Calculate the overall coherence score for each connected component (including continuity reward, coverage reward, and fragmentation penalty).

[0148] The following constraints are applied: if the temporal overlap between two segments is greater than 0.8, they are merged; if the time interval is less than 2 seconds and the semantic similarity is greater than 0.6, they are forced to be adjacent; if the time span of a connected component is greater than 30 seconds and the coherence score is greater than 0.5, the ranking of the search results within that component is improved; and the temporal boundaries of the search results are aligned to speech pauses or acoustic feature abrupt changes, with the total duration of a single returned result not exceeding 300 seconds.

[0149] Final reordering: The results are reordered based on the coherence score, and the top-K temporally coherent audio segments are output.

[0150] Example 2

[0151] See Figure 1 and Figure 6 The system of this invention adopts a dual-channel parallel processing and intent-driven dynamic fusion overall architecture, which mainly includes seven functional modules: Query Interface Module (QIM), Intent Recognition Module (IRM), Speech Semantic Channel Module (SSC), Audio Content Channel Module (ACC), Fusion Decision Module (FDM), Index Storage Module (ISM), and Output Interface Module (OIM).

[0152] Query Interface Module: Receives user-input query requests, supporting both audio upload and text input queries. Performs preliminary parsing and preprocessing of the queries. Outputs the parsed query text vector and query feature vector to the intent recognition module, and outputs the raw audio signal to the dual-channel processing module.

[0153] The intent recognition module identifies the user's query intent category based on query text features, user history behavior, and application context. A BGE-M3 encoder is used to perform triple encoding (sparse embedding, dense embedding, and multi-vector representation) on the query text to extract query text features. Simultaneously, keyword tendency features, user history behavior features, and application context features are extracted. These features are concatenated and input into a lightweight MLP classifier (a three-layer fully connected network, 256 hidden layer dimensions, ReLU activation function, and Softmax output layer), outputting three probability values: P(voice intent), P(content intent), and P(mixed intent), with a sum of probabilities of 1. The intent recognition results are used to guide subsequent dynamic weight allocation.

[0154] The speech-semantic channel module performs speech activity detection, speaker segmentation, ASR transcription and joint optimization, hybrid embedding generation, and semantic retrieval on the input audio, outputting semantic-dimensional retrieval results. Its internal processing flow is as follows: Figure 2 As shown, it specifically includes:

[0155] Speech preprocessing unit: The audio signal is segmented into speech segments and non-speech segments using the Silero-VAD model, and 80-dimensional log-Mel filter bank features are extracted (frame length 25ms, frame shift 10ms).

[0156] Speaker separation unit: The ECAPA-TDNN model is used to extract 192-dimensional speaker embedding vectors, and the speakers are clustered by the spectral clustering algorithm. The number of clusters is automatically determined by the silhouette coefficient.

[0157] ASR transcription unit: The Whisper-large-v3 model was used to transcribe the speech of each speaker after separation, and the retrieval relevance was optimized through a multi-task learning framework (see the joint optimization section below for details).

[0158] Hybrid embedding unit: Transcribed text simultaneously generates sparse embeddings (based on BM25 weighting and optimized by ASR confidence weighting) and dense embeddings (BGE-M3 encoding) to construct speaker-level, dialogue-level and topic-level multi-granularity semantic indexes.

[0159] The audio content channel module performs multi-task audio feature extraction, audio event detection, emotional atmosphere tag generation, search-oriented summary generation, and content retrieval on the input audio, outputting content-based search results. Its internal processing flow is as follows: Figure 3 As shown, it specifically includes:

[0160] Shared feature coding unit: The Mel spectrum is generated through short-time Fourier transform and Mel filter bank, and a 768-dimensional audio feature vector sequence is output by a Transformer-based shared backbone network.

[0161] Multi-task parallel branch unit: performs music genre recognition (four-dimensional labels of genre / mood / scene / instrument, based on PANNs model), ambient sound classification (recognizing more than 500 kinds of ambient sounds), white noise detection and human sound detection in parallel.

[0162] Emotional Atmosphere Construction Unit: Construct a two-layer emotional atmosphere label system. The basic layer is a four-dimensional emotional space (pleasure, energy, tension, and spatiality), and the extended layer is atmosphere scene labels (such as 32 categories such as being alone at night, waking up in the morning, and exercising).

[0163] Abstract generation unit: The shared encoder is reused as the encoder and connected to the BART / T5 decoder to generate audio content abstracts. The retrieval relevance constraint is incorporated into the training objective, and the retrieval relevance loss weight in the training objective is 0.2.

[0164] Content retrieval unit: Constructs a multi-dimensional content index based on music type recognition results, ambient sound classification results, emotional atmosphere tags, and audio content summaries to perform content retrieval.

[0165] The fusion decision module receives the search results from both channels, dynamically assigns weights based on intent recognition probability and channel confidence assessment, performs WRRF fusion and temporal consistency constraint reordering, and outputs the final fused search results. Its internal processing flow is as follows: Figure 4 As shown.

[0166] Index storage module: Stores and manages all index data, including the semantic vector database (Milvus / FAISS), the tag index (Elasticsearch), the acoustic feature library, the summary text library, and the metadata management library.

[0167] Output interface module: Formats, timestamps, and renders the fused search results, providing a standardized API.

[0168] Example 3

[0169] In this embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method steps in Embodiment 1 above.

[0170] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method. The storage medium may be ROM, RAM, a magnetic disk, or an optical disk, etc.

[0171] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. This application is not limited to the exact structures described above and illustrated in the accompanying drawings, and it should not be considered that the specific implementation of this application is limited to these descriptions. For those skilled in the art, various changes and modifications made without departing from the concept of this application should be considered to fall within the protection scope of this application.

Claims

1. An intent-driven intelligent model dual-channel fusion audio retrieval method, characterized in that, include: S1. Obtain the query request and the audio data to be retrieved; S2. Perform semantic analysis on the query request and output the probability distribution of the query request corresponding to voice intent, content intent and mixed intent respectively; S3. Determine the initial fusion weights based on the probability distribution, and perform the following processing in parallel on the audio data to be retrieved: The audio to be retrieved is subjected to speech recognition and semantic representation through the speech semantic channel to generate a first retrieval result set; By extracting the acoustic content features and emotional atmosphere tags of the audio to be retrieved through the audio content channel, a second search result set is generated; S4. Calculate the retrieval confidence of each channel respectively, and dynamically update the fusion weights of the speech semantic channel and the audio content channel based on the initial fusion weights and the retrieval confidence; S5. Based on the updated fusion weights, the first search result set and the second search result set are fused using a weighted inverse ranking fusion strategy to obtain preliminary search results; S6. Apply temporal consistency constraints to the preliminary search results, and output the final search results with coherent temporal sequence by merging overlapping time segments and adjusting the order of adjacent segments.

2. The intent-driven intelligent model dual-channel fusion audio retrieval method according to claim 1, characterized in that, The generation of the first search result set through the voice semantic channel includes: We construct an end-to-end speech recognition engine based on a large-scale pre-trained model and jointly optimize it using a multi-task learning framework. The loss function of the multi-task learning framework consists of a weighted average of the ASR cross-entropy loss, which ensures transcription accuracy, and the retrieval relevance comparison loss, which improves retrieval relevance. The weights of the ASR cross-entropy loss and the retrieval relevance comparison loss are dynamically adjusted according to the ASR recognition accuracy: when the ASR word error rate is higher than a first threshold, the weight of the ASR cross-entropy loss is increased; when the ASR word error rate is lower than the first threshold, the weight of the retrieval relevance comparison loss is increased. The retrieval relevance comparison loss is configured to: bring the transcribed text embedding vector closer to the positive sample document embedding vector in the feature space, while simultaneously increasing the distance between it and the negative sample document embedding vector in the feature space.

3. The intent-driven intelligent model dual-channel fusion audio retrieval method according to claim 1, characterized in that, The temporal consistency constraint applied to the preliminary search results in step S6 specifically includes: Construct a temporal adjacency graph of the preliminary retrieval results, and use depth-first search to cluster connected components; Calculate the overall coherence score for each connected component, which includes a continuity reward, a coverage reward, and a fragmentation penalty. The search results are reordered based on the coherence score, and the following constraints are applied: if the temporal overlap of two segments is greater than the second threshold, they are merged; if the time interval between two segments is less than the third threshold and the semantic similarity is greater than the fourth threshold, they are placed adjacent to each other in the sorting; if the time span of a connected component is greater than the fourth threshold and the coherence score is greater than the fifth threshold, the sorting position of the search results within that connected component is improved; and the temporal boundaries of the search results are aligned to speech pauses or acoustic feature abrupt changes.

4. The intent-driven intelligent model dual-channel fusion audio retrieval method according to claim 1, characterized in that, The process of generating a second search result set via the audio content channel further includes: An encoder-decoder architecture is used to generate audio content summaries, where the encoder reuses a shared backbone network of audio content channels; during training, the accuracy of the summary content and the retrieval relevance of the summary to the original audio are jointly optimized. During training, the effectiveness of the summary retrieval is verified: the generated summary is used as the query to retrieve the original audio in the test set. When the value of the retrieval metric is lower than the sixth threshold, the weight of the retrieval relevance loss is increased.

5. The intent-driven intelligent model dual-channel fusion audio retrieval method according to claim 1, characterized in that, The intent recognition specifically includes: The query text is triple-encoded using an encoder to obtain sparse embedding, dense embedding, and multi-vector representation; Extract keyword preference features, user history behavior features, and application context features from the query text; The triple encoding result and auxiliary features are input into a lightweight MLP classifier, which outputs the probability distribution.

6. The intent-driven intelligent model dual-channel fusion audio retrieval method according to claim 1, characterized in that, The calculation of the retrieval confidence score for each channel specifically includes: Calculate the concentration entropy value of the distribution of search results; Calculate the mean cosine similarity between the query vector and the search result vector; Historical click-through rates of search results are obtained based on historical user behavior data; The overall confidence score is calculated by weighting the concentration entropy value, mean cosine similarity, and historical click rate.

7. An intent-driven intelligent model dual-channel fusion audio retrieval system, characterized in that, include: The query interface module is used to receive query requests input by users, parse and preprocess the queries, and output query feature vectors and raw audio signals. The intent recognition module is used to identify the query intent based on the query feature vector and output the probability distribution of the query request corresponding to voice intent, content intent and mixed intent respectively; The speech semantic channel module is used to receive the original audio signal, perform speech activity detection, speaker separation and speech recognition transcription, and generate semantic retrieval results; The audio content channel module is used to receive the original audio signal, perform multi-task audio feature extraction, audio event detection and emotional atmosphere tag generation, and generate content retrieval results; The fusion decision module is connected to the intent recognition module, the speech semantic channel module, and the audio content channel module, respectively. It is used to determine the fusion weight based on the probability distribution and the retrieval confidence of the two channels, perform weighted fusion of the semantic retrieval result and the content retrieval result using a weighted inverse ranking fusion strategy, apply a temporal consistency constraint to the fused result, and output the final retrieval result. An index storage module is connected to the speech semantic channel module and the audio content channel module respectively, and is used to store and retrieve semantic vector data, tag index and acoustic feature data; The output interface module is used to receive the final search results, format and timestamp them, and then output them to the outside world.

8. The intent-driven intelligent model dual-channel fusion audio retrieval system according to claim 7, characterized in that, The voice semantic channel module includes: The speech preprocessing unit is used to segment effective speech segments from the audio signal; The speaker separation unit has its input end connected to the output end of the speech preprocessing unit and is used to perform speaker clustering on the speech segments. The ASR transcription unit, whose input is connected to the output of the speaker separation unit, is used to transcribe the speech of each speaker into text. A hybrid embedding unit, whose input is connected to the output of the ASR transcription unit, is used to generate sparse and dense embedding vectors of the text, and to construct a multi-granularity semantic index based on the embedding vectors.

9. The intent-driven intelligent model dual-channel fusion audio retrieval system according to claim 7, characterized in that, The audio content channel module includes: Shared feature coding units are used to extract the sequence of acoustic feature vectors from the audio. The multi-task parallel branch unit, whose input is connected to the output of the shared feature encoding unit, is used to perform music type recognition, ambient sound classification, white noise detection and human sound detection in parallel. An emotional atmosphere construction unit, whose input is connected to the output of the shared feature encoding unit, is used to generate emotional space labels and atmosphere scene labels. The summary generation unit, whose input is connected to the output of the shared feature encoding unit, is used to generate a retrieval-oriented audio content summary; The content retrieval unit, whose input is connected to the output of the multi-task parallel branch unit, the emotional atmosphere construction unit, and the summary generation unit, is used to construct a multi-dimensional content index based on the music type recognition result, environmental sound classification result, emotional atmosphere tag, and audio content summary, and to perform content retrieval and generate content retrieval results.

10. The intent-driven intelligent model dual-channel fusion audio retrieval system according to claim 7, characterized in that, The fusion decision module includes: The confidence assessment unit, whose input is connected to the speech semantic channel module and the audio content channel module respectively, is used to calculate the comprehensive confidence score of each channel based on the concentration entropy value of the search result distribution, the mean cosine similarity between the query vector and the search result vector, and the historical click rate of the search results. The dynamic weight calculation unit, whose input is connected to the output of the confidence evaluation unit and the output of the intent recognition module, is used to update the fusion weights of the speech semantic channel and the audio content channel based on the probability distribution of intent recognition and the comprehensive confidence score. The WRRF fusion unit, whose input is connected to the output of the dynamic weight calculation unit, the output of the speech semantic channel module, and the output of the audio content channel module, is used to fuse the semantic retrieval results and the content retrieval results based on the updated fusion weights and using a weighted inverse ranking fusion strategy. The timing constraint unit, whose input is connected to the output of the WRRF fusion unit, is used to apply timing consistency constraints to the fused result and output the final retrieval result.

Images