An end-to-end speech dialogue system and method based on cross-modal retrieval enhancement
The end-to-end voice dialogue system enhanced by cross-modal retrieval achieves direct mapping and generation of voice to voice, solves the problems of module error accumulation and knowledge retrieval limitations, improves the system's real-time interaction capabilities and accuracy, and is suitable for high-requirement scenarios such as intelligent customer service.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNICOM WO MUSIC & CULTURE CO LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-29
AI Technical Summary
Existing voice dialogue systems suffer from problems such as module error accumulation, limited knowledge retrieval, and high response latency, making it difficult to achieve real-time interaction and accurate response, especially in vertical fields.
An end-to-end voice dialogue system with cross-modal retrieval enhancement is adopted. Through voice input and feature extraction modules, cross-modal retrieval enhancement modules, end-to-end voice generation modules, and voice memory modules, it realizes direct mapping from voice to text and personalized voice generation. It combines semantic and pragmatic features for cross-modal retrieval and uses the GLM-4-Voice model architecture and weighted matching strategy to dynamically adjust weights to achieve efficient knowledge retrieval.
It eliminates the error accumulation caused by modular architecture, improves system robustness and real-time interaction capabilities, ensures the accuracy and timeliness of response information, enhances the system's cognitive ability in vertical fields, and is suitable for application scenarios such as intelligent customer service.
Smart Images
Figure CN122116896A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of wireless communication technology, and in particular to an end-to-end voice dialogue system and method based on cross-modal retrieval enhancement. Background Technology
[0002] Currently, mainstream voice dialogue systems typically employ a modular pipeline architecture, comprising three core components: Automatic Speech Recognition (ASR), Large Language Model (LLM), and Text-to-Speech (TTS). The main process is as follows: ASR converts the user's input speech signal into text; LLM performs semantic understanding of the text to generate corresponding text responses; and TTS converts the generated text into speech output. To enhance the model's domain-specific expertise, some voice dialogue systems introduce Retrieval Augmentation (RAG) technology, but its retrieval modality is limited to text. Pipeline-based systems also require ASR transcription before retrieval, and ASR recognition errors can affect retrieval and subsequent generation quality, leading to error accumulation and severely restricting system performance and user experience. Furthermore, while end-to-end voice dialogue systems can directly generate voice responses from speech input, the difference between the input speech and the RAG text knowledge base modality makes it difficult to effectively match the "heard question" with the "text knowledge to be retrieved." This results in these models being unable to update their internal knowledge base in real time and lacking domain-specific expertise, leading to outdated or inaccurate answers. The main shortcomings of existing voice dialogue systems and methods are:
[0003] Accumulated module errors and information loss: In a pipelined architecture, multiple modules operate in series, resulting in a fragmented information flow of audio-text-text-audio. Errors in the ASR (Automatic Speech Recognition) stage lead to the propagation and amplification of errors at each stage, causing subsequent generation errors. Furthermore, the high latency of waiting for complete ASR and TTS processing severely restricts overall system performance and user experience. Paralinguistic information contained in speech (such as intonation, stress, and emotion) is completely lost during text conversion, yet this information is crucial for accurately understanding user intent and retrieving the most relevant knowledge.
[0004] Limitations of knowledge retrieval and insufficient cross-modal capabilities: Existing technologies that incorporate retrieval enhancement are limited to a single retrieval modality, unable to understand and retrieve knowledge from other modalities such as audio. In pipelined dialogue models, retrieval requires ASR to be converted to text before proceeding, failing to achieve direct access from "speech" to "knowledge." The retrieval process relies too heavily on text embedding and lacks utilization of speech modal features, resulting in insufficient retrieval accuracy.
[0005] High response latency: Data needs to be processed sequentially between multiple modules, making it difficult to meet the requirements of real-time interaction.
[0006] Therefore, it is necessary to improve and optimize the design of end-to-end voice dialogue systems and methods. Summary of the Invention
[0007] To address the aforementioned problems, this invention provides an end-to-end voice dialogue system based on cross-modal retrieval enhancement, comprising a voice input and feature extraction module, a cross-modal retrieval enhancement module, an end-to-end voice generation module, and a voice memory module;
[0008] The speech input and feature extraction module is used to input speech signals and extract multi-dimensional features of speech, including semantic features and pragmatic features.
[0009] The cross-modal retrieval enhancement module is used to incorporate semantic and pragmatic features into knowledge retrieval, perform cross-modal retrieval from speech to text, and obtain the most relevant text knowledge;
[0010] The end-to-end speech generation module is used to generate personalized speech based on the most relevant text for voice interaction.
[0011] The voice memory module is used to record, extract, and utilize the user's voice features, tone habits, and semantic preferences during the interaction process to achieve personalized voice style generation.
[0012] As a further explanation of the present invention, the cross-modal retrieval enhancement module includes a pragmatic awareness module and a cross-modal knowledge alignment module, wherein,
[0013] The pragmatic awareness module is used to integrate the semantic and pragmatic features of speech during the knowledge retrieval stage to perform cross-modal adaptive knowledge retrieval.
[0014] The cross-modal knowledge alignment module includes a speech encoder, a text encoder, a projection network, and a cross-modal retrieval unit; the speech encoder is used to process speech to obtain speech embedding vectors, the text encoder is used to process text to obtain text embedding vectors; the projection network is used to map the speech embedding vectors to the text embedding vectors; and the cross-modal retrieval unit is used to perform nearest neighbor search.
[0015] Furthermore, the end-to-end speech generation module adopts a GLM-4-Voice model architecture, including a SpeechTokenizer module, an LLM module, and a Speech Decoder module.
[0016] On the other hand, the present invention also provides an end-to-end voice dialogue method based on cross-modal retrieval enhancement, comprising the following steps:
[0017] S1, User voice input, extract features from the input voice, and convert the input voice signal into a voice feature representation through a voice encoder;
[0018] S2, the speech features are represented as query vectors, and the most relevant text knowledge is retrieved directly from an external text knowledge base in a speech-to-text manner through a cross-modal retrieval mechanism;
[0019] S3 generates voice responses directly through an end-to-end dialogue model based on the most relevant text knowledge retrieved.
[0020] Furthermore, in step S1, the features extracted from the input speech include semantic features and pragmatic features, and in step S2, when performing cross-modal retrieval, the semantic features and pragmatic features are incorporated.
[0021] Furthermore, in the knowledge retrieval process of step S2, a weighted matching strategy is adopted to weight and fuse semantic similarity and pragmatic feature similarity, including the following formula:
[0022] S core =α×Sim semantic +β×Sim pragmatic ①
[0023] α + β = 1, α ≥ β ②
[0024] [α, β] = softmax(W × E) programtic +b) ③
[0025] Where α and β are weighted fusion hyperparameters, Sim semantic Sim uses semantic similarity to reflect the importance of semantic matching. pragmatic For pragmatic similarity, reflecting the degree of influence of tone / emotion, W and b are learnable parameters, and E... programtic For pragmatic embedding, Formula ① is the weighted adjustment rule for semantic and pragmatic similarity, Formula ② is the relative relationship between α and β, and Formula ③ is the adjustment strategy for α and β.
[0026] Furthermore, the weighted fusion hyperparameters α and β are adaptively and dynamically adjusted according to the emotional state and tone intensity of the speech. This is achieved through a weight prediction network, which takes pragmatic embedding as input and outputs semantic weights α and pragmatic weights β after passing through a fully connected layer and Softmax normalization, thus achieving adaptive fusion.
[0027] Furthermore, in the knowledge retrieval process of step S2: a speech encoder is used to process the speech to obtain a compressed speech embedding vector v_audio; a text encoder is used to process the text to obtain a text embedding vector v_text; v_audio and v_text are mapped to the same shared semantic vector space through a projection network, and a loss function is used for training. After training, the parameters of the speech encoder and the projection network are fixed, and a vector index is constructed to map the speech and text to the same semantic space. The speech vector is used as a query to directly perform nearest neighbor search in the text vector index.
[0028] Furthermore, after step S3, there is also step S4 memory update, which records, extracts and utilizes the user's voice features, tone habits and semantic preferences during the interaction process to achieve personalized voice style generation.
[0029] The beneficial effects of this invention are:
[0030] The voice dialogue system and method of this invention can achieve direct mapping and generation of voice to voice, eliminate the error accumulation problem caused by modular architecture, and improve the overall robustness of the system; reduce system complexity, simplify the training and deployment process through end-to-end model, improve the scalability and real-time interaction capability of the system, and make it more suitable for application scenarios such as intelligent customer service with extremely high requirements for accuracy and efficiency; through efficient cross-modal retrieval and enhanced generation method, it can retrieve relevant professional knowledge from the text knowledge base in real time based on the received audio, ensuring the accuracy and timeliness of response information, realizing direct, efficient and accurate retrieval of voice to text knowledge, enhancing the system's cognitive ability in vertical fields, and solving the problems of knowledge retrieval limitations and insufficient cross-modal capabilities. Attached Figure Description
[0031] Figure 1 This is a flowchart illustrating the end-to-end voice dialogue system and method based on cross-modal retrieval enhancement of the present invention. Detailed Implementation
[0032] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0033] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", "first", "second", etc., indicate the orientation, position or order relationship based on the orientation, position or order relationship shown in the drawings, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.
[0034] An end-to-end voice dialogue system based on cross-modal retrieval enhancement is invented, comprising a voice input and feature extraction module, a cross-modal retrieval enhancement module, an end-to-end voice generation module, and a voice memory module;
[0035] The voice input and feature extraction module is used to input voice signals and extract multi-dimensional features of the voice, including semantic features and pragmatic features. The user inputs voice signals through a microphone, and the system extracts multi-dimensional features of the voice through a voice encoder, including: semantic features (content layer information) and pragmatic features (tone, pitch, rhythm, emotion). The voice encoder outputs two vectors: semantic embedding and pragmatic embedding, and sends them to subsequent modules.
[0036] The cross-modal retrieval enhancement module is used to incorporate semantic and pragmatic features into knowledge retrieval, perform cross-modal retrieval from speech to text, and obtain the most relevant text knowledge;
[0037] The end-to-end speech generation module is used to generate personalized speech based on the most relevant text for voice interaction.
[0038] The voice memory module is used to record, extract, and utilize the user's voice features, tone habits, and semantic preferences during the interaction process to achieve personalized voice style generation.
[0039] In practical applications, the cross-modal retrieval enhancement module includes a pragmatic awareness module and a cross-modal knowledge alignment module, wherein,
[0040] The pragmatic perception module is used to integrate the semantic and pragmatic features of speech during the knowledge retrieval stage to perform cross-modal adaptive knowledge retrieval. The module receives speech feature representations from the speech encoder, including semantic embedding vectors and pragmatic embedding vectors. The semantic embedding reflects the semantic information of the speech content, while the pragmatic embedding is composed of acoustic features (pitch, rhythm) and emotional features (emotional tendency, tension, tone category), which are used to characterize the speaker's tone and intention.
[0041] The cross-modal knowledge alignment module includes a speech encoder, a text encoder, a projection network, and a cross-modal retrieval unit. The speech encoder processes speech to obtain speech embedding vectors, and the text encoder processes text to obtain text embedding vectors. The projection network maps the speech embedding vectors to the text embedding vectors. The cross-modal retrieval unit performs nearest neighbor search. The cross-modal retrieval mechanism based on "speech-text contrastive learning" aims to ensure that a speech segment is close to its semantically related text in the vector space, while unrelated text is far away, allowing direct retrieval of text using the speech vectors. First, the speech encoder processes the speech to obtain a compressed speech embedding vector v_audio. Then, a text encoder processes the text to obtain a text embedding vector v_text. A projection network (usually a simple MLP) is then inserted after each encoder to map v_audio and v_text to the same shared semantic vector space. The infoNCE loss function is used for training. The goal is to maximize the cosine similarity of positive sample pairs (v_audio, v_text) in a batch while minimizing the similarity with negative sample pairs. The training process for pragmatic feature matching is similar.
[0042] After training, the parameters of the speech encoder and projection network are fixed. All text entries in the entire knowledge base k can have their vector representations pre-computed using the text encoder and projection network, and a vector index can be constructed. During online inference, the input speech X_audio is processed by the speech encoder and projection network to obtain the query vector q. By performing nearest neighbor search in the vector index, the most relevant text knowledge can be quickly retrieved, achieving cross-modal retrieval of speech and text. This mechanism allows speech input to be directly mapped to the text knowledge base without going through a speech recognition conversion step, fundamentally eliminating the error accumulation and latency problems in traditional pipeline architectures.
[0043] Furthermore, the end-to-end speech generation module adopts a GLM-4-Voice model architecture, including a SpeechTokenizer module, an LLM module, and a Speech Decoder module. The Speech Tokenizer module converts the input raw speech waveform into a series of discrete, semantically rich speech token sequences. This is the first step in achieving end-to-end processing and avoids the error accumulation of traditional ASR. This tokenizer is based on the intermediate layer features of a powerful ASR model and is quantized using FSQ technology to generate discrete tokens.
[0044] The LLM module, as the core of the system, receives speech token sequences from the speech tokenizer. The system integrates an Adapter module. During the LLM decoding process, the Adapter dynamically adjusts the output of the attention mechanism, making the LLM more inclined to use words and sentence structures that conform to the target emotional style when generating semantic tokens.
[0045] The Speech Decoder module is responsible for converting the speech semantic token sequence generated by LLM into high-quality, expressive speech waveforms. It receives two inputs: 1) speech semantic tokens generated by LLM, and 2) emotion control signals from the Adapter. The emotion control signals condition the Flow Matching process and finely tune the generated Mel spectrum prosodic features.
[0046] On the other hand, the present invention also provides an end-to-end voice dialogue method based on cross-modal retrieval enhancement, comprising the following steps:
[0047] S1, User voice input, extract features from the input voice, and convert the input voice signal into a voice feature representation through a voice encoder;
[0048] S2, the speech features are represented as query vectors, and the most relevant text knowledge is retrieved directly from an external text knowledge base in a speech-to-text manner through a cross-modal retrieval mechanism;
[0049] S3 generates voice responses directly through an end-to-end dialogue model based on the most relevant text knowledge retrieved.
[0050] In this embodiment, the features extracted from the input speech in step S1 include semantic features and pragmatic features. When performing cross-modal retrieval in step S2, the semantic features and pragmatic features are incorporated.
[0051] In this embodiment, during the knowledge retrieval process in step S2, a weighted matching strategy is adopted to weight and fuse semantic similarity and pragmatic feature similarity to achieve dual intelligent retrieval of semantic and pragmatic features. When the system detects pragmatic features such as "anxiety, tension, and urging" in the user's voice, the retrieval algorithm will dynamically adjust the weights to selectively recall knowledge content with high timeliness and operational guidance, namely "quick help content," including the formula:
[0052] S core =α×Sim semantic +β×Sim pragmatic ①
[0053] α + β = 1, α ≥ β ②
[0054] [α, β] = softmax(W × E) programtic +b) ③
[0055] Where α and β are weighted fusion hyperparameters, Sim semantic Sim uses semantic similarity to reflect the importance of semantic matching. pragmatic For pragmatic similarity, reflecting the degree of influence of tone / emotion, W and b are learnable parameters, and E... programtic For pragmatic embedding, Formula ① is the weighted adjustment rule for semantic and pragmatic similarity, Formula ② is the relative relationship between α and β, and Formula ③ is the adjustment strategy for α and β.
[0056] In this embodiment, the weighted fusion hyperparameters α and β are adaptively and dynamically adjusted according to the emotional state and tone intensity of the speech. This is achieved through a weight prediction network, which takes pragmatic embedding as input, passes through a fully connected layer and Softmax normalization, and outputs semantic weight α and pragmatic weight β to achieve adaptive fusion, thus realizing a flexible retrieval mechanism that is either "semantic-driven" or "pragmatic-driven". When the user's tone is stable, semantic matching is prioritized (high α); when the user's tone shows anxiety, the pragmatic weight is increased (high β).
[0057] For example, if a user inputs via voice: "Why hasn't my order arrived yet?" (in an anxious tone), the similarity is calculated as follows:
[0058] • Semantic similarity with "order delay explanation" in the knowledge base: 0.78
[0059] • Pragmatic similarity with the entry “Order Delay Explanation”: 0.80
[0060] • Semantic similarity with “Emergency Customer Service Channel”: 0.70
[0061] However, the pragmatic features of the "emergency customer service channel" entry are closer to an anxious tone: pragmatic similarity 0.95.
[0062] Comprehensive calculation:
[0063] S core1 =0.6×0.78+0.4×0.8=0.79
[0064] S core2 =0.6×0.70+0.4×0.95=0.80
[0065] →The system returns an answer such as "Contact customer service for urgent processing".
[0066] This mechanism enables a two-layer knowledge retrieval and dynamic response strategy based on semantic understanding and pragmatic awareness. This allows the system to adaptively adjust the direction of knowledge retrieval according to the tone and emotional state of the user's voice, thereby significantly improving the naturalness, relevance, and intelligence of human-computer interaction.
[0067] In this embodiment, during the knowledge retrieval process in step S2: a speech encoder is used to process speech to obtain a compressed speech embedding vector v_audio; a text encoder is used to process text to obtain a text embedding vector v_text; v_audio and v_text are mapped to the same shared semantic vector space through a projection network, and a loss function is used for training. After training, the parameters of the speech encoder and the projection network are fixed, and a vector index is constructed to map speech and text to the same semantic space. The speech vector is used as a query to directly perform nearest neighbor search in the text vector index.
[0068] In this embodiment, after step S3, there is also step S4 memory update, which records, extracts and utilizes the user's voice features, tone habits and semantic preferences during the interaction process to achieve personalized voice style generation.
[0069] The voice dialogue system and method of this invention can achieve direct speech-to-speech mapping and generation, eliminating the error accumulation problem caused by modular architecture and improving the overall robustness of the system; it reduces system complexity by simplifying the training and deployment process through an end-to-end model, improving the system's scalability and real-time interaction capabilities, making it better suited for application scenarios such as intelligent customer service where accuracy and efficiency are extremely important; through an efficient cross-modal retrieval enhancement generation method, it can retrieve relevant domain-specific professional knowledge from a text knowledge base in real time based on the received audio, ensuring the accuracy and timeliness of response information, achieving direct, efficient, and accurate retrieval of speech-to-text knowledge, enhancing the system's cognitive ability in vertical domains, and solving the problems of limited knowledge retrieval and insufficient cross-modal capabilities. The differences and advantages of the voice dialogue system and method of this invention compared to existing voice dialogue systems are shown in the table below:
[0070]
[0071]
[0072] The above description only illustrates preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. In short, all variations made within the scope of the independent claims of the present invention are within the scope of protection of the present invention.
Claims
1. An end-to-end voice dialogue system based on cross-modal retrieval enhancement, characterized in that: It includes a voice input and feature extraction module, a cross-modal retrieval enhancement module, an end-to-end voice generation module, and a voice memory module; The speech input and feature extraction module is used to input speech signals and extract multi-dimensional features of speech, including semantic features and pragmatic features. The cross-modal retrieval enhancement module is used to incorporate semantic and pragmatic features into knowledge retrieval, perform cross-modal retrieval from speech to text, and obtain the most relevant text knowledge; The end-to-end speech generation module is used to generate personalized speech based on the most relevant text for voice interaction. The voice memory module is used to record, extract, and utilize the user's voice features, tone habits, and semantic preferences during the interaction process to achieve personalized voice style generation.
2. The end-to-end voice dialogue system based on cross-modal retrieval enhancement according to claim 1, characterized in that: The cross-modal retrieval enhancement module includes a pragmatic awareness module and a cross-modal knowledge alignment module, wherein, The pragmatic awareness module is used to integrate the semantic and pragmatic features of speech during the knowledge retrieval stage to perform cross-modal adaptive knowledge retrieval. The cross-modal knowledge alignment module includes a speech encoder, a text encoder, a projection network, and a cross-modal retrieval unit; the speech encoder is used to process speech to obtain speech embedding vectors, the text encoder is used to process text to obtain text embedding vectors; the projection network is used to map the speech embedding vectors to the text embedding vectors; and the cross-modal retrieval unit is used to perform nearest neighbor search.
3. The end-to-end voice dialogue system based on cross-modal retrieval enhancement according to claim 1, characterized in that: The end-to-end speech generation module adopts the GLM-4-Voice model architecture, which includes a Speech Tokenizer module, an LLM module, and a Speech Decoder module.
4. An end-to-end voice dialogue method based on cross-modal retrieval enhancement, characterized in that, Includes the following steps: S1, User voice input, extract features from the input voice, and convert the input voice signal into a voice feature representation through a voice encoder; S2, the speech features are represented as query vectors, and the most relevant text knowledge is retrieved directly from an external text knowledge base in a speech-to-text manner through a cross-modal retrieval mechanism; S3 generates voice responses directly through an end-to-end dialogue model based on the most relevant text knowledge retrieved.
5. The end-to-end voice dialogue method based on cross-modal retrieval enhancement according to claim 4, characterized in that: In step S1, the features extracted from the input speech include semantic features and pragmatic features. When performing cross-modal retrieval in step S2, the semantic features and pragmatic features are incorporated.
6. The end-to-end voice dialogue method based on cross-modal retrieval enhancement according to claim 5, characterized in that: In the knowledge retrieval process of step S2, a weighted matching strategy is adopted to weight and fuse semantic similarity and pragmatic feature similarity, including the following formula: S core =α×Sim semantic +β×Sim pragmatic ① α+β＝1，α≥β ② [α,β]=softmax(W×E programtic +b) ③ Where α and β are weighted fusion hyperparameters, Sim semantic Sim uses semantic similarity to reflect the importance of semantic matching. pragmatic For pragmatic similarity, reflecting the degree of influence of tone / emotion, W and b are learnable parameters, and E... programtic For pragmatic embedding, Formula ① is the weighted adjustment rule for semantic and pragmatic similarity, Formula ② is the relative relationship between α and β, and Formula ③ is the adjustment strategy for α and β.
7. The end-to-end voice dialogue method based on cross-modal retrieval enhancement according to claim 6, characterized in that: The weighted fusion hyperparameters α and β are adaptively and dynamically adjusted according to the emotional state and tone intensity of the speech. This is achieved through a weight prediction network, which takes pragmatic embedding as input and outputs semantic weights α and pragmatic weights β after passing through a fully connected layer and Softmax normalization, thus achieving adaptive fusion.
8. The end-to-end voice dialogue method based on cross-modal retrieval enhancement according to claim 6, characterized in that: In the knowledge retrieval process of step S2: a speech encoder is used to process speech to obtain a compressed speech embedding vector v_audio; a text encoder is used to process text to obtain a text embedding vector v_text; v_audio and v_text are mapped to the same shared semantic vector space through a projection network, and a loss function is used for training. After training, the parameters of the speech encoder and the projection network are fixed, and a vector index is constructed to map speech and text to the same semantic space. The speech vector is used as a query to directly perform nearest neighbor search in the text vector index.
9. The end-to-end voice dialogue method based on cross-modal retrieval enhancement according to any one of claims 4-8, characterized in that: Following step S3, step S4, memory update, is also included, which records, extracts, and utilizes the user's voice features, tone habits, and semantic preferences during the interaction process to achieve personalized voice style generation.