A speech large model question and answer method based on retrieval enhancement generation

Abstract

The application belongs to the technical field of voice interaction and natural language processing, and discloses a voice large model question and answer method based on retrieval enhancement generation. Through ASR error correction and scenario-based retrieval enhancement, the factual accuracy of the answer is improved compared with the traditional scheme, the intent understanding accuracy of multi-round dialogue is improved, and the factual error and context disconnection problems in voice question and answer are effectively solved. Through hard constraint monitoring and dynamic resource scheduling, the response delay is stably controlled within the set threshold, which is lower than the existing fusion scheme, and meets the real-time requirements of voice interaction. The preference customization and hard constraint configuration of different scenes are supported, the adaptation score in medical, vehicle-mounted and home scenes is improved compared with the traditional scheme, and the differentiated needs of professional users and ordinary users can be met. The collaborative optimization of generated text and voice output improves the naturalness score of voice answers, reduces the problem of "smooth text but harsh voice", and improves user interaction willingness.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of voice interaction and natural language processing, and in particular to a question-answering method for a large voice model based on retrieval enhancement generation. Background Technology

[0002] With the rapid development of voice interaction technology and large language models, voice large model question-answering systems have been widely used in fields such as intelligent customer service, smart homes, and in-vehicle interaction. Its core logic involves converting user voice input into text through Automatic Speech Recognition (ASR), generating text answers through a large language model, and finally outputting a voice response through Text-to-Speech (TTS), thus achieving an end-to-end question-answering closed loop of "voice input - voice output".

[0003] Retrieval-enhanced generation (RAG) technology, as a key solution to compensate for the shortcomings of large language models, such as the lag in pre-trained knowledge and the large number of factual errors, has been gradually applied to text question-answering scenarios. Its core principle is to retrieve real-time, accurate knowledge related to the question from an external knowledge base through a retrieval module before generating the answer, and then use the retrieval results as context to supplement the generation model, thereby improving the factual accuracy and timeliness of the answer. In existing technologies, some solutions attempt to combine RAG with large speech models, but most adopt a sequential and independent process of "speech-to-text → text retrieval → text generation → text-to-speech," without coordinating optimization for the specific characteristics of voice interaction.

[0004] Currently, the background technologies closest to this invention fall into two main categories: one is the traditional large-scale speech question answering system, which relies on knowledge learned in the pre-training stage for generation, without introducing external retrieval enhancement. Knowledge updates require retraining or fine-tuning, resulting in high costs and poor real-time performance; the other is a speech question answering scheme that simply integrates RAG, directly reusing the retrieval strategy of text RAG, without considering the ambiguity of speech input, contextual coherence, and the sensitivity of speech interaction to response latency, leading to low collaborative efficiency between retrieval and generation, and a poor overall question answering experience.

[0005] Existing RAG-based large-scale speech model question answering technologies have the following significant limitations:

[0006] (1) Insufficient synergy between voice, text and retrieval: In traditional solutions, ASR, retrieval, generation and TTS modules are independent of each other. The text error of ASR conversion will be directly transmitted to the retrieval stage, resulting in a decrease in retrieval relevance. At the same time, the retrieval results do not fully combine the context of voice interaction (such as the reference and intent evolution in multi-turn dialogue), and the generated answer has poor adaptability to the voice scene.

[0007] (2) Knowledge retrieval is lacking in the speech domain: existing RAG retrieval strategies are mostly designed for pure text questions, without considering the colloquial and fragmented characteristics of speech questions and answers, making it difficult to accurately extract the core search terms after speech is converted to text.

[0008] (3) Lack of collaborative optimization during speech generation: The generation module did not make full use of the structured information of the search results, resulting in unnatural knowledge integration in the answer; and the TTS conversion did not match the tone and rhythm of the generated content, resulting in the answer text being fluent but the speech output being stiff, which affected the user interaction experience. Summary of the Invention

[0009] The purpose of this invention is to propose a large-scale speech question-answering method based on retrieval enhancement, addressing issues such as insufficient collaboration, poor scene adaptability, lack of personalization, and inadequate fulfillment of hard constraints in existing technologies. Specific objectives include: 1) proposing a two-stage error correction method for speech-to-text conversion to reduce the impact of errors on retrieval and generation; 2) designing retrieval strategies adapted to speech scenarios to improve the relevance of retrieval for conversational questions and the coherence of multi-turn dialogues; 3) enhancing the adaptability of generated content to speech output, ensuring accurate, natural, and efficient answers, and providing a reliable solution for speech question answering in different scenarios.

[0010] To address the shortcomings of existing methods, this invention proposes a large-scale speech model question-answering method based on retrieval enhancement, aiming to solve the problems of low retrieval accuracy and stiff audio responses in existing large-scale speech model question-answering methods.

[0011] The technical solution of this invention is as follows: A question-answering method for a large speech model based on retrieval enhancement generation, comprising the following steps:

[0012] Step 1: Initialize parameter configuration, set user preference scenarios, hardware constraints, retrieval knowledge base, speech recognition model, and speech synthesis model;

[0013] Step 2: Acquire voice input through a microphone device and perform noise suppression;

[0014] Step 3: Employ ASR and error correction to achieve speech recognition and post-correction of the recognized text;

[0015] Step 4: Based on the text modified in Step 3, execute a multi-stage retrieval process;

[0016] Step 5: The large language model combines the modified text from Step 3 with the candidate documents retrieved in Step 4 to optimize and generate a more reasonable answer;

[0017] Step 6: Tone tag generation is the tone tag of the answer output synchronously by the language model in Step 5. Then, the speech synthesis model adjusts the parameters according to the tone tag, and finally supports the user to adjust in real time, outputting a natural and fluent speech response.

[0018] Step 7: Adjust the parameters in Step 1 based on user feedback;

[0019] Step 8: Iteration Termination Check Determine if one of the following termination conditions is met:

[0020] (1) Reach the maximum number of iterations;

[0021] (2) The retrieval relevance and answer accuracy of two consecutive rounds of question answering remained stable and met the standards;

[0022] (3) The user explicitly states "satisfied" or voluntarily terminates the transaction;

[0023] If satisfied, output the final voice answer and the corresponding text record; if not satisfied, return to step 3 to continue optimizing the retrieval and generation process.

[0024] Step 3 specifically involves:

[0025] Step 3-1: Use a pre-trained ASR model to identify the denoised audio as the initial text;

[0026] Step 3-2: Based on the large language model, determine whether there are words in the initial text whose semantic similarity to the knowledge base meets the requirements. If they meet the requirements, modify them according to the knowledge base; otherwise, retain the initial text.

[0027] Step 3-3: Use the named entity recognition model to extract the entity nouns contained in the above corpus, and use the large language model to perform semantic matching between the entity nouns and the text modified in Step 3-2. Replace the segments with the ones that meet the similarity requirements, and keep the ones that do not.

[0028] Steps 3-4: Use the BERT-BiLSTM model to extract core information from the corrected question text, providing accurate input for the retrieval process.

[0029] Step 3-2 specifically involves splitting the initial text using a pre-trained word segmenter to obtain a set of words. Extract core semantic units from the knowledge base and standardize them to form a set. ; through semantic embedding models and Mapped to dimensional vector , Calculate the cosine similarity between the two and take the maximum value as the cosine similarity. Final similarity with the knowledge base; setting a threshold. ,satisfy Then use the optimal semantic unit replace Otherwise, keep it; combine all modified parts. Get the corrected text .

[0030] Step 3-3 specifically involves: processing the above-mentioned corpus... Use the NER model to extract entities and remove duplicates to obtain a set. ;Will Obtain a collection of fragments by semantic segmentation. ; computed through semantic embedding model The vectors are averaged to obtain the aggregate vector. Simultaneously calculate each segment vector Set threshold ,satisfy Then, standardized fragments are generated using a large language model. Replace if necessary, otherwise keep; concatenate all fragments to obtain the final text. .

[0031] Step 4 specifically involves:

[0032] Step 4-1: Perform knowledge-based retrieval, use a pre-trained speech-semantic embedding model to convert audio into retrieval vectors, and recall Top-K relevant documents from the knowledge base based on vector similarity;

[0033] Step 4-2: Perform conversational retrieval. Use the TD-IDF algorithm to retrieve Top-M related documents by combining audio feature vectors with conversational keywords.

[0034] Step 4-3: Merge the search results from Step 4-1 and Step 4-2, and use a weighted voting method to obtain the Top-N candidate documents; N is less than M and K.

[0035] Step 4-1 specifically involves: the knowledge-based retrieval takes the audio corresponding to the text optimized in step 3 as input, loads a pre-trained speech-semantic embedding model, and maps the audio signal into a high-dimensional retrieval vector. Preprocess all documents in the knowledge base, and generate semantic vectors for each document using the same embedding model. Build a document vector library The value range is the number of documents; calculate the cosine similarity between the high-dimensional retrieval vector and each document vector, sort them in descending order of similarity, and select the top K documents as the knowledge-based retrieval result set. }

[0036] Step 4-2 specifically involves: the colloquial retrieval process extracting a set of colloquial keywords from the optimized text in step 3. By combining spoken language semantic features extracted from audio feature vectors, a keyword-feature fusion matrix is ​​constructed; the term frequency-inverse document frequency weights of each keyword are calculated for each document in the knowledge base, generating a term frequency-inverse document frequency vector for each document. ;according to Sort in descending order and select the top M documents as the conversational search results set. }

[0037] Step 4-3 specifically involves: merging search results... and By removing duplicate documents from the list, a set of candidate documents is obtained. ; Assign dynamic weights to the two search results: knowledge-based search weights Colloquial search weight = 0.4; The formula for the overall document score is as follows:

[0038]

[0039] Is document D in The ranking order in the middle, Is document D in The ranking order. By Sort the documents in descending order and select the top N documents as the final candidate document set. }, It is the final ranking of candidate documents.

[0040] The beneficial effects of this invention: The large-scale speech model question answering method based on retrieval enhancement proposed in this invention has achieved multi-dimensional breakthroughs compared with existing technologies. Through ASR error correction and contextualized retrieval enhancement, the factual accuracy of answers is improved by 18%-25% compared to traditional solutions, and the intent understanding accuracy in multi-turn dialogues is improved by more than 20%, effectively solving the problems of factual errors and contextual disconnect in voice question answering. Through hard constraint monitoring and dynamic resource scheduling, the response latency is stably controlled within a set threshold (e.g., ≤1.5s in in-vehicle scenarios), which is 20%-30% lower than existing fusion solutions, meeting the real-time requirements of voice interaction. It supports customized preferences and hard constraint configuration for different scenarios, and the adaptation score in scenarios such as medical, in-vehicle, and home is improved by 30% compared to traditional solutions, meeting the differentiated needs of professional and ordinary users. The collaborative optimization of generated text and voice output improves the naturalness score of voice answers by 25%, reducing the problem of "fluent text but stiff voice" and increasing user interaction willingness. It supports multiple deployment environments such as cloud, edge, and terminal. Through hard constraint control and lightweight strategies, the memory usage on edge devices is reduced by 30% while ensuring core performance, and it has broad prospects for industrial application. Attached Figure Description

[0041] Figure 1 is a flowchart of the large speech model question answering method based on retrieval enhancement generation according to an embodiment of the present invention;

[0042] Figure 2 is a flowchart of the ASR error correction process according to an embodiment of the present invention;

[0043] Figure 3 is a flowchart of the hybrid retrieval process according to an embodiment of the present invention. Detailed Implementation

[0044] An embodiment of the present invention will be further described below with reference to the accompanying drawings.

[0045] Step 1: Initialize parameter configuration

[0046] Three types of core parameters are set, and the parameter values ​​can be customized according to the needs of the scenario:

[0047] (1) User preference scenario parameters, scenario type: in-vehicle / home / medical consultation / general; performance weight: answer accuracy weight (e.g., medical scenario = 0.8, in-vehicle scenario = 0.5), response latency weight (e.g., in-vehicle scenario = 0.5, home scenario = 0.3); output style: conversational (home scenario) / professional (medical scenario), speech rate (slow / medium / fast), tone (flat / friendly).

[0048] (2) Hardware constraints: Response latency: ≤1.5s for vehicle scenarios, ≤2s for home scenarios; Resource constraints: GPU memory usage ≤8GB (edge ​​devices), CPU utilization ≤60%; Error tolerance: ASR conversion error rate ≤5%.

[0049] (3) System core configuration: Retrieval knowledge base: general knowledge base (a subset of Wikipedia, updated once a week) + domain knowledge base (such as medical knowledge base, updated once a day); RAG parameters: retrieval Top-K value (default 5, adjustable), context window size (default 10 rounds of dialogue); Speech processing parameters: ASR model (Whisper Large-v3), TTS model (Coqui TTS); Iteration parameters: interaction feedback interval (default 2 rounds of question and answer), maximum number of iterations (default 5 times).

[0050] The speech recognition model used Whisper Large-v3, and the speech synthesis model used Coqui TTS.

[0051] Step 2: Voice Input Processing and Optimization

[0052] Voice acquisition and preprocessing: User voice input is acquired through a microphone device, and an adaptive noise suppression algorithm (such as WebRTC noise reduction algorithm) is used to remove environmental noise (such as vehicle noise and human voice interference) and output a clean voice signal.

[0053] Step 3: Employing ASR and error correction to achieve speech recognition and post-correction of the recognized text: First, a pre-trained ASR model (Whisper Large-v3) is used to convert clean speech into initial text. Correction is achieved through a dual mechanism of "semantic verification + domain dictionary". Semantic verification combines historical dialogue text (e.g., the previous round mentioned "hypertension") to determine if there are semantic conflicts in the initial text (e.g., "high blood pressure" and "hypertension" have a 92% semantic correlation, which is considered an error). The Unbabel COMET model is used here to extract semantic conflicts. The domain dictionary is called from the corresponding domain dictionary for the scenario (e.g., a medical dictionary contains "hypertension" and "hyperglycemic drugs") to replace incorrect words, and finally outputs accurate question text and dialogue context sequence.

[0054] Structured question extraction: The BERT-BiLSTM model is used to extract core information from the corrected question text: Search keywords: For example, for the question "What should elderly people with hypertension pay attention to in their diet?", the keywords are extracted as "dietary precautions for elderly people with hypertension"; Intent tags: Classified as "fact query - dietary advice", providing accurate input for the retrieval process.

[0055] Step 4: Based on the text modified in Step 3, execute a multi-stage retrieval process;

[0056] (1) Context-aware hybrid retrieval: Vector generation is used to fuse the question keywords with the historical dialogue context (such as "the elderly are 65 years old" in the previous round). The retrieval vector is generated by the pre-trained speech semantic embedding model SONAR for the next hybrid retrieval. The Qwen2-1.5B-Instruction model is used to extract colloquial keywords. In the hybrid retrieval, in dense retrieval, the Top-20 relevant documents are recalled from the knowledge base based on vector similarity (focusing on semantic matching to solve the problem of colloquial ambiguity); in sparse retrieval, the Top-15 relevant documents are recalled based on the TF-IDF algorithm and combined with colloquial keyword expansion (such as "diet" expanded to "food taboo recipes") (focusing on precise keyword matching). Finally, the results are fused after obtaining the two retrieval results: the weighted voting method (dense retrieval weight = 0.6, sparse retrieval weight = 0.4) is used to merge the recall results to obtain the Top-K (default 5) candidate documents.

[0057] (2) Multi-round dialogue memory update: This step involves building a dialogue memory pool (using Redis caching) to store the following information for each round of question and answer. This includes search results, generated answers and user feedback. In subsequent searches, relevant information in the memory pool is prioritized (e.g., if the user mentions "diet" again, the previously searched "hypertension diet document" is reused directly), reducing repeated searches and reducing time consumption by 40%.

[0058] (3) Filtering and structuring of search results: Based on the response delay requirements in the hard constraints, redundant documents (such as "hypertension medication documents" that are not related to the intent tag "dietary advice") are filtered out, and 3-5 highly relevant documents are retained; Structuring is to use the TextRank algorithm to extract key sentences from the documents, combine them with the entity recognition model to generate "entity-relationship pairs", output structured knowledge fragments, and reduce the processing complexity of the generation module.

[0059] Step 5: Generate Optimization and Voice Output

[0060] (1) Enhanced Retrieval Generation: Structured knowledge fragments, dialogue context, and question text are input into the speech model, and the generation is optimized through the following mechanisms. First, attention guidance is implemented by adding "retrieval knowledge tags" to the model input layer to guide the model to prioritize structured knowledge fragments and avoid knowledge omissions; then, style adaptation is implemented by adjusting the generation logic according to scene preference parameters.

[0061] Hard constraint verification: The generation time is calculated in real time by the time consumption monitoring module. If the timeout occurs, a lightweight generation strategy is activated: simplifying the knowledge integration process; reducing the model inference accuracy; and finally outputting the answer text that conforms to the preferences and satisfies the delay constraint.

[0062] Step 6: TTS Adaptation and Conversion: First, the tone tag generation module synchronously outputs the tone tags of the answer. Then, the speech synthesis module adjusts the parameters according to the tone tags. Finally, it supports real-time adjustment by the user to output a natural and fluent speech response.

[0063] Step 7: User Interaction and Dynamic Adjustment: When the number of question-and-answer rounds reaches the set interaction feedback interval (default 2 rounds), user interaction is triggered:

[0064] (1) Feedback collection: The system provides four types of feedback options through voice prompts (such as "Do you need more detailed suggestions?") or text pop-ups: Precision adjustment, delay adjustment, style adjustment: supplementary requirements.

[0065] (2) Parameter update: The system dynamically adjusts parameters based on user feedback. Feedback for "more accurate": The answer accuracy weight is increased from 0.5 to 0.8, and the Top-K search is increased from 5 to 8; Feedback for "faster response": The quantization accuracy of the generated model is reduced from FP16 to INT8, and the Top-K search is reduced from 5 to 3;

[0066] Step 8: Iteration Termination Judgment

[0067] Determine if one of the following termination conditions is met:

[0068] (1) Reach the maximum number of iterations (default 5 times);

[0069] (2) The retrieval relevance (≥85%) and answer accuracy (≥90%) of two consecutive rounds of question answering consistently meet the standards;

[0070] (3) The user explicitly states "satisfied" or voluntarily terminates the transaction;

[0071] If satisfied, output the final voice answer and the corresponding text record (including the search source for easy tracing); if not satisfied, return to step 3 to continue optimizing the search and generation process.

[0072] Experimental environment: The hardware platform includes a server equipped with an NVIDIA RTX 4090 GPU and a high-definition noise-canceling microphone; the software environment includes an Ubuntu 22.04 operating system, Python 3.9 programming language, PyTorch 2.4 deep learning framework, Whisper Large-v3 for ASR model, Coqui TTS for TTS model, Llama 3-8B for large language model, and the retrieval knowledge base includes a general knowledge base (a subset of Wikipedia) and a professional domain knowledge base (medical question-answering dataset);

[0073] Initial parameter configuration: The scenario type is set to medical consultation scenario, the answer accuracy weight = 0.8, the response latency preference ≤ 2s, the hard constraints are video memory usage ≤ 10GB, response latency ≤ 2s, search Top-K = 5, interaction feedback interval = 2 rounds, maximum number of iterations = 5;

[0074] Execution Flow: The user inputs a voice question via microphone: "What should patients with hypertension pay attention to in their daily diet?". After noise reduction preprocessing, the system converts the text using a Whisper model and corrects potential ASR errors using a medical domain dictionary (e.g., correcting "gao xue ya" to "hypertension"). The system extracts core search keywords "hypertension," "daily diet," and "precautions," and generates a search vector by combining this with the empty historical dialogue context. A hybrid search strategy retrieves 5 relevant documents from the medical knowledge base, which are then filtered to obtain 3 structured knowledge fragments (e.g., "limit sodium intake to ≤5g per day" and "eat more potassium-rich foods"). These knowledge fragments and the question text are input into the Llama 3 model to generate a medically professional and conversational answer text. The generation time is verified to be 0.8s, meeting the latency constraint. The TTS module adjusts the speech rate to a moderate level and maintains a stable tone based on the answer's descriptive tone, generating the voice output. The system triggers user feedback; the user inputs "need more detailed food recommendations." The system updates its preference parameters (answer accuracy weight = 0.9), returns to the search stage to supplement the search with "recommended foods for hypertension," regenerates the answer, and outputs it.

[0075] Experimental results: The method of this invention achieves an ASR error correction rate of 85%, improves retrieval relevance by 23%, stabilizes response latency within 1.5s, improves answer accuracy by 18% compared to traditional methods, and achieves a user satisfaction rate of 92%.

[0076] This invention is applicable to various voice Q&A scenarios such as intelligent customer service, in-vehicle interaction, home control, and professional field consultation. It supports multiple deployment environments such as cloud servers, edge computing devices, and mobile terminals, and has good scenario migration capabilities and scalability.

[0077] In summary, the innovative design of the voice-text error correction mechanism and the semantic sharing layer between modules solves the problems of error propagation and insufficient collaboration in traditional solutions, thereby improving the coherence and accuracy of the overall question-and-answer process.

[0078] This paper proposes a context-aware hybrid retrieval system that adapts to the conversational and fragmented characteristics of voice question answering, ensuring the relevance of retrieval results and the coherence of multi-turn dialogues. It achieves synchronous optimization of generated text and TTS parameters, enhancing the naturalness of speech output and improving the user's voice interaction experience through tone tag delivery and semantic sharing. It is independent of specific ASR, large models, or knowledge base types, supports the configuration of preference parameters and knowledge base expansion for new scenarios, and can be flexibly applied to voice question answering needs in both general and professional fields.

Claims

1. A question-answering method for a large speech model based on retrieval enhancement generation, characterized in that, The steps include the following: Step 1: Initialize parameter configuration, set user preference scenarios, hardware constraints, retrieval knowledge base, speech recognition model, and speech synthesis model; Step 2: Acquire voice input through a microphone device and perform noise suppression; Step 3: Employ ASR and error correction to achieve speech recognition and post-correction of the recognized text; Step 3 specifically involves: Step 3-1: Use a pre-trained ASR model to identify the denoised audio as the initial text; Step 3-2: Based on the large language model, determine whether there are words in the initial text whose semantic similarity to the knowledge base meets the requirements. If they meet the requirements, modify them according to the knowledge base; otherwise, retain the initial text. Step 3-2 specifically involves splitting the initial text using a pre-trained word segmenter to obtain a set of words. Extract core semantic units from the knowledge base and standardize them to form a set. ; Through semantic embedding model and Mapped to dimensional vector , Calculate the cosine similarity between the two and take the maximum value as the cosine similarity. Final similarity with the knowledge base; setting a threshold. ,satisfy Then use the optimal semantic unit replace Otherwise, keep it; combine all modified parts. Get the corrected text ; Step 3-3: Use the named entity recognition model to extract the entity nouns contained in the above corpus, and use the large language model to perform semantic matching between the entity nouns and the text modified in Step 3-2. Replace the segments with the ones that meet the similarity requirements, and keep the ones that do not. Steps 3-4: Use the BERT-BiLSTM model to extract core information from the corrected question text, providing accurate input for the retrieval process; Step 4: Based on the text modified in Step 3, execute a multi-stage retrieval process; Step 5: The large language model combines the modified text from Step 3 with the candidate documents retrieved in Step 4 to optimize and generate a more reasonable answer; Step 6: Tone tag generation is the tone tag of the answer output synchronously by the language model in Step 5. Then, the speech synthesis model adjusts the parameters according to the tone tag, and finally supports the user to adjust in real time, outputting a natural and fluent speech response. Step 7: Adjust the parameters in Step 1 based on user feedback; Step 8: Iteration termination check, determine if one of the following termination conditions is met: (1) Reach the maximum number of iterations; (2) The retrieval relevance and answer accuracy of two consecutive rounds of question answering remained stable and met the standards; (3) The user explicitly states "satisfied" or voluntarily terminates the service; If satisfied, output the final voice answer and the corresponding text record; if not satisfied, return to step 3 to continue optimizing the retrieval and generation process.

2. The large-scale speech model question-answering method based on retrieval enhancement generation according to claim 1, characterized in that, Step 4 specifically involves: Step 4-1: Perform knowledge-based retrieval. Use a pre-trained speech-semantic embedding model to convert audio into retrieval vectors, and recall the Top-K relevant documents from the knowledge base based on vector similarity. Step 4-2: Perform conversational retrieval. Use the TD-IDF algorithm to retrieve the Top-M relevant documents by combining the audio feature vector with the conversational keywords. Step 4-3: Merge the search results from Step 4-1 and Step 4-2, and use a weighted voting method to obtain the Top-N candidate documents; N is less than M and K.

3. The large-scale speech model question-answering method based on retrieval enhancement generation according to claim 2, characterized in that, Step 4-1 specifically involves: the knowledge-based retrieval takes the audio corresponding to the text optimized in step 3 as input, loads a pre-trained speech semantic embedding model, and maps the audio signal into a high-dimensional retrieval vector; preprocesses all documents in the knowledge base, generates semantic vectors for each document through the same embedding model, and constructs a document vector library; Calculate the cosine similarity between the high-dimensional retrieval vector and each document vector, sort them in descending order of similarity, and select the top K documents as the knowledge-based retrieval result set. } 4. The large-scale speech model question-answering method based on retrieval enhancement generation according to claim 3, characterized in that, Step 4-2 specifically involves: The colloquial retrieval process extracts a set of colloquial keywords from the optimized text in step 3, combines this with the colloquial semantic features extracted from the audio feature vector, and constructs a keyword-feature fusion matrix; it then calculates the term frequency-inverse document frequency weights for each keyword in the knowledge base documents, generating a term frequency-inverse document frequency vector for each document. ;according to Sort in descending order and select the top M documents as the conversational search results set. } 5. The large-scale speech model question-answering method based on retrieval enhancement generation according to claim 4, characterized in that, Step 4-3 specifically involves: merging search results... and By removing duplicate documents from the list, a set of candidate documents is obtained. ; Assigning dynamic weights to the two search results: knowledge-based search weights Colloquial search weight = 0.4; The formula for the overall document score is as follows: Is document D in The ranking order in the middle, Is document D in The ranking order in the middle; by Sort the documents in descending order and select the top N documents as the final candidate document set. }, It is the final ranking of candidate documents.

Images