Voice question answering method, system, medium and product based on retrieval-enhanced generation

Abstract

The present invention provides a voice question-answering method, system, medium, and product based on retrieval-enhanced generation. The method comprises: obtaining an audio file to be processed and an original question corresponding to the audio file to be processed; transcribing the audio file to be processed using a preset ASR model to obtain a text transcription result; inputting the text transcription result, the original question, and a first preset prompt word into a first large language model to obtain a key question corresponding to the audio file to be processed, output by the first large language model; the first preset prompt word is used to prompt the first large language model to generate a corresponding key question based on the text transcription result and the original question; based on retrieval-enhanced generation technology, searching for the key question by accessing an external search engine to obtain multiple semantically related paragraphs; and inputting the multiple semantically related paragraphs, the key question, and a second preset prompt word into a second large language model to obtain a target answer. The present invention can improve the processing effect of voice question-answering tasks and facilitate deployment and maintenance.

Description

Technical Field

[0001] The present invention relates to the field of voice question answering technology, and in particular to a voice question answering method, system, medium and product based on retrieval-enhanced generation. Background Art

[0002] Traditional text-based question answering (Text QA) focuses solely on understanding and extracting answers from clean text input. Spoken Question Answering (SQA) aims to integrate speech and text modalities, enabling models to use speech input to perform question answering tasks in more natural and realistic scenarios. Compared to Text QA, SQA, by incorporating raw speech signals, is closer to real-world applications, providing greater practical value and research significance. Therefore, the development of SQA contributes to the advancement of multimodal information processing systems and improves speech understanding, reasoning, and real-time response capabilities in spoken question answering.

[0003] In recent years, numerous solutions have been developed to improve the performance of SQA tasks. For example, SpeechBERT jointly pre-trains speech and text representations to reduce the impact of Automatic Speech Recognition (ASR) errors; DUAL discretizes speech into a sequence of units, enabling SQA without paired speech and text data, bypassing ASR entirely; MRD-Net introduces multimodal residual knowledge distillation, leveraging speech-aware attention mechanisms to learn robust features and mitigate ASR noise; GhostT5 proposes a low-cost method for generating efficient latent representations, facilitating text-free spoken question answering. SpeechDPR constructs an unsupervised dense retriever to perform semantic-level speech retrieval without transcription. While these approaches have achieved significant progress, they primarily focus on modeling speech content, overlooking a key aspect of real-world SQA: user questions are often based on more than just the speech itself. In many cases, answering questions requires not only understanding the spoken content but also integrating external knowledge. Some questions are time-sensitive, and the answers may change dynamically. However, existing methods rely heavily on static datasets, emphasize shallow understanding of speech, and lack mechanisms to dynamically incorporate external or real-time information. This limits their ability to handle complex, knowledge-intensive, and time-dependent problems.

[0004] For example, if only questions about the audio content and a piece of audio are input into a large language model (LLM), the large language model can understand the audio content and answer simple questions to obtain the target answer. However, if the question requires external knowledge or is time-sensitive, the large language model cannot answer it well.

[0005] Another existing solution is an open-domain natural language inference answering system driven by a large language model. This system uses a question rewriting module to rewrite user questions, merges internet search text obtained through a search engine API with local text data search results, and generates a final query result. This result, as evidence candidate, is directly input into the downstream question-answering model for reasoning and answer generation. However, this solution does not involve audio modality, and the main search method still relies on manually constructed knowledge bases and the use of commonly used traditional query methods (BM25, SQL queries, knowledge graphs, etc.). Information is still solidified offline and relies on continuous expert re-entry, error correction, and disambiguation. The use of knowledge graphs and SQL templates is very labor-intensive, and its complexity limits the expansion of the field. After obtaining the search content, it still needs to go through a lot of cleaning, re-sorting, etc. In addition, although the web search method using API is open, it is still mixed with a variety of traditional search methods. The overall search process is very complex and has high latency. It is difficult to track the specific search results of each step of each search method, as well as the contribution of the original search results to the answer. Moreover, each search method basically includes the training of retrievers, generators, etc., and also requires a lot of resources in the fusion module.

[0006] Another existing solution is a multi-engine intelligent question-answering system for multi-type knowledge bases. This system uses a speech recognition model to transcribe spoken questions in a specific language into Chinese natural language descriptions. Although this solution introduces an audio module, the audio content remains the subject itself and does not include any other content, thus eliminating the need for additional potentially interfering content. Furthermore, the text form of the question is also input along with the audio. Therefore, its focus remains on processing the text modality and fails to reflect the necessity of the speech modality, making it difficult to use in SQA tasks.

[0007] In addition, existing datasets for SQA tasks also show shortcomings, which limit their application in real-world scenarios. They mainly face two limitations: (1) Lack of real semantics: Many widely used SQA datasets rely on synthetic speech generated by text-to-speech (TTS) systems. Although the quality of TTS has improved significantly, it still cannot capture the diversity and complexity of speech. (2) Limited question complexity: Most existing questions can be answered directly from the speech content and rarely involve real-time or knowledge-intensive reasoning. However, real-world SQA usually requires common sense reasoning, context understanding, and domain expertise, which are mostly missing in current datasets. The above limitations make existing SQA benchmarks unable to fully evaluate key real-world capabilities such as knowledge enhancement and temporal reasoning. Summary of the Invention

[0008] The first object of the present invention is to provide a voice question answering method based on retrieval enhancement generation.

[0009] The second object of the present invention is to provide a voice question answering system based on retrieval enhancement generation.

[0010] The third object of the present invention is to provide a computer-readable storage medium for implementing the above-mentioned voice question answering method based on retrieval enhancement generation.

[0011] A fourth object of the present invention is to provide a computer program product for implementing the above-mentioned voice question-answering method based on retrieval enhancement generation.

[0012] In order to achieve the above-mentioned first purpose, the present invention provides a voice question-answering method based on retrieval-enhanced generation, which includes the following steps: obtaining audio to be processed and an original question corresponding to the audio to be processed; the original question corresponding to the audio to be processed is a question that requires understanding of the audio to be processed; transcribing the audio to be processed through a preset ASR model to obtain a text transcription result; inputting the text transcription result, the original question corresponding to the audio to be processed, and a first preset prompt word into a first large language model to obtain a key question corresponding to the audio to be processed output by the first large language model; the first preset prompt word is used to prompt the first large language model to generate a key question corresponding to the audio to be processed based on the text transcription result and the original question corresponding to the audio to be processed; the key question corresponding to the audio to be processed is a question that does not require understanding of the audio to be processed; based on retrieval-enhanced generation technology, searching the key question corresponding to the audio to be processed by accessing an external search engine to obtain multiple semantically related paragraphs; inputting the multiple semantically related paragraphs, the key question corresponding to the audio to be processed, and the second preset prompt word into the second large language model to obtain the target answer to the key question corresponding to the audio to be processed output by the second large language model.

[0013] As can be seen from the above scheme, the present invention utilizes the language understanding and knowledge integration capabilities of a large language model to generate corresponding key questions based on the audio to be processed and the original questions corresponding to the audio to be processed, and further introduces retrieval enhancement generation technology, and performs retrieval by using the key questions obtained by reasoning, dynamically captures the latest web page paragraphs and then pushes them to the large language model for processing, and generates the final answer through the integration of the large language model. The present invention can answer more complex and event-sensitive questions, and improves the processing effect of voice question-answering tasks. In addition, the present invention is easier to maintain and deploy, and can be used without training, and the replacement of models in the module is also very convenient, and can be replaced with the current best-performing model at any time. Compared with existing solutions, the retrieval of the present invention is more lightweight, fast, and highly real-time, the retrieval content can be extended to any field, and the retrieval content is highly visualized, which is convenient for tracing.

[0014] A further solution is that the first language model and the second language model are the same large language model.

[0015] A further solution is that the ASR model is a Paraformer-zh model.

[0016] A further solution is that the first language model and the second language model are GLM-4-Plus models.

[0017] A further solution is to determine the first and second largest language models to be selected based on the evaluation of multiple large language models on a pre-constructed voice question and answer dataset; the construction process of the voice question and answer dataset includes: obtaining voice segments of preset length for each category; obtaining the original question, key question, and question answer corresponding to each voice segment; wherein, in each voice segment, the corresponding original question and key question are real-time or non-real-time, and the key question is obtained based on the original question and voice segment; for each voice segment, the voice segment and the original question, key question, and question answer corresponding to the voice segment are stored as a data set sample to obtain a voice question and answer dataset.

[0018] A further solution is that the voice clip is obtained by intercepting a video from an online website, and the voice clip includes the speaker's accent and background noise.

[0019] It can be seen from this that we can select a suitable large language model based on the constructed voice question answering dataset and facilitate the evaluation of the processing effect of the SQA task.

[0020] A further solution is that, in a preset proportion of dataset samples of the voice question-answering dataset, at least one network search is required to obtain corresponding answers to questions based on key questions in the dataset samples.

[0021] This shows that it can ensure that for most of the dataset samples in the voice question answering dataset, at least one network search is required in the process of obtaining answers to key questions, thereby improving the evaluation effect of the voice question answering dataset.

[0022] In order to achieve the above-mentioned second purpose, the present invention provides a voice question-answering system based on retrieval enhancement generation, which includes: an input acquisition module, a text transcription module, a large language model module, and a retrieval enhancement generation module; the original question corresponding to the audio to be processed is a question that requires the understanding of the audio to be processed; the input acquisition module is used to obtain the audio to be processed and the original question corresponding to the audio to be processed; the text transcription module is used to transcribe the audio to be processed through a preset ASR model to obtain a text transcription result; the large language model module is used to input the text transcription result, the original question and the first preset prompt word into the first large language model, and obtain the output of the first large language model corresponding to the audio to be processed. The first preset prompt word is used to prompt the first large language model to generate the key question corresponding to the audio to be processed based on the text transcription result and the original question corresponding to the audio to be processed; the key question corresponding to the audio to be processed is a question that does not rely on the understanding of the audio to be processed; the retrieval enhancement generation module is used to search the key question corresponding to the audio to be processed by accessing an external search engine based on the retrieval enhancement generation technology, and obtain multiple semantically related paragraphs; the large language model module is also used to input multiple semantically related paragraphs, key questions, and the second preset prompt word into the second large language model, and obtain the second large language model outputting the target answer to the key question corresponding to the audio to be processed.

[0023] In order to achieve the third purpose mentioned above, the present invention provides a computer-readable storage medium on which a computer program is stored, wherein: when the computer program is executed by a processor, the above-mentioned voice question-answering method based on retrieval enhancement generation is implemented.

[0024] In order to achieve the fourth purpose mentioned above, the present invention provides a computer program product, including computer instructions, wherein: when the computer instructions are executed by a processor, the above-mentioned voice question answering method based on retrieval enhancement generation is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a system framework diagram of an embodiment of a voice question-answering system based on retrieval enhancement generation according to the present invention.

[0026] Figure 2 It is a flow chart of an embodiment of the voice question answering method based on retrieval enhancement generation of the present invention.

[0027] Figure 3 It is a schematic diagram of an embodiment of the voice question answering method based on retrieval enhancement generation of the present invention.

[0028] Figure 4 2 is a schematic diagram of the construction process of the OKE-SQA dataset in an embodiment of the voice question answering method based on retrieval enhancement generation of the present invention.

[0029] Figure 5This is a schematic diagram of the categories, question ratios, and examples of questions in each category of the OKE-SQA dataset in an embodiment of the voice question answering method based on retrieval enhancement generation of the present invention.

[0030] Figure 6 Schematic diagram of the score distribution of different categories of real-time and non-real-time questions in the OKE-SQA dataset in an embodiment of the voice question answering method based on retrieval enhancement generation of the present invention.

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0032] The voice question-answering system based on retrieval enhancement generation of the present invention can understand the audio content and answer the questions only about the audio content when it obtains audio content and questions only about the audio content, and output the corresponding target answers; when it obtains audio content and questions that require external knowledge or are time-sensitive, it can answer the questions through the detected additional knowledge and output the corresponding target answers.

[0033] Example of a voice question answering system based on retrieval enhancement generation:

[0034] See also Figure 1 The voice question answering system based on retrieval enhancement generation of this embodiment includes an input acquisition module 11, a text transcription module 12, a large language model module 13, and a retrieval enhancement generation module 14. The input acquisition module 11 is connected to the text transcription module 12 and the large language model module 13 respectively, the text transcription module 12 is connected to the large language model module 13, and the large language model module 13 is connected to the retrieval enhancement generation module 14.

[0035] The input acquisition module 11 is used to acquire the audio to be processed and the original question corresponding to the audio to be processed. The original question corresponding to the audio to be processed is a question that requires understanding of the audio to be processed.

[0036] The text transcription module 12 is used to transcribe the acquired data to be processed into text based on a pre-selected ASR model to obtain a text transcription result.

[0037] The ASR model in this embodiment is preferably the Paraformer-zh model. As a non-autoregressive ASR model, Paraformer-zh efficiently converts speech to text. Compared to traditional autoregressive methods, it offers advantages such as high transcription speed, strong robustness, and reduced error propagation. In other embodiments, other ASR models may be selected based on actual needs.

[0038] The large language model module 13 is used to generate key questions corresponding to the original questions, and generate target answers corresponding to the key questions by combining multiple paragraphs semantically related to the key questions obtained from the retrieval enhancement generation module 14.

[0039] Specifically, the large language model module 13 includes a key question generation unit and a question solving unit. The key question generation unit generates a key question corresponding to the original question based on the first large language model, combining the text transcription results, the original question, and the first preset prompt word; the question solving unit generates an answer corresponding to the key question based on the second large language model, combining multiple paragraphs semantically related to the key question, the key question, and the second preset prompt word. The key question corresponding to the audio to be processed is a question that does not rely on the understanding of the audio to be processed.

[0040] The key question is a well-structured and semantically complete question generated by the first language model based on the text transcription results, the original question, and the first preset prompt word. By generating the key question, colloquial, ambiguous, or fragmented input can be converted into a standardized, search-friendly query basis, ensuring that multiple paragraphs semantically relevant to the key question can be accurately retrieved in the search enhancement generation module 14.

[0041] Optionally, in different embodiments, the first large language model and the second large language model can be different large language models, or can be the same large language model. In this embodiment, they are preferably the same large language model.

[0042] Optionally, in different embodiments, the first large language model and / or the second large language model may be directly set in the text transcription module 12, or may be an external large language model called through an API call.

[0043] The retrieval enhancement generation module 14 is based on the retrieval enhancement generation (RAG) technology and is used to search the key question by accessing an external search engine to obtain multiple paragraphs semantically related to the key question.

[0044] Embodiment of the voice question answering method based on retrieval enhancement generation:

[0045] This embodiment is based on the above embodiment of the voice question answering system based on retrieval enhancement generation. Figure 2 and Figure 3 This embodiment is implemented based on the execution of a computer program, and each step will be introduced below.

[0046] First, step S11 is performed to obtain the audio to be processed and the original question corresponding to the audio to be processed.

[0047] Then, step S12 is executed to transcribe the audio to be processed using a preset ASR model to obtain a text transcription result.

[0048] Then, step S13 is executed to input the text transcription result, the original question, and the first preset prompt word into the first large language model to obtain the key question output by the first large language model.

[0049] Then, step S14 is executed to search for the key question by accessing an external search engine to obtain multiple paragraphs semantically related to the key question.

[0050] Finally, step S15 is executed to input multiple paragraphs semantically related to the key question, the key question, and the second preset prompt word into the second largest language model to obtain the target answer corresponding to the key question output by the second largest language model.

[0051] To validate the aforementioned embodiment of the voice question answering system based on retrieval enhancement and generation and the aforementioned voice question answering method based on retrieval enhancement and generation, the following describes a detailed experimental process. Specifically, we first constructed a voice question answering dataset (hereinafter referred to as the OKE-SQA dataset) for evaluating the performance of RAGs in large language models. We then compared the OKE-SQA dataset with other approaches to SQA tasks.

[0052] The OKE-SQA dataset is specifically designed for speech tasks that combine LLM and RAG, and has the following properties: (1) Questions are not limited to the explicit content of the speech. Instead, the questions require the model to understand the speech content and then utilize external knowledge to derive answers; (2) Each question has a single, well-defined answer to facilitate reliable evaluation, and both questions and answers are concise and clear; (3) To enhance applicability in real-world scenarios, the speech is collected from real sources, so that realistic speech characteristics, including accents and background noise, can be captured.

[0053] See also Figure 4 ,The construction process of the OKE-SQA dataset includes four steps, each of which will be introduced below.

[0054] First, step S21 is executed to classify the data set.

[0055] Among them, see Figure 4, the questions are divided into six different categories, including daily life, sports and competitions, history and culture, companies and products, professional fields, and others (i.e., unclassified fields). Each category contains real-time questions (the answers change over time) and non-real-time questions (the answers do not change over time). Regardless of whether the question is real-time or non-real-time, it may require external knowledge to answer the question. Distinguishing between real-time and non-real-time questions also facilitates targeted updates to the answers to the questions in the dataset. Human annotators were instructed to maintain a roughly even ratio between these two question types.

[0056] Then, step S22 is executed: task definition and allocation.

[0057] Each manual annotator is assigned one or two categories and is accompanied by sample question prompts. Before creating a question, the manual annotator needs to determine the category of their main interest, search for online videos related to their category, and record the start and end timestamps of the online video, as well as the video link. In addition, the manual annotator needs to ensure that the length of the extracted start and end timestamps is 3 to 10 seconds, and the main topic mentioned in the question must clearly appear in the video clip. The online video corresponding to the video link should include both audio and video information. The online video can be obtained from an existing online video website. In this embodiment, all online videos are online videos.

[0058] Then, step S23 is executed to determine the original question, the answer to the question, and the key question.

[0059] Among them, the corresponding original questions, answers and key questions designed by the human annotators based on the online videos they searched are obtained, and their expressions are ensured to be clear and unambiguous. When drafting questions and answers, the human annotators need to provide an original question and a key question. The original question cannot reveal the key speech content to ensure that the model must interpret the speech clip. Instead, the key question is replaced with the key terms explicitly mentioned in the speech, making it directly applicable to the retrieval of external knowledge. For example, for an example of a real-time question, the original question is "How many major caves are there in the grotto mentioned in the audio?", and the corresponding key question is "How many major caves are there in the Yungang Grottoes?"; for an example of a non-real-time question, the original question is "How many dead skin cells does the object mentioned in the audio shed per second?", and the corresponding key question is "How many dead skin cells does the human body shed per second?".

[0060] Furthermore, a variety of expression styles can be employed in constructing original questions of different categories to avoid overly homogenous generated content and thus cover a wider range of possible scenarios. This diversity enables the LLM fine-tuned on the OKE-SQA dataset to better understand human questioning and answering styles, thereby generating output that is closer to actual user needs.

[0061] Finally, step S24 is executed to organize the data and obtain the OKE-SQA dataset.

[0062] This involves extracting a speech clip from the start and end timestamps of an online video. This speech clip, along with its corresponding original question, answer, and key question, is stored as a data sample. All acquired data samples are categorized and stored to form the OKE-SQA dataset. The OKE-SQA dataset provides a more challenging and realistic SQA environment, requiring external knowledge to effectively answer questions.

[0063] The OKE-SQA dataset is the first speech-based dataset specifically designed to evaluate the performance of RAGs in large language models. See Table 1 for statistics of the OKE-SQA dataset.

[0064] Table 1. Statistics of the OKE-SQA dataset

[0065]

[0066] The OKE-SQA dataset contains approximately 1,600 original questions, each with a corresponding key question and answer. The OKE-SQA dataset covers six categories, of which approximately 28% are real-time questions (453 questions), and the remaining 72% are non-real-time questions (1,174 questions). The OKE-SQA dataset includes a variety of categories: Daily Life (6%), Sports and Competitions (17%), History and Culture (22%), Companies and Products (18%), Professional Fields (12%), and Other (i.e., unclassified fields, which account for 25%). Question length averages 19.45 words, with the longest question reaching 45 words. Answer length averages 4.97 words, with the longest being 68 words. The total speech duration in the OKE-SQA dataset is 11,856.35 seconds. These statistics demonstrate that the dataset is well-balanced, focusing on real-time questions while ensuring sufficient coverage of all categories. The wide range in question lengths highlights the diversity of content, which is crucial for assessing LLM performance in relation to the RAG.

[0067] See also Figure 5 , Figure 5 Examples of categories and proportions in the OKE-SQA dataset and examples of questions in each category are shown.

[0068] To further assess the difficulty of the OKE-SQA dataset, we also annotated the scores of each original question by more than 20 human annotators. Each original question was scored based on the number of retrieval steps required to answer it, using the following criteria: (1) score 1: can be answered without any external knowledge; (2) score 5: requires one web search; (3) score 8: requires two web searches; (4) score 10: requires three or more web searches. See Table 2, which shows a summary of the scores of real-time and non-real-time questions in the OKE-SAQ dataset.

[0069] Table 2. Summary of all OKE-SAQ scores

[0070]

[0071] In the overall distribution of difficulty scores, 30,330 questions received a score of 5, representing 85% of the 35,794 questions, highlighting the significant emphasis placed on retrieval-based reasoning in the OKE-SQA dataset. Furthermore, 7.4% of questions received a high score (8 or 10), indicating the presence of multiple reasoning-intensive queries requiring complex information integration and reasoning.

[0072] Combined with Table 2 and Figure 6 , which shows the score distribution of each question type in different categories. Score 5 dominates all categories, confirming that the OKE-SQA dataset is mainly composed of retrieval-based questions. Importantly, Table 2 and Figure 6 The distribution of real-time and non-real-time questions at different difficulty levels is also shown. Temporal dynamics also plays a key role in task complexity. As shown in Table 2, 28% of all questions are real-time, appearing across all difficulty levels. For example, among the highest difficulty questions (scored 10), 31% are real-time, highlighting the challenges of dynamic knowledge retrieval. However, 69% of the 10-score questions are not time-dependent, indicating that static questions still require advanced domain expertise, multi-step reasoning, and comprehensive expertise. In summary, the OKE-SQA dataset presents diverse challenges in terms of retrieval difficulty, real-time dynamics, and domain diversity. Since the OKE-SQA dataset ensures that the majority of samples in the speech question answering dataset require at least one web search to obtain the answer to the key questions, it improves the evaluation of speech question answering datasets and effectively assesses the model's capabilities in temporal reasoning and knowledge integration. Therefore, as a rigorous and comprehensive benchmark, the OKE-SQA dataset helps advance the development and evaluation of speech question answering systems.

[0073] This example introduces several baseline methods for comparison. Specifically, two tasks with different objectives are defined: (1) Generate key questions. This task is used to evaluate whether the system can accurately generate "key questions". Tasks related to RAG require the model to interpret the input question and speech content, and then infer the key information required for knowledge retrieval, and refer to this retrieval-oriented query as a "key question". Whether the system can accurately generate this question is the first key indicator for measuring performance. (2) Generate target answers corresponding to key questions. This task is used to evaluate whether the model can correctly provide target answers corresponding to key questions, and the target answers need to be as close as possible to the answer to the question corresponding to the key question. In this method, the speech is first transcribed into text and then input into a large language model. The large language model uses its contextual reasoning ability to autoregressively generate answers, predicting one word at a time. For methods that do not require automatic speech recognition, the speech is directly input into the speech language model, which captures multiple layers of features such as phonemes, rhythm, and intonation to interpret the underlying semantics. The answer is then generated in the same autoregressive manner, directly completing the conversion process from speech to text.

[0074] The baselines are divided into two parts: (1) ASR-based baselines: The first category of baselines follows an ASR-based pipeline. In these methods, ASR is performed using a Paraformer model to convert the user's voice questions into text. The recognized text is then combined with predefined prompt words and input into a language model to generate answers. Specifically, the recognized text and task-specific prompts are input into the language model together. The model embeds each token (word or subword) into a high-dimensional semantic space and models contextual dependencies through multiple Transformer layers. This enables the model to understand the context, grammar, and intent of the question. Finally, the large language model generates the answer in an autoregressive manner, predicting one token at a time until a complete natural language response is formed. (2) No-ASR baselines: The second category of baselines does not use ASR. To alleviate the performance bottleneck caused by ASR errors, an end-to-end voice question answering system is adopted, which uses a speech language model Speech-LLM, which is a large language model that can directly process raw speech input. These models allow for more natural and fluent question answering by performing semantic understanding and answer generation directly from speech.

[0075] The speech language model first uses a pre-trained speech encoder to extract speech representations from the input signal, capturing intonation and pragmatic features such as speaking rate, rhythm, and pauses. These speech features are then mapped into the language modeling space. When the task includes additional textual context, the model jointly encodes this information with the speech features to achieve cross-modal semantic integration within a unified representation space. Once the speech is processed and the text prompt and question are received, the speech language model autoregressively generates the answer. This generation process is similar to that of text-based language models, but its foundation lies in the speech encoding space.

[0076] In the experimental setup, the pre-configured ASR model for this example is Paraformer-zh. For the ASR baseline, three text-only LLMs were selected: Qwen2.5-7B Instruct, GLM-Zero-Preview, and ChatGLM3-6B. For baselines that do not require ASR, Qwen2-Audio-7B-Instruct and GLM-4-Voice were selected. These two language models can handle both speech and text. For the web search used in the benchmark, the web-searchpro tool was selected.

[0077] Optionally, the GLM-4-PLUS used in the voice question answering system of this embodiment can be fine-tuned on the OKE-SQA dataset. If the GLM-4-PLUS used in the voice question answering system of this embodiment is fine-tuned on the OKE-SQA dataset, the fine-tuning is done on a portion of the OKE-SQA dataset, and the portion of the OKE-SQA dataset used in the following comparison with other baseline methods will not overlap with this portion.

[0078] In determining the evaluation metrics, four metrics were used to comprehensively evaluate the quality of the generated text: (1) BLEU: This is used to measure vocabulary accuracy by evaluating local matching through n-gram precision. (2) ROUGE-N: This emphasizes recall to measure the coverage of key content relative to the reference text, thereby evaluating content completeness. (3) F1-Recall: This balances precision and recall, reducing the bias caused by a single metric, making it very suitable for question answering tasks. (4) BERTScore: This captures deeper semantic similarity by considering lexical substitutions and syntactic changes, complementing the surface-level focus of the n-gram metric.

[0079] By combining traditional statistical methods (BLEU, ROUGE) and deep learning-based metrics (BERTScore), and incorporating the F1 balance mechanism, a more robust evaluation of both formal accuracy and semantic fidelity is performed.

[0080] See Table 3, which shows the performance of the method of this embodiment and other baseline methods in generating key questions.

[0081] Table 3. Performance of the method in this example and other baseline methods in generating key questions

[0082]

[0083] The method of this embodiment (ours) consistently achieved the best performance across all key evaluation metrics. Specifically, the F1-Recall metric achieved a 47.3% improvement over Qwen2.5-7B-Instruct, highlighting its superior ability to preserve and replace key factual elements during question rewriting. The ROUGE-1 metric saw a 21.8% improvement, reflecting enhanced lexical fidelity and surface form alignment. Compared to GLM-Zero-Preview, the BLEU metric score of this embodiment's method increased by 159.6%, from 3.2756 to 8.5040. For the F1-Recall metric, the method of this embodiment achieved a score of 37.34%, while GLM-4-Voice achieved a mere 2.14%. These significant improvements demonstrate that the method of this embodiment is not only able to identify key entities in spoken queries, but also accurately rephrase them into standard form in the rewritten question.

[0084] See Table 4, which shows the performance of the method of this embodiment and other baseline methods in generating target answers.

[0085] Table 4. Performance of the method in this example and other baseline methods in generating target answers

[0086]

[0087] Table 4 shows the results for generating target answers to key questions. Our solution (our RagSQA) achieves top performance across all evaluation metrics, particularly in terms of ground truth accuracy and content coverage. Compared to Qwen2.5-7B-Instruct, the F1-Recall metric improves by 167.1%, from 11.99% to 32.03%. The ROUGE-1 metric also improves significantly, from 6.80% to 20.49%, equivalent to a 201.3% increase, indicating greater lexical overlap with the correct answer. Compared to GLM-Zero-Preview, the BLEU metric improves by over eightfold, from 1.3300 to 12.0619. This demonstrates that RagSQA is not only factually accurate but also more fluent and grammatically consistent. GLM-4-Voice's F1-Recall metric is only 0.06%, indicating a near-inability to retrieve the correct information. In contrast, our approach achieves 32.03%, over 530 times higher than the previous approach, demonstrating a stronger ability to identify relevant content from the input.

[0088] To isolate the effect of retrieval, we compare the performance of GLM-4-Plus without RAG with RagSQA of this example. After adding RAG, the F1-Recall metric improved by 119.1%, the ROUGE-1 metric improved by 165.7%, and the BLEU metric improved by 227.6%. These results show that without retrieval, the model has difficulty handling time-sensitive or domain-specific queries, often resulting in inaccurate or incomplete answers. However, by introducing RAG, the model is able to answer high-complexity questions that require this dynamic external information. This improvement emphasizes the importance of retrieval-enhanced large language models in handling knowledge-intensive SQA tasks in the real world.

[0089] In order to intuitively compare the difficulty of the OKE-SQA dataset with existing datasets, an experiment is also conducted to evaluate the performance of the baseline model on multiple datasets, as shown in Table 5.

[0090] Table 5. Performance of different models on the OKE-SQA dataset and other baseline datasets

[0091]

[0092] Table 5 shows performance on different datasets, with lower values ​​indicating greater dataset difficulty. The performance of each baseline model on the OKE-SQA dataset drops significantly across multiple metrics, highlighting its increased complexity and the limitations of current models' generalization. While exceeding 50% on LibriSQA and Spoken-SQuAD, these metrics drop by over 70% on OKE-SQA, with some models experiencing reductions of up to 90%. This indicates serious challenges in vocabulary alignment and information matching. The degradation of the F1-Score and F1-Recall metrics is even more pronounced: models that achieve F1-Score and F1-Recall on Spoken-SQuAD drop below 1% on OKE-SQA, indicating a critical failure in both answer accuracy and recall.

[0093] Two main factors contribute to this challenge. First, OKE-SQA contains a large number of real-time questions, whose answers change dynamically over time. Second, many questions require external knowledge, raising the bar for understanding and reasoning. Compared to existing Chinese SQA datasets, OKE-SQA imposes higher requirements in terms of both time and knowledge base. It reveals the limitations of current SQA models in real-world applications and provides a new benchmark for developing more cognitively capable and knowledge-adaptive systems.

[0094] In addition, although the OKE-SQA dataset was constructed to evaluate the performance of complex tasks in the field of voice question answering that require retrieving external knowledge, the OKE-SQA dataset sets standard key questions (in text form) as a reference, so it can also be extended to tasks such as text question answering, and is not limited to the field of voice question answering.

[0095] In summary, the present invention utilizes the language understanding and knowledge integration capabilities of large language models, and further introduces retrieval enhancement generation technology. By using the key questions obtained by reasoning to perform retrieval, the latest web page paragraphs are dynamically captured and then pushed to the large language model for processing, thereby improving the processing effect of voice question answering. The retrieval of the present invention is more lightweight, fast, and highly real-time. The retrieval content can be extended to any field, and the retrieval content is highly visualized, which is convenient for tracing. The overall process of the present invention is clear, maintenance and deployment are easier, and it can be used without training. The replacement of the model in the module is also very convenient, and it can be replaced with the current best-performing model at any time.

[0096] Computer readable storage medium embodiment:

[0097] If the modules integrated into the computer device of the above embodiment are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the process of the embodiment of the voice question-answering method based on retrieval enhancement generation can also be completed by a computer program instructing the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a controller, the computer program can implement the steps of the embodiment of the voice question-answering method based on retrieval enhancement generation. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The storage medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0098] Computer program product embodiment:

[0099] The computer program product of this embodiment includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform each step of the embodiment of the voice question answering method based on retrieval enhancement generation.

[0100] Finally, it should be emphasized that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A voice question answering method based on retrieval-enhanced generation, characterized in that: The following steps are involved: Obtaining audio to be processed and an original question corresponding to the audio to be processed; the original question corresponding to the audio to be processed is a question that requires understanding of the audio to be processed; The audio to be processed is transcribed using a preset ASR model to obtain a text transcription result; Inputting the text transcription result, the original question corresponding to the audio to be processed, and a first preset prompt word into a first large language model, to obtain a key question corresponding to the audio to be processed output by the first large language model; The first preset prompt word is used to prompt the first large language model to generate a key question corresponding to the audio to be processed based on the text transcription result and the original question corresponding to the audio to be processed; The key problem corresponding to the audio to be processed is a problem that does not rely on the understanding of the audio to be processed; Based on the retrieval enhancement generation technology, the key questions corresponding to the audio to be processed are searched by accessing an external search engine to obtain multiple semantically related paragraphs; The multiple semantically related paragraphs, the key questions corresponding to the audio to be processed, and the second preset prompt words are input into the second largest language model to obtain the target answers to the key questions corresponding to the audio to be processed output by the second largest language model.

2. The voice question answering method based on retrieval enhancement generation according to claim 1, characterized in that: The first large language model and the second large language model are the same large language model.

3. The voice question answering method based on retrieval-enhanced generation according to claim 1, characterized in that: The first language model and the second language model are GLM-4-Plus models.

4. The voice question answering method based on retrieval-enhanced generation according to claim 1, wherein: The ASR model is the Paraformer-zh model.

5. The voice question answering method based on retrieval-enhanced generation according to any one of claims 1 to 4, characterized in that: Determining the first and second largest language models to be selected based on evaluations of multiple large language models on a pre-built voice question-answering dataset; The process of constructing the voice question answering dataset includes: Get speech segments of preset length for each category; Obtaining the original question, key question, and answer corresponding to each voice segment; wherein, in each voice segment, the corresponding original question and key question are real-time or non-real-time, and the key question is obtained based on the original question and the voice segment; For each of the voice segments, the voice segment and the original question, the key question, and the answer to the question corresponding to the voice segment are stored as a data set sample to obtain a voice question and answer data set.

6. The voice question answering method based on retrieval-enhanced generation according to claim 5, characterized in that: The voice clip is obtained by intercepting a video from an online website, and the voice clip includes the speaker's accent and background noise.

7. The voice question answering method based on retrieval-enhanced generation according to claim 6, characterized in that: In the dataset samples with a preset proportion of the voice question and answer dataset, at least one network search is required to obtain corresponding answers to key questions in the dataset samples.

8. A voice question answering system based on retrieval enhancement generation, characterized in that: include: Input acquisition module, text transcription module, large language model module, retrieval enhancement generation module; The input acquisition module is used to obtain the audio to be processed and the original question corresponding to the audio to be processed; the original question corresponding to the audio to be processed is a question that requires understanding of the audio to be processed; The text transcription module is used to transcribe the audio to be processed through a preset ASR model to obtain a text transcription result; The large language model module is used to input the text transcription result, the original question and the first preset prompt word into the first large language model to obtain the key question corresponding to the audio to be processed output by the first large language model; The first preset prompt word is used to prompt the first language model to generate a key question corresponding to the audio to be processed based on the text transcription result and the original question corresponding to the audio to be processed; the key question corresponding to the audio to be processed is a question that does not rely on the understanding of the audio to be processed; The retrieval enhancement generation module is used to search for key questions corresponding to the audio to be processed by accessing an external search engine based on the retrieval enhancement generation technology to obtain multiple semantically related paragraphs; The large language model module is also used to input the multiple semantically related paragraphs, the key questions corresponding to the audio to be processed, and the second preset prompt words into the second large language model, and obtain the target answer to the key question corresponding to the audio to be processed output by the second large language model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice question answering method based on retrieval enhancement generation as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the voice question answering method based on retrieval enhancement generation as described in any one of claims 1 to 7 is implemented.

Images