A dual-mode retrieval enhanced voice interaction method and system based on a heterogeneous multi-core

Abstract

The application provides a dual-mode retrieval enhanced voice interaction method and system based on a heterogeneous multi-core. The application solves the data island problem caused by the isolation of offline recording data and online knowledge base in the prior art through a heterogeneous dual-core architecture and dual-mode retrieval enhancement. The application solves the problems of redundant power consumption, delay and privacy leakage risk caused by software scheduling in the prior art through ULP coprocessor and RTC memory to realize hardware-level mode switching. At the same time, the application solves the frame loss problem caused by I / O blocking during long-time high-fidelity recording of low-power hardware through a heterogeneous dual-core cooperation and shared ring buffer mechanism. The application guarantees the integrity and privacy safety of offline recording through hardware-level mode isolation and dual-core asynchronous anti-frame loss mechanism. Through the fusion of local private vector library and cloud public knowledge base, the application realizes personalized and accurate intelligent question answering, and significantly improves the reliability and intelligent level of voice interaction.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and in particular to a dual-mode retrieval enhanced voice interaction method and system based on heterogeneous multi-core. Background Technology

[0002] The rapid development of intelligent voice interaction technology has brought convenience to scenarios such as office work and education. Existing technical solutions generally employ heterogeneous processors for basic division of labor and collaboration, and use the software scheduling mechanism of the upper-level operating system to switch between offline recording and online question-and-answer modes. Audio acquisition and processing tasks largely rely on single-core processors or homogeneous multi-core general task scheduling, and data buffering typically uses statically pre-allocated circular buffers. Meanwhile, Retrieval Enhancement Generation (RAG) technology has been adopted by some systems to retrieve information from fixed cloud-based public knowledge bases to enhance the accuracy of online question-and-answer.

[0003] However, these technical approaches that unify resource scheduling at the software layer and enable offline data and cloud-based knowledge bases to operate independently still have the following fundamental shortcomings in application scenarios requiring high privacy, high reliability, and deep data fusion: 1) Existing RAG solutions are mostly geared towards fixed public knowledge bases and cannot convert user-generated private and dynamic meeting records offline into searchable and usable vector knowledge, creating data silos and preventing personal historical data from empowering online intelligent question answering; 2) Mode switching relies on software scheduling, requiring all hardware peripherals to be powered on before being softly shut down by the system, generating redundant power consumption peaks. Moreover, this software-layer isolation is not thorough, posing a risk of privacy leakage due to background wake-up; 3) In low-power offline recording scenarios, there is a fundamental contradiction between high-fidelity audio writing and memory card I / O blocking, which traditional single-core or task scheduling architectures cannot solve, leading to inevitable frame drops during long recording sessions and undermining the foundation for subsequent data mining; 4) Existing low-power solutions reduce power consumption through state machine management, but the main processor or complex monitoring circuits still need to be maintained during standby, which cannot meet the requirements for ultra-long battery life. Therefore, a voice interaction method and system that can solve the above problems is urgently needed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a dual-mode retrieval method and system for enhanced voice interaction based on heterogeneous multi-core architecture, which solves the data silo problem caused by the isolation between offline recording data and online knowledge base in existing technologies.

[0005] According to an embodiment of the present invention, a dual-mode retrieval method for enhanced voice interaction based on heterogeneous multi-core processors includes the following steps:

[0006] S1: Trigger the ULP coprocessor interrupt via a physical button, modify the mode flag bit in the RTC memory based on the ULP coprocessor, and generate a switching signal;

[0007] S2: During the boot process, the heterogeneous multi-core processor reads the mode flag and executes the corresponding hardware initialization process to configure the system into database working mode or knowledge base working mode.

[0008] S3: When the system is configured in database working mode, the heterogeneous dual-core processor performs real-time audio processing on the collected audio signal, encrypts the processed pure audio frame and stores it in the storage device, and further generates vectorized data with semantic tags from the audio or converted text data and stores it in the first vector database.

[0009] When the system is configured to work in knowledge base mode, it receives user queries and retrieves relevant information fragments from the first vector database and the second knowledge base in parallel through enhanced dual-channel detection.

[0010] The private information fragments from the first vector database and the general knowledge fragments from the second knowledge base are weighted and fused to form enhanced prompt information, which is then input into the large language model to obtain the final answer.

[0011] Furthermore, the hardware initialization process includes:

[0012] If it is in database working mode, the I2S interface, microphone array driver and AES encryption module are initialized. The microphone array driver is configured to support multi-channel synchronous sampling for beamforming, and the AES encryption module is configured to hardware-accelerated AES-256 encryption mode.

[0013] If the knowledge base is in working mode, the Wi-Fi driver and vector database interface are initialized. The Wi-Fi driver is used to establish a network connection, and the vector database interface is used to adapt to the cloud vector retrieval service.

[0014] Furthermore, the heterogeneous multi-core processor includes a first processor core and a second processor core, and the two cores interact with each other through a shared circular buffer, wherein:

[0015] The first processor core is a real-time control core, bound to the I2S interface, which performs real-time signal enhancement and multi-source separation preprocessing on the audio signal, and writes the processed tagged audio data into the shared circular buffer.

[0016] The second processor core is an application core that asynchronously reads the data stream from the shared circular buffer and calls the AES hardware accelerator for encryption and database management.

[0017] Furthermore, the signal enhancement employs a delay-summation beamforming algorithm, configured to dynamically adjust the beam pointing angle to track the direction of the sound source;

[0018] The multi-source separation preprocessing includes speech activity detection and speaker separation, wherein the speech activity detection is used to distinguish speech segments from non-speech segments, and the speaker separation is used to attach speaker tags to speech segments of different speakers.

[0019] Furthermore, the AES hardware accelerator encryption process and the audio frame writing to the vector database process are executed in parallel;

[0020] The AES hardware accelerator is configured as a pipelined encryption unit;

[0021] The first vector database is configured as a vector storage structure that supports indexing by timestamp and speaker tag.

[0022] Furthermore, the capacity of the shared circular buffer is dynamically allocated based on the maximum physical blocking delay of the storage device, and the minimum capacity of the shared circular buffer must meet the following requirements:

[0023] ;

[0024] in, To determine the minimum capacity of the shared circular buffer, For audio sampling rate, The number of bytes corresponding to the sampling bit width. Number of audio channels This represents the maximum write blocking latency for the storage device.

[0025] Furthermore, the enhanced dual-channel detection for retrieval includes:

[0026] Based on the Token conditional probability distribution output by the large language model, the average negative log probability of the generated sequence is calculated;

[0027] Knowledge matching degree calculation: Calculate the average maximum cosine similarity between the sentence-level embedding of the generated text and the token-level embedding of the retrieved knowledge fragment;

[0028] Fusion Decision: Construct a decision matrix based on the average negative logarithmic probability and the average maximum cosine similarity, and output the hallucination type and processing strategy.

[0029] Furthermore, the weighted fusion includes:

[0030] The number of knowledge fragments in the first vector database that have a similarity to the user query exceeding a basic threshold is counted, and the knowledge base coverage rate is calculated.

[0031] Based on the knowledge base coverage and the preset domain benchmark value, a dynamic filtering threshold is generated through a weighted linear combination.

[0032] Private meeting minutes fragments are retrieved from the first vector database based on the dynamic filtering threshold and combined with general knowledge fragments retrieved from the second knowledge base.

[0033] On the other hand, according to embodiments of the present invention, a dual-mode retrieval enhanced voice interaction system based on heterogeneous multi-core is also provided. The system is used to implement a dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in any of the above aspects, comprising:

[0034] The mode switching hardware unit includes a physical button, a ULP coprocessor, and an RTC memory; the physical button is connected to the interrupt pin of the ULP coprocessor, and the ULP coprocessor is connected to the RTC memory, used to modify the mode flag bit stored in the RTC memory in response to button interrupts;

[0035] The heterogeneous multi-core processor includes a first processor core and a second processor core; the heterogeneous multi-core processor is connected to the RTC memory and is configured to read the mode flag bit during the boot process and select either the database working mode or the knowledge base working mode based on its value.

[0036] An audio acquisition and processing unit is enabled in the database working mode and is connected to the first processor core for acquiring audio signals;

[0037] A first vector database, connected to the heterogeneous multi-core processor, is used to store vectorized representations of audio or text data with semantic tags generated in the database working mode;

[0038] The second knowledge base, which is enabled in the knowledge base working mode, is used to store general knowledge data;

[0039] The retrieval enhancement generation unit is enabled in the knowledge base working mode, connects the first vector database and the second knowledge base, and is used to retrieve information fragments from both in parallel according to the user query, and combine the results to form enhanced prompt information;

[0040] A large language model, connected to the retrieval enhancement generation unit, is used to receive the enhanced prompt information and generate an answer.

[0041] Furthermore, the system also includes a differentiable correction module, which takes the knowledge alignment degree as the objective function, differentiates the hidden state of the large language model, adjusts the probability distribution of high-risk tokens in reverse through gradient, regenerates the corrected fragment, and verifies the correction effect through a loop until the alignment degree is higher than a set threshold.

[0042] The technical principle of this invention is as follows: Hardware-level mode switching triggered by physical buttons is achieved using ULP and RTC memory. During the startup phase, different peripherals are thoroughly initialized based on flag bits, achieving physical isolation between offline and online modes. In offline mode, the heterogeneous dual cores collaborate through a dynamically sized shared circular buffer. The first core processes audio in real-time, while the second core asynchronously encrypts and stores the audio, ensuring high-fidelity recording with zero frame loss. In online mode, the system searches the local private vector library and the cloud-based public knowledge base in parallel, assesses risks through dual-channel pre-detection, and deeply integrates private history and public knowledge using a dynamic threshold fusion strategy. Finally, the input to the large language model generates accurate and personalized answers.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This invention solves the technical problem that existing search enhancement generation schemes cannot utilize local private dynamic data, resulting in data silos, by parallel searching and weighted fusion of a private vector database generated in offline mode and a public knowledge base in the cloud, thus achieving personalized, accurate and intelligent question answering.

[0045] 2. This invention employs a hardware-level mode awareness and switching mechanism implemented with an ultra-low-power coprocessor and RTC memory, and performs separate hardware initialization during the startup phase. It solves the technical problems of redundant power consumption, latency, and privacy leakage risks associated with software scheduling mode switching, achieving complete physical-level isolation, extremely low standby power consumption, and ensuring absolute privacy.

[0046] 3. In database working mode, a heterogeneous dual-core architecture and a shared circular buffer with dynamic capacity allocation based on memory card blocking latency are adopted. This solves the problem of audio data frame loss caused by I / O blocking during long-term high-fidelity recording of low-power hardware, ensuring the integrity of the original audio data.

[0047] 4. This invention maintains only the ULP coprocessor and RTC memory during deep sleep, reducing the overall standby power consumption of the system to an extremely low level (below 5μA). This solves the problem of high power consumption caused by the main processor or monitoring circuit still needing to run during standby in existing solutions, and significantly extends the device's battery life. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the steps of an embodiment of the present invention.

[0049] Figure 2 This is a flowchart of the ULP low-power mode sensing state machine according to an embodiment of the present invention.

[0050] Figure 3 This is a data flow diagram of a dual-core asynchronous anti-frame-loss pipeline according to an embodiment of the present invention.

[0051] Figure 4 This is a data flow diagram of the cross-modal knowledge base construction and generation system in an embodiment of the present invention.

[0052] Figure 5 This is a global diagram of an end-to-cloud collaborative intelligent voice system according to another embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] like Figure 1 As shown in the figure, this invention proposes a dual-mode retrieval method for enhanced voice interaction based on heterogeneous multi-core processors, including:

[0055] S1: Trigger the ULP coprocessor interrupt via a physical button, modify the mode flag in the RTC memory based on the ULP coprocessor, and generate a switching signal;

[0056] S2: During the boot process, the heterogeneous multi-core processor reads the mode flag and executes the corresponding hardware initialization process to configure the system into database working mode or knowledge base working mode.

[0057] S3: When the system is configured in database working mode, the heterogeneous dual-core processor performs real-time audio processing on the acquired audio signal, encrypts the processed clean audio frames and stores them in the storage device, and further generates vectorized data with semantic tags from the audio or converted text data and stores it in the first vector database.

[0058] When the system is configured to work in knowledge base mode, it receives user queries and retrieves relevant information fragments from the first vector database and the second knowledge base in parallel through enhanced dual-channel detection.

[0059] The private information fragments from the first vector database and the general knowledge fragments from the second knowledge base are weighted and fused to form enhanced prompts. The enhanced prompts are then input into the large language model to obtain the final answer.

[0060] This method achieves high privacy, high reliability, and low power consumption intelligent voice interaction through hardware-level pattern awareness and initialization, dual-core asynchronous frame loss prevention processing, and cross-pattern vector fusion. The detailed workflow includes:

[0061] In step S1, the user operates a physical button to trigger a ULP interrupt. The ULP directly modifies the mode flag in the RTC memory to generate a switching signal, without waking up the main CPU.

[0062] The industry practice is to use physical buttons to control GPIO interrupts, wake up the CPU via ULP to start the OS, and then use the application layer to determine the current mode and the software to shut down unused peripherals to achieve low power consumption. However, all peripherals need to be powered on and initialized before being shut down by the software, which leads to unavoidable power consumption peaks. Incomplete shutdown at the software layer poses a privacy risk of being restarted by background processes or malicious software. Its essence is a "fully enabled first, then selectively disabled" strategy.

[0063] To address the above issues, this application employs a method where the ULP is directly written to the RTC memory flag. The Bootloader reads this flag before loading the OS and branches out to execute a completely different hardware initialization tree. First, the RTC memory stores a persistent mode configuration across sleep cycles. Even if the system is completely powered off and restarted, the flag remains valid, essentially acting as a dynamically writable hardware boot configuration vector. Second, the Bootloader does not initialize peripherals required by this mode at all. The clock gating of the RF module is never enabled, and the power domain remains off; it is "never powered on" rather than "powered on and then turned off." Finally, this physical-level privacy isolation ensures that even if the OS is compromised, it cannot enable a peripheral that has never been initialized, completely eliminating the privacy risks associated with incomplete software-level isolation.

[0064] like Figure 2 As shown, the hardware-level mode perception and switching process of the present invention is demonstrated: In the present invention, the system is normally in a deep sleep state, at which time the main CPU and each functional module are powered off, and only the ULP coprocessor and the RTC (real-time clock) memory area maintain extremely low power operation.

[0065] The RTC memory pre-stores a mode flag, for example: Flag=0 represents database / conference mode, and Flag=1 represents knowledge base / question-answer mode. When the user needs to switch working modes, they operate a physical toggle switch or button on the device. This physical button is directly connected to the RTC_GPIO interrupt pin of the ULP coprocessor. The button action generates a level transition, triggering a ULP interrupt.

[0066] In the interrupt service routine, ULP directly reads and modifies the mode flag in the RTC memory without waking up the main CPU. This operation is completed in microseconds with extremely low power consumption. The modification of the mode flag generates the hardware signal for system mode switching. This mechanism is fundamentally different from existing technologies that monitor network status or receive commands through software and then have the operating system schedule resources for a soft switch. It fundamentally avoids the risk of incomplete software shutdown and lays the foundation for subsequent hardware initialization branches at the bootloader level.

[0067] This invention achieves true physical-level, hardware-level mode perception and switching signal generation. In standby mode, the overall system power consumption can be less than 5μA, which greatly extends battery life and provides hardware protection for absolute physical network disconnection, completely eliminating the risk of privacy leakage caused by software vulnerabilities or background processes.

[0068] In step S2, after the main CPU (i.e. the heterogeneous multi-core processor of the main controller) is powered on, it reads the RTC mode flag bit in the bootloader boot phase and executes two completely independent hardware peripheral initialization processes according to its value.

[0069] In response to the high instantaneous power consumption, latency, and resource waste caused by the traditional solution of powering on all hardware peripherals first and then shutting down unnecessary peripherals by the operating system software, this invention advances the mode decision to the lowest-level hardware initialization stage before the operating system loads, achieving the most thorough resource isolation and power consumption optimization.

[0070] After ULP completes the flag modification, it can trigger the main CPU to wake up, power on, and begin executing the bootloader code. In the early stages of the bootloader, before loading the operating system kernel or any complex drivers, it first reads the mode flag set by step S1 from the RTC memory. Subsequently, the bootloader code executes two distinct hardware initialization branches based on the value of the flag:

[0071] If Flag=0 (database mode): The bootloader will directly shield or shut down the clock and power of RF-related modules (such as Wi-Fi and Bluetooth) at the chip register level, achieving physical network disconnection. Subsequently, only the hardware peripherals necessary for this mode are initialized, including: an I2S interface for connecting the microphone array (configured for a high sampling rate, such as 48kHz / 16bit), a microphone array driver for driving the microphone array to perform beamforming, an SPI interface for connecting local storage, and a hardware AES accelerator module for data encryption (configured in AES-256 mode). This initialization path is entirely for high-fidelity, offline, and secure audio acquisition and storage services.

[0072] If Flag=1 (Knowledge Base Mode): The Bootloader will initialize the Wi-Fi driver normally, establishing a connection with the wireless router and a secure link with the cloud server. Simultaneously, to optimize online question-and-answer response speed and system resources, it may suspend or reduce the priority of SD card storage-related operations. Additionally, it will initialize the vector database interface used to connect to the cloud vector retrieval service. This path is entirely dedicated to online retrieval and intelligent question-and-answer services.

[0073] The present invention provides physical isolation between the two working modes at the hardware level. The system enters the optimal hardware configuration state upon startup, eliminating the latency and power consumption peaks caused by software scheduling, significantly improving the efficiency and energy efficiency of mode switching, and providing a stable and dedicated hardware operating environment for subsequent steps.

[0074] In step S3, if in database mode, the first processor verifies the audio and stores the result in a circular buffer. The second processor asynchronously reads the buffer data, encrypts and stores it to form a local vector database.

[0075] To address the fundamental problem of audio data frame loss caused by memory card I / O blocking during long-term high-fidelity recording in low-power hardware, and the scheduling conflicts that easily arise when a single core or homogeneous multi-core processes real-time computing and high-latency I / O simultaneously, this invention adopts a heterogeneous multi-core architecture to decouple real-time computing and asynchronous I / O on the physical core, and smooths the data flow through a dynamic capacity circular buffer.

[0076] After the system is initialized to database mode according to step S2, the dual-core asynchronous anti-frame-loss pipeline is executed. The specific data flow is as follows: Figure 3 As shown, the audio signal is first enhanced and separated by Core0 and then written to a circular buffer. It is then asynchronously read, encrypted, and stored / vectorized by Core1 in a parallel processing path. The circular buffer capacity is guaranteed by a computational mechanism. Specifically, this includes:

[0077] Real-time audio processing (executed by the first processor core, Core0): Core0 is a processor core optimized for real-time tasks. It is bound to the I2S DMA controller and is dedicated to processing the audio stream from the microphone array. Core0 runs the speech algorithm library and performs the following operations in sequence:

[0078] Signal enhancement: A delay-summation beamforming algorithm is employed. This algorithm dynamically adjusts the beam pointing angle by calculating the time delay difference of the sound signals received by different microphones, thereby enhancing the signal from the direction of the target speaker and suppressing ambient noise and reverberation.

[0079] Multi-source separation preprocessing includes Voice Activity Detection (VAD) and speaker separation. VAD distinguishes speech segments from non-speech segments (such as silence or noise) in the audio stream, and only speech segments undergo subsequent processing. Speaker separation adds speaker tags to speech segments from different speakers, enabling the structuring of meeting transcripts.

[0080] Core0 writes each frame of clean, tagged audio data into a circular buffer in shared memory in a non-blocking manner.

[0081] Asynchronous encryption and storage (executed by the second processor core, Core1): Core1 is a general-purpose application core responsible for high-latency, interruptible tasks. It continuously monitors the read pointer of the circular buffer. When new data is available in the buffer, Core1 asynchronously reads these audio frames and then performs the following operations:

[0082] Encryption: Audio data is sent to the SoC's built-in AES hardware accelerator for real-time encryption. This accelerator is configured for pipelined operation and supports the AES-256 encryption algorithm to ensure data security before storage.

[0083] Storage and Vectorization: Encrypted data is persistently stored on a local SD card via the SPI bus. Simultaneously, these tagged text or audio segments (which can be converted locally or via cloud-based ASR) are transformed into vectorized representations using an embedding model and stored in a local first vector database. This database supports indexing by timestamp and speaker tags for easy subsequent retrieval.

[0084] Frame loss prevention mechanism: The capacity of the shared circular buffer is crucial for preventing frame loss. Its minimum capacity must satisfy the formula:

[0085] ;

[0086] in, To determine the minimum capacity of the shared circular buffer, This is the audio sampling rate (e.g., 48kHz in database mode). The number of bytes corresponding to the sampling bit width. This represents the number of audio channels (1 for a single channel). This represents the maximum write blocking latency of the SD card as measured in practice. As long as the buffer capacity is greater than this calculated value (such as 16KB in the example), even if Core1 pauses reading due to SD card blocking, Core0 can continue writing data without loss, achieving physical-level zero frame loss. This invention completely solves the problem of data frame loss in long-term, high-fidelity offline recording scenarios for low-power hardware, ensuring 100% integrity of the original conference data. This provides a reliable data foundation for subsequent generation of high-quality vector embeddings and accurate cross-modal retrieval. The dual-core physically isolated architecture maximizes the computing power advantages of heterogeneous multi-core processors.

[0087] In step S3, when in knowledge base mode, the system receives user queries, retrieves relevant information from the local private vector library and the cloud public knowledge base in parallel, and then inputs the fused information into the large language model to generate an answer.

[0088] Considering that existing RAG (Research and Analysis of Knowledge) schemes have fixed processes, making it impossible to predict risks before generation and dynamically adjust retrieval strategies based on those risks, and that the generated results may be "illusory," this invention introduces a dual-channel pre-detection and decision-making mechanism based on benchmark indicators before retrieval. This mechanism can pre-determine the difficulty and potential risks of the problem, thereby intelligently deciding whether to activate and how to configure subsequent private-public knowledge fusion retrieval strategies, achieving a leap from a "fixed process" to an "intelligent decision-making process."

[0089] Once the system is initialized to knowledge base mode according to step S2 and successfully connected to the network, the process for handling user queries adopts a unique dual-wheel architecture of "initial generation → detection → adjustment → fine generation" as described in this application:

[0090] 1. Receiving and parsing user queries: The system receives user questions via voice or text interface. For example: "What should the project acceptance standards discussed in yesterday's meeting be, based on the latest regulations?"

[0091] 2. Parallel retrieval: such as Figure 4 The diagram illustrates the cross-modal knowledge base retrieval data flow of this invention, demonstrating the interaction between private memory and public knowledge in the retrieval, evaluation, and fusion stages, including:

[0092] 1) Local cross-domain retrieval: The terminal first decrypts the first vector database, performs precise retrieval based on the semantic similarity of the user's query, and extracts relevant data recorded during the offline meeting. A private context (such as a fragment of acceptance criteria mentioned in a meeting). The retrieval formula is as follows:

[0093] ;

[0094] in, For the search results, For query vector, Given candidate vectors, argmax is used to find the maximum parameter.

[0095] 2) Cloud-based knowledge supplement (second knowledge base): At the same time, user queries are sent to the cloud and retrieved from the second knowledge base, which stores general encyclopedias, technical specifications and other data, to obtain relevant public knowledge fragments (such as national regulations, technical standards and so on).

[0096] 3. Initial Response Generation: The system performs initial splicing of private and public fragments retrieved in parallel to form a preliminary context. Based on this, it calls the large language model to perform a lightweight, draft-style initial generation to obtain the initial response. This round of generation uses a low-cost method to anticipate risks and provides a basis for subsequent precise resource allocation.

[0097] 4. Dual-channel pre-detection and decision matrix construction: Based on the initial response, the system performs dual-channel detection, including: 1) Generation sequence accuracy detection: The system calculates the average negative logarithmic probability (ANLP) representing the linguistic logical fluency of the current generation sequence based on the Token conditional probability distribution output by the LLM. The formula is as follows:

[0098] ;

[0099] in, For the currently generated sequence, The length of the generated sequence, Given the context, user question, and retrieved knowledge, this represents the conditional probability of the token. A higher ANLP value indicates that the model has dispersed probabilities in multiple places, the generated content is logically chaotic, and there is a higher risk of illusion.

[0100] 2) Knowledge Matching Degree Calculation: By comparing the semantic embeddings of the generated text and the retrieved knowledge, the consistency between the generated content and the facts is evaluated. The formula is as follows:

[0101] ;

[0102] in, To generate sentences, To retrieve knowledge fragments, , These are the context embeddings for the corresponding tokens. To generate the sequence length, This represents the length of the knowledge segment. Among them, Represents the calculation to generate a token With Knowledge Token The cosine of the angle between two context vectors has a range of values. The closer the value is to 1, the more semantically consistent the generated token is with the corresponding knowledge token.

[0103] A higher AlignScore indicates that the generated content is more semantically aligned with the retrieved knowledge and has better factual consistency; conversely, a lower AlignScore indicates that the generated content deviates from the retrieved knowledge and has factual bias.

[0104] Based on the aforementioned ANLP and AlignScore values, a decision matrix is ​​constructed to output the current illusion type and processing strategy, providing a basis for subsequent dynamic threshold adjustment. As shown in Table 1, this application defines four types of illusions: Type 1 is fact-deviation illusion, which has high confidence but the content is inconsistent with the retrieved knowledge, making users most easily misled; Type 2 is knowledge blind spot illusion, which is caused by the lack of sufficient relevance between the retrieved content and the database, resulting in deviation due to the model having no basis; Type 3 is logical break illusion, which, although citing correct knowledge fragments, has a broken reasoning chain or reversed causality; and Type 4 is source conflict illusion, which is caused by factual contradictions between the search results of private and public databases, leading to the model oscillating between contradictory information.

[0105]

[0106] Table 1

[0107] Based on the type of hallucination identified, the decision matrix outputs the corresponding processing strategy:

[0108] Dynamic threshold readjustment: Adjusting the dynamic threshold in subsequent fusion based on the type of hallucination. Weighting coefficients (Proportional to the dynamic threshold), expanding or narrowing the scope of knowledge fragment selection, where the weighting coefficient... The hallucination type output from the decision matrix is ​​determined by looking up a table:

[0109] If it is a fact-deviation type of hallucination: Lower the threshold (e.g., 0.3-0.5) and relax the screening criteria to include more candidate knowledge fragments to combat factual deviations;

[0110] If it is a hallucination caused by a logical break: Increase the threshold (e.g., 0.7-0.9) to tighten the filtering criteria, retaining only the most relevant segments to the query and reducing interference;

[0111] If it is a hallucination caused by a knowledge gap: Maintain a neutral level (0.5-0.6) and trigger query decomposition and re-retrieval;

[0112] If it is a hallucination with conflicting sources: Maintain neutral (0.5-0.6), triggering source-weighted adjustment;

[0113] Query decomposition and re-retrieval: The original query is decomposed into multiple subqueries, and the insufficient knowledge domains are retrieved again in a targeted manner.

[0114] Source weighting adjustment: Adjust the weight distribution of private vector libraries and public knowledge bases during weighted fusion, and prioritize the one with higher coverage. For example, if the coverage of the private vector library is greater than that of the public library, then increase the weight of the private segment (e.g., 0.7 vs 0.3), and vice versa.

[0115] In some examples, considering that existing retrieval enhancement generation schemes may still produce factual errors or illusions in complex or knowledge-boundary scenarios, and that traditional post-processing or hint engineering methods are difficult to correct accurately and interpretably, this invention also introduces a differentiable correction mechanism:

[0116] This module defines "knowledge alignment" as a differentiable objective function that integrates the semantic similarity between the generated answer and the retrieved knowledge. The correction process is an iterative optimization loop:

[0117] Gradient calculation: This module differentiates the hidden state of the large language model corresponding to the generated answer and calculates the gradient of the objective function with respect to these model internal parameters. This answers the question of "how to fine-tune the internal representation of the model to make the answer more closely reflect the knowledge".

[0118] Backward adjustment: Using the calculated gradient, a subtle adjustment is made to the probability distribution of a small range of high-risk tokens through gradient backpropagation. This is not retraining the model, but rather a local, guided deduction within the current generation context, aimed at reducing the probability of these tokens generating contradictory or fabricated content, while increasing the probability of tokens consistent with knowledge fragments.

[0119] Regeneration: Based on the adjusted internal state or probability distribution of the model, the module regenerates the segments marked as high-risk in the answer, resulting in a corrected answer.

[0120] Verification loop: The system immediately calculates the AlignScore value of the corrected answer. If its alignment is higher than the preset threshold, the correction is successful and the corrected answer is output; if it still does not meet the threshold, the steps are repeated to form a verification loop for the correction effect until an answer that meets the alignment requirements is generated or the maximum number of iterations is reached.

[0121] This differentiable correction mechanism can not only detect potential problems in the answer, but also proactively and accurately correct problems through gradient-based optimization methods, thereby raising the factual consistency and reliability of the generated answer to a new level in complex question-and-answer scenarios.

[0122] In some examples, confidence downgrade output can also be performed: an uncertainty statement is attached to the final answer, explicitly informing the user of the reliability level of the answer.

[0123] 5. Information Fusion and Decision-Making: Based on the results of the decision matrix, the fusion parameters are adjusted, including dynamic threshold adjustment, source weight adjustment, and whether to trigger a secondary retrieval. Subsequently, the adjusted parameters are used to filter and weight all candidate knowledge fragments obtained from parallel retrieval, forming the final enhanced prompt information. Specifically, this includes:

[0124] 1) Knowledge Base Coverage Calculation: This metric measures the proportion of knowledge fragments in the search results that are truly relevant to the user's question. The calculation formula is as follows:

[0125] ;

[0126] in, For user issues, For user issues With knowledge fragments cosine similarity, A preset correlation threshold is used (usually between 0.5 and 0.7). This is an indicator function that returns 1 if the condition is true and 0 if the condition is false. In this preferred embodiment: .

[0127] 2) Generate dynamic threshold: Dynamic threshold This formula combines the basic difficulty of the generation task with the quality of the current search results to more finely filter the final knowledge fragments for generating answers.

[0128] ;

[0129] in, These are weighting coefficients, adjusted by the output of the decision matrix; , This is the baseline average negative log probability value calculated on a pre-trained sample set for a specific domain. This represents the total number of knowledge fragments retrieved.

[0130] 3) Knowledge fusion: Utilizing dynamic thresholds As a threshold, all candidate knowledge fragments obtained from parallel retrieval (including local private and cloud public fragments) are filtered, retaining only... > High-quality clips. These clips are weighted and combined to form the final enhanced cue message. The data is then sent to a large language model (LLM) in the cloud. The large model aligns private history with public knowledge through an attention mechanism, generating personalized answers that include both the user's personal meeting details and macro-level accuracy.

[0131] This invention breaks down the data barriers between offline recording and online Q&A through a fusion mechanism, achieving deep integration of personal historical data and public knowledge. This not only significantly improves the personalization and accuracy of Q&A but also greatly increases the system's practical value and the breadth of its application scenarios.

[0132] like Figure 5 As shown in the embodiments, this invention also provides a dual-mode retrieval enhanced voice interaction system based on heterogeneous multi-core architecture. This system is used to implement a dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core architecture as described in any of the above embodiments, including an edge-side intelligent voice terminal and a cloud-side remote fusion server.

[0133] The edge-side intelligent voice terminal is based on a heterogeneous multi-core processor and specifically includes:

[0134] The mode switching hardware unit includes physical buttons, an ULP ultra-low power coprocessor, and RTC memory. The physical buttons are connected to the ULP's interrupt pin. The ULP responds to button interrupts and directly modifies the mode flag in the RTC memory to achieve hardware-level mode awareness.

[0135] Heterogeneous multi-core processor: Includes a first processor core and a second processor core. During the boot process, this processor reads the mode flags in the RTC and selects to execute a completely independent hardware initialization process to configure the system into database mode or knowledge base mode.

[0136] Audio Acquisition and Processing Unit: Enabled in database mode, this unit typically includes a microphone array and connects to the first processor core via an I2S interface for acquiring audio signals. The first processor core is configured to run real-time acoustic algorithms for processing.

[0137] The first vector database is located in local storage (such as an SD card) and connected to a heterogeneous multi-core processor. It is used to store the vectorized representation of audio / text data with semantic tags (such as timestamps and speakers) generated in database mode.

[0138] Second Knowledge Base and Retrieval Enhancement Generation Unit: In knowledge base mode, a Wi-Fi connection is established with the cloud side. The cloud side deploys a second knowledge base, a retrieval enhancement generation unit, and a large language model. Based on the user's query, the retrieval enhancement generation unit retrieves information fragments in parallel from the first vector database and the second knowledge base, performs weighted fusion to form enhanced prompts, and inputs these prompts into the large language model to generate the final answer.

[0139] The cloud-based remote fusion server includes an ASR speech-to-text engine, an Embedding vectorization extractor, a first vector database, a second knowledge base, a RAG retrieval engine, and an LLM large language model, which together complete the tasks of knowledge storage, retrieval, fusion, and generation.

[0140] Through the collaborative work of the aforementioned units, the system fully realizes the entire process of functions, from hardware-level low-power mode switching and reliable offline data acquisition to cross-mode intelligent enhanced question answering.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dual-mode retrieval method for enhanced voice interaction based on heterogeneous multi-core processors, characterized in that: Includes the following steps: S1: Trigger the ULP coprocessor interrupt via a physical button, modify the mode flag bit in the RTC memory based on the ULP coprocessor, and generate a switching signal; S2: During the boot process, the heterogeneous multi-core processor reads the mode flag and executes the corresponding hardware initialization process to configure the system into database working mode or knowledge base working mode. S3: When the system is configured in database working mode, the heterogeneous dual-core processor performs real-time audio processing on the collected audio signal, encrypts the processed pure audio frame and stores it in the storage device, and further generates vectorized data with semantic tags from the audio or converted text data and stores it in the first vector database. When the system is configured to work in knowledge base mode, it receives user queries and retrieves relevant information fragments from the first vector database and the second knowledge base in parallel through enhanced dual-channel detection. The private information fragments from the first vector database and the general knowledge fragments from the second knowledge base are weighted and fused to form enhanced prompt information, which is then input into the large language model to obtain the final answer.

2. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 1, characterized in that: The hardware initialization process includes: If it is in database working mode, the I2S interface, microphone array driver and AES encryption module are initialized. The microphone array driver is configured to support multi-channel synchronous sampling for beamforming, and the AES encryption module is configured to hardware-accelerated AES-256 encryption mode. If the knowledge base is in working mode, the Wi-Fi driver and vector database interface are initialized. The Wi-Fi driver is used to establish a network connection, and the vector database interface is used to adapt to the cloud vector retrieval service.

3. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 1, characterized in that: The heterogeneous multi-core processor includes a first processor core and a second processor core. The two cores interact with each other through a shared circular buffer, wherein: The first processor core is a real-time control core, bound to the I2S interface, which performs real-time signal enhancement and multi-source separation preprocessing on the audio signal, and writes the processed tagged audio data into the shared circular buffer. The second processor core is an application core that asynchronously reads the data stream from the shared circular buffer and calls the AES hardware accelerator for encryption and database management.

4. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 3, characterized in that: The signal enhancement employs a delay-summation beamforming algorithm, configured to dynamically adjust the beam pointing angle to track the direction of the sound source. The multi-source separation preprocessing includes speech activity detection and speaker separation, wherein the speech activity detection is used to distinguish speech segments from non-speech segments, and the speaker separation is used to attach speaker tags to speech segments of different speakers.

5. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 3, characterized in that: The AES hardware accelerator encryption process and the audio frame writing to the vector database are executed in parallel. The AES hardware accelerator is configured as a pipelined encryption unit; The first vector database is configured as a vector storage structure that supports indexing by timestamp and speaker tag.

6. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 3, characterized in that: The capacity of the shared circular buffer is dynamically allocated based on the maximum physical blocking delay of the storage device, and the minimum capacity of the shared circular buffer must meet the following requirements: ； in, To determine the minimum capacity of the shared circular buffer, For audio sampling rate, The number of bytes corresponding to the sampling bit width. Number of audio channels This represents the maximum write blocking latency for the storage device.

7. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 1, characterized in that: The enhanced dual-channel detection for retrieval includes: Based on the Token conditional probability distribution output by the large language model, the average negative log probability of the generated sequence is calculated; Knowledge matching degree calculation: Calculate the average maximum cosine similarity between the sentence-level embedding of the generated text and the token-level embedding of the retrieved knowledge fragment; Fusion Decision: Construct a decision matrix based on the average negative logarithmic probability and the average maximum cosine similarity, and output the hallucination type and processing strategy.

8. The dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in claim 1, characterized in that: The weighted fusion includes: The number of knowledge fragments in the first vector database that have a similarity to the user query exceeding a basic threshold is counted, and the knowledge base coverage rate is calculated. Based on the knowledge base coverage and the preset domain benchmark value, a dynamic filtering threshold is generated through a weighted linear combination. Private meeting minutes fragments are retrieved from the first vector database based on the dynamic filtering threshold and combined with general knowledge fragments retrieved from the second knowledge base.

9. A dual-mode retrieval enhanced voice interaction system based on heterogeneous multi-core architecture, characterized in that: The system is used to implement a dual-mode retrieval enhanced voice interaction method based on heterogeneous multi-core as described in any one of claims 1-8, comprising: The mode switching hardware unit includes a physical button, a ULP coprocessor, and an RTC memory; the physical button is connected to the interrupt pin of the ULP coprocessor, and the ULP coprocessor is connected to the RTC memory, used to modify the mode flag bit stored in the RTC memory in response to button interrupts; The heterogeneous multi-core processor includes a first processor core and a second processor core; the heterogeneous multi-core processor is connected to the RTC memory and is configured to read the mode flag bit during the boot process and select either the database working mode or the knowledge base working mode based on its value. An audio acquisition and processing unit is enabled in the database working mode and is connected to the first processor core for acquiring audio signals; A first vector database, connected to the heterogeneous multi-core processor, is used to store vectorized representations of audio or text data with semantic tags generated in the database working mode; The second knowledge base, which is enabled in the knowledge base working mode, is used to store general knowledge data; The retrieval enhancement generation unit, enabled in the knowledge base working mode, connects the first vector database and the second knowledge base, and is used to retrieve information fragments from both in parallel according to the user query, and combine the results to form enhanced prompt information; A large language model, connected to the retrieval enhancement generation unit, is used to receive the enhanced prompt information and generate an answer.

10. The dual-mode retrieval enhanced voice interaction system based on heterogeneous multi-core as described in claim 9, characterized in that: The system also includes a differentiable correction module, which takes the knowledge alignment degree as the objective function, differentiates the hidden state of the large language model, adjusts the probability distribution of high-risk tokens in reverse through gradient, regenerates the corrected fragment, and verifies the correction effect through a loop until the alignment degree is higher than a set threshold.

Images