Speech interaction method based on retrieval enhancement and multi-model cooperation and application thereof

Abstract

The application relates to the field of artificial intelligence and voice dialogue technology and discloses a voice interaction method based on retrieval enhancement generation and multi-model cooperation and application thereof, which comprises the following steps: voice collection and preprocessing; local multi-round intention recognition; local RAG retrieval and candidate aggregation; decision and cloud generation; feedback, synthesis and persistence; the application also provides application of the voice interaction method based on retrieval enhancement generation and multi-model cooperation. Through local structured intention recognition and RAG retrieval aggregation, combined with the generation capability of a cloud large model, the application takes into account response timeliness and privacy protection, significantly improves the accuracy and robustness of recommendations in a multi-round voice interaction scene, reduces retrieval redundancy and improves context quality through entity-level merging deduplication, rearrangement and fuzzy matching strategies, guarantees the consistency of data updates and the stability of online services through an offline atomic replacement mechanism, thereby improving the terminal user experience and facilitating engineering deployment and expansion.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and voice dialogue technology, specifically to a voice interaction method and its application based on retrieval-enhanced generation and multi-model collaboration. Background Technology

[0002] In recent years, with the development of artificial intelligence, speech recognition and natural language processing technologies, intelligent recommendation systems based on voice interaction have been widely used in retail and catering scenarios. These systems aim to understand user needs and provide ordering or product recommendation services through natural language dialogue, thereby improving user experience and operational efficiency.

[0003] Existing voice ordering systems primarily rely on two technical approaches: one is complete reliance on cloud servers, uploading collected voice data for recognition and natural language processing, and utilizing the understanding and generation capabilities of large cloud models to return results; the other employs local rule or keyword matching methods, using pre-defined logic trees and fixed response scripts to handle simple user requests. Some solutions are also beginning to explore the introduction of knowledge graphs or vector retrieval technologies to improve the semantic matching ability of user intent.

[0004] However, existing voice interaction systems rely heavily on cloud-based models, leading to high network communication latency, untimely interaction responses, and privacy risks associated with the leakage of user dialogue data. Local methods based on keywords or rules struggle to handle ambiguous natural language expressions and complex constraints involving multi-turn contexts, resulting in low recommendation accuracy. Furthermore, simple vector retrieval often generates redundant or repetitive information, lacking strategies for serializing and merging multiple text descriptions of the same entity and filtering at the attribute level. Therefore, this invention provides a voice interaction method and its application based on retrieval-enhanced generation and multi-model collaboration to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a voice interaction method and its application based on retrieval-enhanced generation and multi-model collaboration. This method solves the problems of high response latency, high risk of user privacy leakage, and difficulty in ensuring the consistency of local knowledge base updates caused by the reliance on cloud models in existing voice interaction systems.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The first aspect of this invention provides a voice interaction method based on retrieval-enhanced generation and multi-model collaboration, comprising the following steps:

[0008] S1. Voice Acquisition and Preprocessing: The user's voice signal is acquired in real time through a microphone array, converted into text through speech recognition, and the server is requested to perform language detection based on a pre-trained model.

[0009] S2, Local Multi-Turn Intent Recognition: Taking the recent dialogue history as input, the local lightweight language model strictly extracts order / recommendation related fields from the historical dialogue and outputs them in structured JSON format;

[0010] S3, Local RAG Retrieval and Candidate Aggregation: Based on the structured constraints of S2, candidate entities are selected in the local knowledge base / knowledge graph, a set of text blocks corresponding to the candidate entities is constructed, and semantic retrieval is performed in the vector retrieval module. Multiple text blocks of the same entity are merged and deduplicated in order to generate entity summaries.

[0011] S4. Decision-making and cloud generation: When a user confirms an order and a saleable item is matched, a confirmation instruction is issued. Otherwise, the entity summary and conversation history are encapsulated and sent to the cloud model for natural language processing and strategy decision-making. The cloud-returned results are verified for completeness and then fed back to the front end. Real-time translation is performed when necessary to match the user's language.

[0012] S5, Feedback, Synthesis, and Persistence: The front end synthesizes and plays voice responses; the server saves session history and order metadata for statistics and backtracking; and periodically pulls remote product data and updates official resources using atomic replacement while maintaining consistency.

[0013] Preferably, in step S1, the user's voice signal is acquired in real time through a microphone array, and the input analog voice signal is converted from analog to digital to generate digital voice data; then, the acquired voice signal is processed by noise reduction, echo cancellation, and endpoint detection to remove environmental noise and silent segments; the processed voice signal is normalized by sampling rate and feature extracted (such as MFCC, Mel spectrum, etc.), and converted into text by speech recognition; the preprocessed text and necessary metadata are sent to the server; the server performs language detection based on the Lingua library and returns language tags.

[0014] Preferably, in step S2, the most recent rounds of preprocessed dialogue history are sequentially processed by calling a local lightweight open-source large language model; the output format and extraction rules are enforced in the system prompts: based only on historical dialogue information, all fields must exist, unmentioned fields return null, and the generation of natural language descriptions is prohibited; a normalization mapping table is formulated for different expressions of each field, for example, fields such as "no sweetness" and "no added sugar" are uniformly normalized to "sugar-free", and fields such as "non-alcoholic" and "alcohol-free" are uniformly normalized to "alcohol-free", etc., mapping diverse user expressions to a set of standard values ​​to ensure consistency of downstream fields; the model should return a single pure JSON object; the model output is formatted and fault-tolerantly repaired: code block wrapping is removed, trailing commas or quotation marks are fixed, etc., and if the verification fails, the default structure of all nulls is returned and a re-question or friendly prompt strategy is triggered.

[0015] Preferably, in step S3, the entity set is filtered in the local knowledge graph based on the returned structured constraints; a text block set is constructed for each candidate entity, and embeddings are generated using the local SentenceTransformer and semantic retrieval is performed in vector databases such as ChromaDB; low-relevance results are eliminated according to the similarity threshold; multiple text blocks of the same entity are merged in order and deduplicated (removing the introductory prefix and deduplicating based on sentence level) to generate an entity summary.

[0016] Preferably, in step S4, if the user confirms the order and matches the available products, inventory detection and price verification are performed, and a confirmation instruction is issued to complete the order process; otherwise, the entity summary and conversation history are encapsulated into messages and sent to the cloud processing model for natural language generation and strategy decision-making; after verifying the sentence integrity of the cloud return, the results are translated in real time if necessary to match the user's language before being returned to the terminal.

[0017] Preferably, in step S5, the final response is returned to the front end for speech synthesis and playback by the terminal; simultaneously, the conversation history, intent recognition results, and order metadata are saved for statistical analysis and backtracking. The backend periodically pulls remote product data to generate a temporary knowledge graph and vector library; under the condition that all devices are briefly offline and protected by an operation lock, the official resources are switched atomically, with backups retained before and after the replacement to support rollback.

[0018] The second aspect of this invention provides an application of a voice interaction method based on retrieval-enhanced generation and multi-model collaboration, including:

[0019] The front-end multimodal terminal module is used for user face detection and recognition, and triggers microphone wake-up based on the face detection results. During voice interaction, multimodal noise reduction technology is used to suppress environmental noise in real time, and completes high-fidelity voice acquisition, speech recognition and speech synthesis processing.

[0020] The local multi-turn intent recognition module is used to extract order or recommendation fields from the historical dialogue information based on the recent dialogue history using a lightweight language model and return them as structured JSON.

[0021] The local retrieval enhancement generation module includes an embedding generator, a vector database, and a knowledge graph. It is used to filter candidate beverages by category, alcohol content, or sugar content knowledge graph constraints, and to retrieve relevant text blocks from the vector database based on user dialogue history. Multiple text blocks of the same entity are merged sequentially and deduplicated to generate entity summaries.

[0022] The cloud-based dialogue processing module is used to receive historical dialogue information and retrieved beverage data, call the cloud model for natural language processing, and return the data in a streaming manner.

[0023] The order confirmation module is used to perform inventory checks, price verifications, and issue an order confirmation instruction when it detects that a user has confirmed their intention to place an order and matches available beverages.

[0024] The offline update and atomic replacement module is used to atomically replace temporarily generated knowledge graphs, attribute tables, and vector databases with official resources when the device is temporarily offline and protected by an operation lock.

[0025] Preferably, the local multi-turn intent recognition module is further used for:

[0026] Before returning the structured JSON, perform format validation and error correction on the structured JSON;

[0027] The structured JSON contains fields such as state, variety, specific_variety, alcohol, taste, sweet, and recommend_number, and fields that do not appear in the user's message are represented by null values.

[0028] Preferably, the local retrieval enhancement generation module includes:

[0029] An embedding generator, wherein the embedding generator is a model based on sentence vectors or semantic embedding;

[0030] A vector retrieval unit, wherein the vector retrieval unit is a vector database that supports similarity queries;

[0031] The name fuzzy matching unit uses fuzzy string matching based on distance similarity, combined with an alias mapping table and sets a similarity threshold to tolerate spelling or pronunciation differences.

[0032] Preferably, the cloud-based dialogue processing module includes a language switching and local translation unit, used to provide multilingual responses when the language of the user and the generated result is inconsistent;

[0033] When performing an offline update operation, the offline update and atomic replacement module requires the client to sign the request using pre-allocated credentials and a key-based signature. The payload returned by the server includes both a timestamp and the server's signature. After receiving the response, the client must verify the response signature and timestamp.

[0034] Preferably, the multimodal noise reduction technology is achieved through a multimodal neural network architecture that combines convolutional neural networks and recurrent neural networks;

[0035] The order confirmation module is used to send order information through a local communication protocol and record order metadata and session history for retrospective purposes.

[0036] This invention provides a voice interaction method based on retrieval-enhanced generation and multi-model collaboration, and its application. It has the following beneficial effects:

[0037] 1. This invention reduces network latency in simple interactions and order placement processes by deploying a lightweight language model locally to perform intent recognition and RAG retrieval, and prioritizes processing order confirmation locally. This improves the system's response speed. Since most of the dialogue history and intent analysis are completed locally, the amount of data reported to the cloud is reduced, effectively protecting the user's session privacy.

[0038] 2. This invention first converts the dialogue history into a structured JSON object through a local multi-turn intent recognition module. Then, the JSON object serves as a strong constraint for retrieving information from the local knowledge graph and vector library. This ensures that the input for enhanced retrieval is strictly limited to the user's specific intent and local knowledge of available products. Combined with entity summary merging and deduplication, this provides highly relevant and non-redundant context for subsequent cloud-based generation, thereby improving the accuracy of recommended content and suppressing model illusions.

[0039] 3. This invention generates and verifies temporary knowledge graphs and vector databases in the background, and switches to formal resources in an atomic replacement manner under the condition of obtaining an operation lock, while retaining rollback backups. This ensures high availability and data consistency of local product data and services during the update process, avoids system service instability caused by resource read / write conflicts or update interruptions, and facilitates the engineering deployment and subsequent operation and maintenance of the system. Attached Figure Description

[0040] Figure 1 This is a system architecture diagram of the present invention;

[0041] Figure 2This is a complete flowchart of the dialogue history analysis of the present invention;

[0042] Figure 3 This is a flowchart of the method steps of the present invention. Detailed Implementation

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] See attached document Figure 1 , Figure 1 This is a system architecture diagram according to an embodiment of the present invention. The present invention provides an application of a voice interaction method based on retrieval-enhanced generation and multi-model collaboration, including: a front-end multimodal terminal module, a local multi-turn intent recognition module, a local retrieval-enhanced generation module, a cloud-based dialogue processing module, an order confirmation module, and an offline update and atomic replacement module. The data interaction and processing flows of each module together constitute the overall workflow of the present invention.

[0045] Reference Figure 3 , Figure 3 This is a flowchart of a method according to an embodiment of the present invention. The present invention provides a voice interaction method based on retrieval enhancement generation and multi-model collaboration, comprising the following steps:

[0046] S1. Voice Acquisition and Preprocessing. The user's voice signal is acquired by the front-end multimodal terminal module, and preprocessed by noise reduction and endpoint detection. Then, the voice recognition function generates the dialogue text for the current round.

[0047] S2. Local Multi-Turn Intent Recognition. The local multi-turn intent recognition module appends the generated current-turn dialogue text to the historical dialogue, constructing an updated complete dialogue history. This module uses a locally deployed lightweight language model to analyze the updated complete dialogue history, extracting preset order or recommendation-related fields and outputting them as a structured JSON object.

[0048] S3. Local RAG Retrieval and Candidate Aggregation. The local retrieval enhancement generation module receives a structured JSON object and uses the fields and values ​​in the JSON object as retrieval constraints to filter candidate entities that meet the constraints from the local knowledge base or knowledge graph. For the filtered candidate entities, the local retrieval enhancement generation module further performs semantic retrieval in the vector database and merges and deduplicates multiple text blocks belonging to the same entity in a predetermined order to finally generate an entity summary.

[0049] S4. Decision-making and Generation. The system makes decisions based on user intent information contained in a structured JSON object. If the system determines that the user intends to place an order, and this intention matches a salable product item in the local knowledge base, the order confirmation module performs inventory verification and price verification locally and issues a confirmation command.

[0050] Conversely, if the conditions for local order placement are not met, the system encapsulates the entity summary and the updated complete dialogue history, and sends the encapsulated information to the cloud-based dialogue processing module. The cloud-based dialogue processing module calls the cloud model to perform natural language processing and generate the final natural language response text.

[0051] S5, Feedback, Synthesis, and Persistence. The system sends the natural language response text generated in S4 to the front-end multimodal terminal module for speech synthesis and playback to the user. Simultaneously, it persistently stores the structured JSON object, the final response text, and other session data for subsequent statistical analysis and information retrieval.

[0052] The offline update and atomic replacement module runs periodically in the background. It is responsible for pulling product data from remote sources, building temporary knowledge graphs and vector databases locally, and updating the local deployed official resources in an atomic replacement manner when preset security conditions are met.

[0053] To further clarify the present invention, the specific technical implementation of each step in the above method flow will be described in detail below.

[0054] Step S1 involves the front-end multimodal terminal acquiring ambient sound signals in real time via a microphone array. In a preferred embodiment, the microphone array employs six microphone units arranged in a circular, equidistant layout. The plane containing the array can form an angle of no more than 10 degrees with the horizontal plane. This layout facilitates sound source localization and beamforming, thereby enhancing the ability to capture speech signals from specific directions. Depending on the physical limitations of the actual application scenario, microphone arrays of other numbers or layouts can also be used.

[0055] The acquired analog speech signal is then converted from analog to digital (ADC) within the terminal to generate a digitized speech data stream. To improve the accuracy of subsequent speech recognition, this digitized speech data needs to be preprocessed. Specifically, the system performs noise reduction processing on the acquired speech signal to suppress ambient background noise, performs echo cancellation processing to remove interference generated by the device's own sound playback, and performs voice activity detection (VAD) to identify and separate valid speech segments and remove silent parts.

[0056] After the above processing, clear speech data is obtained. This speech data is then further processed by sampling rate normalization to ensure that the data format input to the speech recognition module remains consistent. Acoustic features of the speech signal are extracted, such as Mel-frequency cepstral coefficients (MFCC) or Mel spectrograms. These acoustic features are then fed into the speech recognition module, which converts them into corresponding text results.

[0057] After obtaining the initial identified text, the front-end multimodal terminal uploads the text fragment and necessary session metadata to the server, requesting language detection. Upon receiving the data, the server calls a pre-trained language detection model to determine the language of the text. In this embodiment, to reduce the privacy risks of data transmission and ensure processing efficiency, the server preferably uses a lightweight, locally deployable language detection library, such as Lingua.

[0058] The output of the language detection model contains two key fields: the language code, such as "zh" for Chinese or "en" for English; and a confidence score that indicates the credibility of the judgment result, with a value between 0 and 1.

[0059] To ensure the accuracy of language identification and the robustness of the system, this embodiment employs a confidence-based decision-making strategy. The system presets a confidence threshold, for example, 0.7. When the confidence score returned by the detection is greater than or equal to this threshold, the system adopts the detection result as the language of the current dialogue. If the confidence score is lower than this threshold, or if the language detection service times out or becomes unavailable, the system will fall back to a preset default language (e.g., "zh") to ensure that the interaction flow is not interrupted.

[0060] To optimize system performance and reduce unnecessary computational overhead, this embodiment also introduces a session-level caching mechanism. Once a user's preferred language is confirmed in a session, this language information will be written into the session metadata. In subsequent interaction rounds, the system will prioritize using this cached language information, thereby avoiding repeated detection for every message from the same user. In addition, to cope with high-concurrency request scenarios, the system supports batch merging and asynchronously calling multiple detection requests within a short period of time to improve overall throughput.

[0061] The language information determined by the language detection step will be included as part of the session metadata and will be processed along with the identified text in subsequent steps. This language information will guide subsequent processes, such as selecting the language of prompt words used when generating dialogue using the cloud model, and determining whether the system's output text needs to be processed by a translation module before being fed back to the front end.

[0062] See attached document Figure 2For step S2, select the most recent rounds of dialogue from the records containing multiple rounds of dialogue in chronological order, such as the most recent 3 to 5 rounds. Each round of dialogue contains information such as roles, content, timestamps and speaker identities. After verification and splicing, this information is constructed into an ordered dialogue history sequence.

[0063] Subsequently, the system calls a locally deployed lightweight language model to process the dialogue history sequence. To ensure that the model can complete the extraction task accurately and reliably, the system provides preset, mandatory system prompts before calling the model.

[0064] The system's prompts strictly limit the model's output behavior, specifically including:

[0065] First, the model must return a single JSON object without any additional natural language explanations or descriptions.

[0066] Second, the JSON object must contain a predefined and complete set of fields;

[0067] 3. For fields that are not mentioned in the conversation history or cannot be inferred, their corresponding values ​​must be set to null.

[0068] During local inference, the system prompts and the processed dialogue history sequence are combined into an input message and submitted to the local lightweight language model. This local lightweight language model is preferably a quantized model and is invoked through efficient inference frameworks such as vLLM to ensure low-latency response on edge devices. The system sets parameters such as inference timeout and temperature to balance the stability and diversity of inference.

[0069] After receiving the raw text output from the language model, a format validation and error correction process is executed. This process first removes any code block wrapping marks that may exist in the model output using preset rules, and then accurately extracts the JSON string from the text using regular expressions and other methods, and corrects common format errors, such as extra commas at the end of arrays or objects.

[0070] After the repair is complete, the system attempts to parse the string. If parsing is successful, it proceeds to the next step. If parsing fails, it indicates a serious error in the model's output format. In this case, the system generates a predefined default JSON structure with all field values ​​set to null. This empty structure is passed to subsequent modules, and the system triggers a re-questioning or user-friendly prompt strategy to guide the user to provide clearer information, thus ensuring the robustness of the entire interaction process.

[0071] For successfully parsed JSON objects, the field names may be natural languages, such as an object containing keys like "Order Status" and "Drink Variety". The system will further perform field mapping and normalization operations to convert it into an internally unified and standardized field structure. For example, map "Order Status" to "state", "Drink Variety" to "variety", "Specific Drink" to "specific_variety", etc. This standardized JSON object contains fields such as "state", "variety", "specific_variety", "alcohol", "taste", "sweet", and "recommend_number", providing regular and reliable input data for subsequent Retrieval-Augmented Generation (RAG) and order placement decision-making steps.

[0072] By locally combining dialogue history management, constrained prompt engineering, lightweight model inference, and a robust output post-processing and verification mechanism, the goal of efficiently and accurately extracting structured intent from multi-turn natural language conversations while ensuring response timeliness and user privacy is achieved. At the same time, the system will record information such as processing time consumption and the model's raw output for subsequent performance monitoring and offline optimization.

[0073] In step S3, based on the constraint conditions in the JSON object, a preliminary entity screening is performed in the local knowledge graph, which consists of entity triples and an attribute table. The system uses fields such as "variety", "alcohol", "taste", "sweet" in the JSON object and their corresponding values to filter entities in the knowledge graph.

[0074] During this process, the system gives priority to processing the "specific_variety" field, which is the specific drink name clearly mentioned by the user. The system will attempt to perform an exact match or alias match for this name in the knowledge graph. If the name uniquely and accurately hits an entity, the system can directly return the complete information of that entity, thus short-circuiting the subsequent complex retrieval process. If "specific_variety" does not hit any entity or hits multiple candidate entities, the subsequent semantic retrieval process will continue.

[0075] After completing the preliminary screening, the system obtains a set of candidate entities that meet the constraints. For this set of candidate entities, relevant text chunks need to be retrieved from a pre-built vector database. In this embodiment, when indexing the text chunks, their IDs are designed in a format that includes the entity name prefix, such as "Pearl Milk Tea_chunk1". The system filters out all text chunk IDs with the corresponding entity name as the prefix based on the set of candidate entities. If no corresponding text chunk IDs are found, all the data of the candidate entities in the knowledge graph attribute table will be directly returned.

[0076] The system merges the most recent rounds of dialogue chronologically to form a coherent query text. To enhance semantic accuracy, this query text prioritizes retaining the user's own statements, supplemented by the assistant's responses as context. A locally deployed embedding model, such as one based on the SentenceTransformer architecture, is then invoked to convert the query text into a high-dimensional embedding vector. To ensure consistency in subsequent similarity calculations, the system performs L2 norm normalization on the generated query vector. Similarly, all text block vectors stored in the vector database undergo the same normalization process.

[0077] The system performs similarity queries in a locally deployed vector database (such as ChromaDB). The system submits the normalized query vector to the database, along with parameters such as the expected number of beverages to be returned and the set of text block IDs to be retrieved, in order to perform local retrieval within a limited entity scope.

[0078] After the vector database returns the search results, the system performs post-processing operations, applying a preset similarity threshold to filter the results and remove text blocks with low relevance. The process of determining this similarity threshold is offline: the manually labeled "query-related text" data pairs are vectorized, the similarity scores of these data pairs are calculated, and the precision and recall rates under multiple candidate thresholds are tested. The threshold that ensures the recall rate is not lower than a certain standard (e.g., 85%) and has the highest precision rate is selected as the configuration value for the online service.

[0079] The filtered text blocks need to be aggregated and integrated into a more readable entity summary. The system groups the retrieved text blocks according to their respective entities, maintaining the original sequence number order of the text blocks within each entity. For multiple text blocks belonging to the same entity, a merge and deduplication operation is performed. This operation specifically includes: removing common descriptive prefixes from the text blocks; splitting the text content according to sentence boundaries; deduplicating content at the sentence level; and concatenating the deduplicated sentences into coherent text according to the original sequence number of the text blocks. The generated entity summary, which includes information such as beverage name, category, alcohol content, taste, price, and a brief description, will serve as context for subsequent steps.

[0080] If the input JSON in step S2 specifies a specific_variety and multiple candidates are matched during the initial screening, a reordering step will be performed. The embedding vector of specific_variety and the embedding vector of each candidate entity name will be calculated respectively, and the candidate list will be reordered based on cosine similarity, placing the entity that is closest to the user's intent first, thereby improving the fault tolerance for naming similarities or aliases.

[0081] Step S4 determines whether the conditions for local ordering are met based on the structured JSON object. The conditions for local ordering are: the intent recognition result clearly indicates that the user has the intent to confirm the order, and a single, specific product (specific_variety) has been parsed out, and the specific product is confirmed to be available for sale in the local product database.

[0082] When the above conditions are met, the local order placement process is executed first, performing inventory checks and price verifications on the matched products. If the verification passes, the system will construct order metadata containing information such as product ID, price, and timestamp, and send a confirmation command to the order execution system via a local communication protocol (such as Open Voice Control (OSC) or User Datagram Protocol (UDP)). The system records in detail the conditions that triggered the order, product matching details, verification results, and command issuance response for subsequent information backtracking and auditing. If the inventory or price verification fails, the system will generate a corresponding business prompt (such as "Insufficient Inventory") and redirect the processing flow to the next cloud processing logic.

[0083] If the conditions for placing an order locally are not met, such as when the user's intention is to make an exploratory inquiry, require personalized recommendations, or does not explicitly specify available products, the system executes cloud processing logic. In this logic, input messages used to request the cloud model are encapsulated. The structure of these messages is carefully designed: the first element is a system prompt word to guide the cloud model's behavior; recent dialogue history is appended chronologically, including the user's input and the system's historical responses; and a generated entity summary (RAG retrieval content) related to the current dialogue is appended to the end of the message.

[0084] After encapsulation, the input message is sent to the cloud-based dialogue processing module, which calls the cloud-based large language model to perform natural language processing and policy decision-making in order to generate a natural language response that conforms to the current dialogue context.

[0085] When receiving responses from the cloud model, this embodiment employs a streaming reception and processing mechanism to enhance user experience. The cloud model returns responses in the form of text fragments. After receiving each fragment, the server caches it and performs an integrity check. This integrity check aims to determine whether the currently cached text constitutes a complete sentence or semantic unit. For example, it checks whether the ending is a period, question mark, or other punctuation mark, and may combine this with a text length threshold. When it is determined that a complete sentence or segment has been formed, the text segment is summarized and pushed to the front end.

[0086] Before being pushed to the front end, the system performs necessary translation steps. The system compares the language of the cloud-generated text with the user's preferred language in the conversation. If the two are inconsistent, the system will call the local translation module on the server to translate the generated text in real time to ensure that the language of the reply finally received by the user is consistent with the language they use.

[0087] To ensure the stable operation of the system, this embodiment also includes fault tolerance and rollback strategies. During the process of calling the cloud-generated interface, if a network timeout or interface return error occurs, the system will record the error event and return a locally preset downgraded response to the user, such as "Please try again later" or cached general recommendation information. If the language detection or translation module fails, the system will roll back to the preset default language (e.g., "zh") for the response and record the event for offline analysis and model improvement.

[0088] Step S5: The final natural language response text is sent to the front-end multimodal terminal. After receiving the text, the speech synthesis module inside the terminal converts it into waveform audio data and plays it back to the user in natural speech through a playback device, completing the interactive feedback process.

[0089] The server performs persistent storage operations on the session data. In order to support subsequent business statistics, behavior auditing and offline optimization training of the model, the key data generated in this interaction is recorded. This data is saved in a structured manner, preferably stored in a local CSV file or database.

[0090] The persisted entries contain several key fields, such as session ID, device ID, timestamp, user input text, system reply text, structured intent JSON, RAG retrieval source, and order ID and order status at the time of the transaction. To protect data privacy and conserve storage resources, the system sets a fixed retention period for this persisted data, such as one week.

[0091] In addition to front-end feedback and data persistence, it also includes a key subsystem that executes asynchronously in the background: periodic updates and atomic replacements of knowledge resources.

[0092] Running automatically at a preset cycle (e.g., every 10 minutes), this system requires real-time synchronization of highly dynamic information such as inventory status and price changes. Too low an update frequency may lead to ineffective recommendations, while too high a frequency of real-time requests will increase server CPU, I / O load, and database pressure. Therefore, this embodiment preferably uses a polling interval of minutes (e.g., 10 minutes) to maximize the real-time performance and accuracy of the local knowledge base data without excessively consuming system computing and network resources.

[0093] The system retrieves the latest product information from a remote data source. After obtaining the raw data, it performs a consistency check against existing local data (e.g., comparing data fingerprints, hash values, or version timestamps). Only when the check indicates inconsistency between the remote and local data does the system determine that a substantial change has occurred and trigger the subsequent update process; if the data remains consistent, the task is terminated to conserve resources. After determining that an update is necessary, the system generates a completely new knowledge graph file and vector database index in a temporary directory based on the newly retrieved data.

[0094] Once all validation items pass, the newly generated resource is marked as ready and can enter the replacement phase. If any validation item fails, the system will abandon the update and trigger the logic for re-fetching and generating the resource.

[0095] Resource replacement is a high-risk process that must be performed under strict conditions to ensure data consistency and service continuity. In this embodiment, performing an atomic replacement operation requires all of the following preconditions to be met: both the update request initiated by the client and the response from the server must be verified through key-based signature verification; the temporary resource has passed the aforementioned integrity check; all associated front-end devices are confirmed to be in a short-term offline or preset maintenance state; and the system has successfully acquired a global operation lock to prevent concurrent operation conflicts.

[0096] After all conditions are met, the system performs an atomic replacement. This process leverages the atomicity of file renaming or moving operations provided by the operating system to instantly replace the official online resource with the new resource in the temporary directory. The system uses a version number management mechanism to back up the old version of the resource before replacement and retains the most recent backup versions.

[0097] After the replacement is completed, a quick functional verification is performed on the newly launched resources. If the verification fails, the system will immediately trigger an automatic rollback mechanism to restore the resources to their state before the replacement using the backup version and record detailed error logs. The entire replacement and rollback process is carried out under the protection of a global operation lock, ensuring the concurrent security of the operation.

[0098] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A voice interaction method based on retrieval-enhanced generation and multi-model collaboration, characterized in that, Includes the following steps: S1. The front-end multimodal terminal identifies the speaker's identity, performs noise reduction and endpoint detection on the collected speech, and then generates the current round of dialogue text through speech recognition. S2. The generated current round dialogue text is appended to the historical dialogue. The updated complete dialogue history is constructed as input. The updated complete dialogue history is analyzed using a local lightweight language model. Fields related to ordering or recommendation are extracted from it, and a structured JSON object containing the ordering or recommendation fields is generated. S3. Using the fields and values ​​in the structured JSON object as search constraints, candidate entities that meet the constraints are selected in the local knowledge base or knowledge graph. A corresponding set of text blocks is constructed for the candidate entities. Semantic search is performed in the vector search module. Multiple text blocks belonging to the same entity are merged and deduplicated in order to generate an entity summary for decision-making. S4. Based on the user intent in the structured JSON object, make a decision: if it is determined to confirm the order and a salable product item is matched, then issue a confirmation instruction locally; otherwise, encapsulate the entity summary and the updated complete dialogue history and send them to the cloud model for natural language processing and strategy decision-making to generate the final natural language response text. S5. The natural language response text is given to the front end for speech synthesis and playback, and the structured JSON object and the final natural language response text are persistently stored for statistics and backtracking. The remote product data is periodically pulled to build a temporary knowledge graph and vector library, and the local official resources are updated by atomic replacement.

2. The voice interaction method based on retrieval enhancement generation and multi-model collaboration according to claim 1, characterized in that, In step S2, the step of generating a structured JSON object containing order or recommendation fields further includes: The JSON results returned by the successful analysis of the lightweight language model are subjected to format validation and fault tolerance repair. The fault tolerance repair operation includes removing code block wrapping marks and repairing trailing commas. When the lightweight language model analysis fails, a preset empty structure is returned and a re-questioning strategy is triggered.

3. The voice interaction method based on retrieval enhancement generation and multi-model collaboration according to claim 1, characterized in that, Step S3 further includes: When filtering out candidate entities that meet the constraints, fuzzy string matching based on distance or semantic similarity is used, and it is further corrected by combining an alias mapping table. When a specific name is specified in the structured JSON object, matching is performed first by the specific name or alias. If multiple candidates are matched, they are reordered according to the vector similarity between the specific name and the candidate names. For multiple matched candidates, remove the descriptive prefixes from the text blocks, deduplicate the content by sentence, and concatenate them into continuous text according to the original sequence number of the text blocks.

4. The voice interaction method based on retrieval enhancement generation and multi-model collaboration according to claim 1, characterized in that, The process of generating the final natural language response text by the cloud model supports streaming output. The server performs integrity checks on the streaming results according to sentence or paragraph boundaries and forwards them in batches, and translates them according to the user's language before forwarding. The atomic replacement steps include replacing the official resource with a temporary path and atomic renaming or moving strategy, provided that consistency protection and operation locks are satisfied, and retaining backups before and after the replacement to support rollback.

5. The voice interaction method based on retrieval enhancement generation and multi-model collaboration according to claim 1, characterized in that, The method further includes: The front-end multimodal terminal triggers microphone wake-up based on face detection results and uses multimodal noise reduction technology that combines microphone array and visual information to suppress environmental noise; Before collecting biometric information, user authorization should be obtained, sensitive data should be encrypted and stored, and data retention and deletion mechanisms should be set up.

6. A system for a voice interaction method based on retrieval-enhanced generation and multi-model collaboration, used to implement the voice interaction method based on retrieval-enhanced generation and multi-model collaboration as described in any one of claims 1-5, characterized in that, include: The front-end multimodal terminal module is used for user face detection and recognition, and triggers microphone wake-up based on the face detection results. During voice interaction, multimodal noise reduction technology is used to suppress environmental noise in real time, and completes high-fidelity voice acquisition, speech recognition and speech synthesis processing. The local multi-turn intent recognition module is used to extract order or recommendation fields from the historical dialogue information based on the recent dialogue history using a lightweight language model and return them as structured JSON. The local retrieval enhancement generation module includes an embedding generator, a vector database, and a knowledge graph. It is used to filter candidate beverages by category, alcohol content, or sugar content knowledge graph constraints, and to retrieve relevant text blocks from the vector database based on user dialogue history. Multiple text blocks of the same entity are merged sequentially and deduplicated to generate entity summaries. The cloud-based dialogue processing module is used to receive historical dialogue information and retrieved beverage data, call the cloud model for natural language processing, and return the data in a streaming manner. The order confirmation module is used to perform inventory checks, price verifications, and issue an order confirmation instruction when it detects that a user has confirmed their intention to place an order and matches available beverages. The offline update and atomic replacement module is used to atomically replace temporarily generated knowledge graphs, attribute tables, and vector databases with official resources when the device is temporarily offline and protected by an operation lock.

7. The system based on retrieval enhancement generation and multi-model collaboration for voice interaction according to claim 6, characterized in that, The local multi-turn intent recognition module is further used for: Before returning the structured JSON, perform format validation and error correction on the structured JSON; The structured JSON contains fields such as state, variety, specific_variety, alcohol, taste, sweet, and recommend_number, and fields that do not appear in the user's message are represented by null values.

8. The system based on retrieval-enhanced generation and multi-model collaboration for voice interaction according to claim 6, characterized in that, The local search enhancement generation module includes: An embedding generator, wherein the embedding generator is a model based on sentence vectors or semantic embedding; A vector retrieval unit, wherein the vector retrieval unit is a vector database that supports similarity queries; The name fuzzy matching unit uses fuzzy string matching based on distance similarity, combined with an alias mapping table and sets a similarity threshold to tolerate spelling or pronunciation differences.

9. The system based on retrieval enhancement generation and multi-model collaboration for voice interaction according to claim 6, characterized in that, The cloud-based dialogue processing module includes a language switching and local translation unit, which is used to provide multilingual responses when the language of the user and the generated result are inconsistent. When performing an offline update operation, the offline update and atomic replacement module requires the client to sign the request using pre-allocated credentials and a key-based signature. The payload returned by the server includes both a timestamp and the server's signature. After receiving the response, the client must verify the response signature and timestamp.

10. The system based on retrieval-enhanced generation and multi-model collaboration for voice interaction according to claim 6, characterized in that, The multimodal noise reduction technology is achieved through a multimodal neural network architecture that combines convolutional neural networks and recurrent neural networks; The order confirmation module is used to send order information through a local communication protocol and record order metadata and session history for retrospective purposes.

Images