An edge-side local large model interaction method

Abstract

The application discloses an edge side local large model interaction method, relates to the technical field of edge computing and natural language processing, and comprises the following steps: collecting a face image of a user to call recent historical interaction records of the user; meanwhile, after converting a voice stream of the user into current text, performing intention extraction, and controlling a routing distribution unit to perform short circuit judgment according to an intention recognition result, an attribute label route is generated; based on the attribute label route, according to the recent historical interaction records and the current text, a local knowledge base is searched to recall a segment, a Prompt context with a constraint instruction is constructed, a user attribute label is acquired, adaptive local LLM inference is performed, and after fact verification, super library verification and language verification are performed in sequence, a streaming output text segment is generated, and a voice conversion and playing operation is performed by using a lightweight acoustic model on an edge computing device. The application provides a new technical inspiration for a local large model interaction scheme.

Description

Technical Field

[0001] This invention relates to the fields of edge computing and natural language processing technology, and more specifically, to an interaction method for a local large model on the edge side. Background Technology

[0002] In retail and hotel scenarios, edge devices such as self-service ordering machines and service robots need to have natural interaction capabilities. Existing local large language model deployment solutions have many shortcomings: edge devices have limited computing power and memory resources, local inference is prone to OOM and response latency, and separate models for intent recognition and compliance verification bring double redundancy in computing power and storage; the model is prone to hallucinations, which may lead to factual and compliance risks such as recommending out-of-stock products and pushing alcohol to minors; when high load causes hardware overheating, simply reducing the frequency and restarting the process will directly interrupt the interaction; multimodal serial processing generates a large number of temporary objects, and uncontrollable GC can cause stuttering and memory overflow; at the same time, there is a lack of a safe and stable model self-evolution mechanism, making it difficult to meet the usage requirements of lightweight, highly reliable, and uninterrupted service. Summary of the Invention

[0003] To address the aforementioned issues, the present invention aims to provide an interaction method for a large local model on the edge side. This method aims to reduce computational overhead by reusing the core algorithm components of the edge device, improve the factual accuracy, compliance, and language consistency of the large model's output content, ensure uninterrupted interaction services through strategic degradation when the hardware overheats, avoid memory overflow and GC stuttering issues from the underlying layer, and ensure the safety and stability of the model's local self-evolution process.

[0004] To achieve the above technical objectives, this application provides an interaction method for a local large model on the edge side, comprising the following steps: The system collects the user's facial image to retrieve the user's recent historical interaction records; at the same time, it converts the collected user's voice stream into current text, extracts intent, and controls the routing distribution unit to perform short-circuit judgment based on the intent recognition result in order to generate attribute label routes. Based on attribute tag routing, and using recent historical interaction records and current text as a basis, retrieve and recall fragments from the local knowledge base to construct a Prompt context with constraint instructions; Based on the Prompt context, by obtaining user attribute tags, adaptive local LLM inference is performed, and then fact verification, super library verification, and language verification are performed in sequence to generate streaming text fragments. The lightweight acoustic model on the edge computing device is then used for speech conversion and playback.

[0005] Preferably, when acquiring a user's facial image and voice stream, a dual-channel wake-up mechanism of visual wake-up and voice wake-up is deployed locally. Through the wake-up operation, the user's facial image and voice stream are acquired. The visual channel performs face detection and lip motion capture using a lightweight CV model; the voice channel continuously extracts speech acoustic features using a low-power audio engine.

[0006] Preferably, when performing short-circuit judgment, if the identified business is a preset first type of business, and the business variables are in the standard database and have passed inventory verification, and the user attribute tags do not trigger compliance interception, then an execution instruction is directly generated and a voucher is pushed, and the process short-circuit ends; otherwise, execution is blocked, and the recent historical interaction records, current text, and attribute tags are routed to the next process to obtain the Prompt context with constraint instructions.

[0007] Preferably, when constructing the Prompt context with constraint instructions, a local lightweight semantic matching algorithm is used based on attribute tag routing to retrieve relevant fragments from the local knowledge base according to recent historical interaction records and the current text; and the retrieved fragments are concatenated with recent historical interaction records and the current text to construct the Prompt context with constraint instructions.

[0008] Heterogeneous computing units include, but are not limited to, at least one of a neural network processor (NPU), a brain processor (BPU), a central processing unit (CPU), or a graphics processing unit (GPU).

[0009] Preferably, during the retrieval process, the load and memory availability of the heterogeneous computing units are obtained in real time, and the Top-K value of the retrieval recall is dynamically calculated based on the load and memory availability. According to the Top-K value, a first threshold is set when resources are sufficient, and the second threshold is adaptively downgraded when the load is high.

[0010] Preferably, during adaptive local LLM inference, the Prompt context and user attribute tags are fed into the local lightweight LLM model for inference. The model is generated based on the execution conditions of the user attribute tags. In this process, the load and memory availability of the heterogeneous computing units are obtained in real time, and a target inference model is dynamically selected from the candidate model library based on the load and memory availability. When resources are sufficient, the local first model is set, and under high load, it is adaptively downgraded to the local second model. If the operation is still laggy, a generation limit downgrade is adopted: the length of the LLM's KVCache is truncated, and the maximum token limit for a single generation is reduced.

[0011] Preferably, after performing adaptive local LLM inference, fact verification is performed: the local LLM-generated text is cross-validated with the knowledge fragments recalled by the local RAG, and factual illusions are marked and corrected. Through super-database verification, the local LLM output text is sent back to the local global lightweight intent parsing engine to extract the product business variables mentioned therein, compare them with the standard product database, and if a product is found not in the database, an interception is triggered, and it is corrected or regenerated. Language validation checks the consistency between the local LLM response language and the user's input language. If they do not match, the local lightweight translation model is invoked for alignment.

[0012] Preferably, during speech conversion playback, a text segment is received from the streaming output of the local large language model. When the text segment meets the preset sentence segmentation conditions, it is used as the current text to be synthesized. The synthesized text is processed in the front end, including text regularization, polyphonic character disambiguation and prosody prediction, to generate phoneme sequences and corresponding prosodic features; Phoneme sequences and prosodic features are input into a lightweight acoustic model pre-deployed on an edge computing device to generate Mel spectrum features. The lightweight acoustic model is a lightweight neural network model that has undergone knowledge distillation, model pruning, or quantization to adapt to the computing power limitations of the edge computing device. Mel spectral features are converted into time-domain audio waveform data using a lightweight vocoder; Write time-domain audio waveform data to the audio buffer and control the audio playback thread to read and play audio data blocks in a streaming manner.

[0013] Preferably, when performing local large model interaction on the edge side, a DMA-based zero-copy mechanism is used to construct a shared memory space to achieve cross-module data transfer and avoid data copying involving the central processing unit (CPU); A ring-shaped asynchronous buffer channel consisting of LLM streaming output and TTS block consumption is constructed. When the token output by LLM reaches a sentence, it is directly supplied to the TTS module for block-synthesized speech, so as to realize simultaneous speech output and synthesis. After a single interactive task is completed, the default garbage collection mechanism of the operating system is bypassed, and the underlying explicit release instruction is called. Based on the pre-allocated memory block linked list and reference counting mechanism, the storage resource pool, inference context and RAG vector cache of the heterogeneous computing unit are forcibly cleared.

[0014] Preferably, during the execution of local large-scale model interaction on the edge side, the underlying daemon process collects data in real time on the utilization rate of heterogeneous computing units, remaining memory, device temperature, memory fragmentation rate, and task queue length of the device, and calculates a comprehensive load index through weighted average. When the comprehensive load index exceeds a safety threshold, the load is progressively downgraded in a fixed order from most to least impactful on computing power. If the index falls back to a safe range after each downgrade, the subsequent downgrade stops; otherwise, it proceeds to the next level. A privacy-de-identifying sandbox is built on the edge computing device to filter out abnormal interaction data that fails the triple verification of facts, super library and language, as well as high-value difficult case data containing user negative feedback indicators; After the selected data is localized and anonymized in a privacy sandbox, it is transmitted to the central server through a secure encrypted channel when the data reaches a certain quantity or when a certain amount of time has passed since the last model update. Using the central server, the received de-identified data is subjected to lightweight incremental training using low-rank adaptive technology. The incremental weights generated during training are then compressed using 8-bit quantization to generate model incremental weights that are adapted to the computing power of edge devices, and then distributed to the edge computing devices.

[0015] The present invention discloses the following technical effects: This invention comprehensively addresses various pain points in the deployment of large-scale edge models, achieving significant technical gains in five dimensions: resource consumption, content compliance, interactive experience, system stability, and device adaptation. It is particularly suitable for highly compliant business scenarios such as retail, and builds an efficient, secure, stable, and low-power edge AI operation capability.

[0016] In terms of resource utilization and recognition accuracy, this invention achieves global reuse optimization of NLP capabilities, effectively avoids resource waste caused by repeated component construction, and significantly reduces storage occupation and computing power consumption of edge devices; at the same time, it unifies the front-end and back-end business variable recognition logic, completely solves the problem of misunderstanding between the front-end and back-end, significantly improves the accuracy of business entity recognition and content verification, and ensures the consistency and accuracy of business judgment.

[0017] In terms of content security and compliance, this invention establishes a local triple closed-loop verification system to effectively suppress the illusion problem of content generated by large models, accurately complete the compliance verification of business entities and the localization adaptation of content language, and comprehensively ensure the compliance, authenticity and adaptability of AI-generated content on the edge, meeting the industry's stringent compliance control requirements.

[0018] In terms of interactive response efficiency, the dual-channel fusion wake-up method designed in this invention greatly improves the success rate and stability of device wake-up in complex environments; the hierarchical direct path routing mechanism avoids the large model scheduling overhead of high-frequency simple services, effectively shortens service response latency, significantly reduces the overall computing power consumption, and improves user interaction experience and service processing efficiency.

[0019] In terms of operational stability and adaptability, the step-by-step adaptive degradation strategy designed in this invention breaks away from the drawbacks of traditional extensive degradation. While ensuring that the system runs without downtime or interruption, it maximizes the preservation of the intelligence of AI dialogue, balancing stability and product experience. The exclusive streaming asynchronous memory scheduling mechanism completely solves the core problems of high latency, memory overflow, and memory fragmentation in the multimodal processing of edge devices, significantly reducing the hardware entry threshold and ensuring that ordinary edge devices with 8GB of memory can run various AI services stably and smoothly. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the method described in this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] like Figure 1As shown, this invention provides an interaction method for a local large-scale model on the edge side. This method relies on visual and voice dual channels to achieve low false wake-up. After wake-up, it collects user face and voice data to complete identity matching, intent parsing, and user compliance determination. It achieves layered routing by using a lightweight intent engine and uses compliant standard instructions for direct execution to achieve low latency. For complex scenarios, it constructs constraint prompt context through hardware load-aware adaptive RAG retrieval. It completes inference based on the edge-side quantized large-scale model and avoids output errors through a triple closed-loop verification of facts, super library, and language. At the same time, it builds a streaming TTS speech synthesis pipeline to achieve parallel processing of text and audio. At the underlying level, it solves latency and stuttering problems through zero-copy communication, asynchronous pipeline, and explicit memory scheduling. It is combined with a five-dimensional hardware monitoring and progressive adaptive degradation mechanism that includes heterogeneous computing unit utilization, memory remaining amount, device temperature, memory fragmentation rate, and task queue length to ensure interaction stability. Finally, through the edge-cloud collaboration mode, it uploads difficult interaction data in an anonymized manner, and uses LoRA technology to incrementally iterate the model, using the device idle window to complete the smooth update and self-healing of the lightweight model.

[0024] Step S100 Local Multimodal Dual-Channel Wake-up and Context Acquisition In one implementation, a dual-channel wake-up mechanism consisting of visual wake-up and voice wake-up is deployed locally.

[0025] For example, the visual channel performs face detection and lip motion capture (such as mouth opening) using a lightweight CV model; the voice channel continuously extracts speech acoustic features using a low-power audio engine; when either channel triggers a wake-up threshold, the wake-up command is converted into speech and played.

[0026] For example, the present invention effectively reduces the false wake-up rate in noisy store environments through dual-channel cross-validation, and successfully wakes up and obtains complete interaction requests.

[0027] For example, upon being woken up, a high-definition facial image and a complete voice stream are simultaneously acquired; the user ID is matched through face recognition and recent historical interaction records (QA) are retrieved; the voice stream is converted into current text (current Q) through an ASR model; age group features are extracted from the facial image to generate user attribute labels (e.g., whether a specific age threshold is met, "user is an adult" or "user is a minor").

[0028] Step S200 is based on the front-end call of the local global intent parsing engine, namely intent extraction and hierarchical routing.

[0029] In one implementation, the current Q and historical QA are input together into a local global lightweight intent parsing engine, which integrates a local multi-scale model library (rule-based algorithms, BERT-LSTM Large / Base) and dynamically selects the model accuracy based on the current device load.

[0030] For example, the engine outputs: customer intent category and business variables (such as product attribute parameters, such as beverage name, sugar content, alcohol content, etc.).

[0031] For example, the routing strategy is as follows: The routing distribution unit performs a short-circuit judgment: If it is identified as "Type 1 business instruction (to place an order)", and the business variables are in the standard database, and the inventory verification is passed, and the user attribute tag does not trigger compliance interception (such as the mutual exclusion between minors and specific product attributes, minors cannot order alcoholic beverages), then the execution instruction is directly generated and the voucher is pushed, and the process short-circuit ends (zero call to the large model, extremely low latency); otherwise, the execution is blocked, and the current Q, historical QA and attribute tag are routed to step S300.

[0032] Step S300 is based on hardware load-aware local adaptive retrieval enhancement (RAG).

[0033] In one implementation, after receiving the attribute tag route from step S200, a local lightweight semantic matching algorithm (such as lightweight vector indexing + BM25 hybrid retrieval) is used to recall relevant fragments for the current Q + historical QA in the local knowledge base; by acquiring the load and memory availability of heterogeneous computing units in real time, the number of retrieved fragments (Top-K value) is dynamically calculated, and a first threshold (such as K=10) is set when resources are sufficient, and the threshold is adaptively downgraded to a second threshold (such as K=5) when the load is high; the recalled fragments are concatenated with the current Q and historical QA to construct a Prompt context with constraint instructions.

[0034] Step S400: Adaptive local LLM inference and post-local triple closed-loop verification.

[0035] In one implementation, the received Prompt context and user attribute tags from step S300 are sent to a local lightweight LLM (such as the Qwen2.5-7B / 1.5B quantized model) for inference. The model is generated based on the user attribute tags and execution conditions (such as internal logic blocking alcoholic beverage recommendations when minors are included). The model is dynamically selected by acquiring the load and memory availability of the heterogeneous computing units in real time. When resources are sufficient, a local first model (such as Qwen2.5-7B) is set, and under high load, it is adaptively downgraded to a local second model (such as Qwen2.5-1.5B). If the operation is still laggy, a generation limit downgrade is adopted: the length of the LLM's KVCache is truncated, and the maximum token limit for a single generation is reduced. For example, after the LLM output, triple verification is performed sequentially: ① Fact Verification: Cross-validate the local LLM-generated text with the knowledge fragments recalled by the local RAG, and mark and correct factual illusions: Slice the knowledge fragments recalled by the local RAG according to beverage type, and verify each slice with the local LLM-generated text to determine the relationship of "contradictory", "neutral", or "implied". If it is a "contradictory" relationship, then correct the factual illusion.

[0036] ②Super Library Verification (Second Call to Global Intent Engine): The local LLM output text is sent back to the local global lightweight intent parsing engine (Note: Here, the model instance is kept in memory to achieve inference reuse without reloading the model). The product business variables mentioned are extracted and compared with the standard product library. If a product is found not in the library, an interception is triggered, and it is corrected or regenerated. ③ Language Verification: The local lightweight language recognition model checks the consistency between the local LLM response language and the user's input language. If they do not match, the local lightweight translation model is used for alignment. For beverage name translation, the standard names from the database are used.

[0037] For example, the LLM response can only proceed to the final output stage after all three layers of verification have passed.

[0038] Step S500: Design a streaming text-to-speech mechanism.

[0039] In one embodiment, a text segment is received from the streaming output of a local large language model. When the text segment meets the preset sentence segmentation conditions (such as encountering punctuation marks or reaching a preset character threshold), it is used as the current text to be synthesized. The synthesized text is processed in the front end, including text regularization, polyphonic character disambiguation and prosody prediction, to generate phoneme sequences and corresponding prosodic features; Phoneme sequences and prosodic features are input into a lightweight acoustic model pre-deployed on an edge computing device to generate Mel spectrum features; the lightweight acoustic model is a lightweight neural network model that has undergone knowledge distillation, model pruning, or quantization to adapt to the computing power limitations of edge computing devices; Mel spectral features are converted into time-domain audio waveform data using a lightweight vocoder; Time-domain audio waveform data is written to the audio buffer, and the audio playback thread is controlled to read and play audio data blocks in a streaming manner, realizing parallel pipelined processing of large language model text generation, TTS speech synthesis and audio playback.

[0040] Step S600 addresses the design of edge-side streaming asynchronous inference and explicit memory scheduling.

[0041] In one implementation, this step serves as the underlying execution mechanism for steps S100 to S500, without altering the business logic, but it can completely resolve memory and latency issues: Zero-copy cross-module communication uses a DMA (Direct Memory Access)-based zero-copy mechanism to build a shared memory space, enabling cross-module data transfer and avoiding data copying involving the central processing unit (CPU).

[0042] The streaming-speech asynchronous pipeline constructs a ring-buffer channel of "LLM streaming output - TTS block consumption". When the token output by LLM reaches a sentence, it is directly supplied to the TTS module for block synthesis of speech, realizing simultaneous speech output and synthesis, and reducing the first sound delay by more than 50%.

[0043] Explicit memory pool management bypasses the operating system's default garbage collection mechanism after a single interactive task ends, calls the underlying explicit release instruction, and forcibly clears the storage resource pool, inference context, and RAG vector cache of the heterogeneous computing unit based on the pre-allocated memory block linked list and reference counting mechanism, thus preventing uncontrollable lag caused by asynchronous garbage collection.

[0044] Step S700 edge hardware state awareness and five-dimensional progressive adaptive degradation (guardian mechanism).

[0045] In one implementation, the underlying daemon process collects real-time data on the utilization rate of heterogeneous computing units on the device, remaining memory, device temperature, memory fragmentation rate, and task queue length, and calculates a comprehensive load index through weighted averages. When the comprehensive load index exceeds a safety threshold, the load is progressively downgraded in a fixed order from heaviest to lightest impact on computing power. If the index falls back to a safe range (i.e., below the safety threshold) after each downgrade, subsequent downgrades stop; otherwise, the process proceeds to the next level. 1) Downgrade large model parameters: Switch the LLM version (e.g., downgrade from Qwen2.5-7B to Qwen2.5-1.5B); 2) Intent model downgrade: Downgrade the intent engine from BERT-LSTM Large to Base, or downgrade it to a pure rule algorithm; 3) Generation limit downgrade: Truncate the length of the LLM's KVCache to reduce the maximum token limit for a single generation by the LLM; 4) Deep RAG Degradation: Reduce the number of Top-K values ​​in the RAG module from 10 to 5; 5) Audio rendering degradation: Reduce the audio output sampling rate or bit depth of the TTS module (e.g., 24kHz → 16kHz).

[0046] For example, this sequence ensures that core conversational functionality is preserved as much as possible, rather than crashing outright.

[0047] Step S800 end-center collaborative model adaptive evolution.

[0048] In one implementation, a privacy-de-identifying sandbox is built on the edge computing device to filter out abnormal interaction data that fails the triple verification of facts, super-database and language, as well as high-value difficult case data containing user negative feedback indicators (i.e. error correction instructions); After the selected data is localized and anonymized in a privacy sandbox, it is transmitted to the central server through a secure encrypted channel when the data reaches a certain quantity or a certain time has passed since the last model update.

[0049] The central server uses the received de-identified data to perform lightweight incremental training using low-rank adaptive (LoRA) technology, and performs 8-bit (q8) quantization compression on the incremental weights generated during training. Finally, it generates model incremental weights that are adapted to the computing power of edge devices and distributes them to the edge computing devices.

[0050] For example, the present invention automatically captures the period of low-peak time predicted by historical interaction traffic statistics, and the time when there are no new voice / visual wake-up requests for X consecutive minutes, as a non-interactive window.

[0051] For example, in order to minimize the memory usage on the client side, this invention automatically triggers a stop-and-reload mechanism based on a non-interactive window. This mechanism constructs a fully automatic update and fallback closed loop: receiving weights from the cloud → automatically pausing the inference service → securely writing the new weights to disk → automatically restarting the engine to load the new model → performing automated A / B verification → if the verification fails, automatically rolling back to the old weights. Through the design of this scheme, the problem of doubling the memory peak caused by the double buffering mechanism is completely avoided. Under the premise of zero additional memory overhead, a simple, safe and self-healing fully automated smooth model iteration is achieved.

[0052] Example 1: Scenario Setting 1 - Normal Ordering: This example uses a smart store shopping guide terminal deployed on a "sweet potato development board" (a typical edge computing device with limited computing power and memory). This terminal is equipped with a camera, microphone array, speaker, and temperature and humidity sensor to provide customers with product consultation and shopping guide services in physical stores.

[0053] The specific interaction and system scheduling process is as follows: Step 1: Multimodal dual-channel wake-up and context acquisition: A customer approaches the sales terminal and prepares to ask a question. The terminal's visual channel captures the customer's facial data and subtle lip movements (visual data), while the voice channel picks up the customer saying, "Give me a cup of milk tea" (voice data).

[0054] The system uses facial recognition to match the user's ID and retrieves recent interaction history (ID, QA. Since this is the customer's first interaction, QA is empty here). The speech stream is converted into the current text (current Q, "Give me a cup of milk tea") using an ASR model. Age group features are extracted from facial images to generate user attribute labels (currently "user is an adult").

[0055] S2: Front-end call based on global intent parsing engine: After being woken up, the global lightweight intent parsing module extracts the intent from the speech-transcribed text at the front end (device side).

[0056] The intent extraction result is: {'ORDER':['Ordering'],'TYPE':['Tea Drink'],'DRINK':[],'ALCOHOL':[],'SUGAR':[]} Since the current ORDER is not "order successful", the order cannot be placed successfully directly, triggering a "short-circuit routing" failure. It needs to be allowed to proceed to the backend large model for processing.

[0057] Meanwhile, the underlying daemon process of the hardware perception module reads device sensor and system status data to obtain the current five-dimensional hardware indicators: the current device temperature (such as CPU and BPU core temperature) is 47°C, the remaining memory is 29% (i.e., memory utilization rate is 71%), the heterogeneous computing unit utilization rate (specifically, the BPU utilization rate is 35% in this embodiment) is low, the memory fragmentation rate is 12%, and the task queue length is 2. Based on the above five-dimensional data, the system calculates a comprehensive load index through weighted calculation and determines that the current state is 'healthy resource state'.

[0058] Step S3: Hardware Load Aware Adaptive Retrieval Enhancement (RAG) For complex requests that have been granted permission, the Adaptive Search Enhancement (RAG) module sets the number of Top-K semantic matching searches to 10 based on the "healthy resource status".

[0059] During the process of retrieving the knowledge base and concatenating 10 text fragments into a Prompt, the system fully utilizes streaming asynchronous and explicit memory scheduling modules.

[0060] When data is transferred between the BPU and the CPU, zero-copy shared memory technology is used, and the data is transferred through a ring-shaped asynchronous buffer channel.

[0061] The entire process bypasses the operating system's automatic garbage collection (GC), with the explicit memory pool forcibly allocating and releasing memory blocks, completely eliminating memory fragmentation and random pauses, and ensuring extremely low latency in the retrieval and splicing process.

[0062] Step S4: Adaptive LLM Inference and Post-Triple Loop Validation (Second Invocation of Intent Engine) Based on the "health resource status", the module sets the lightweight large language model to a 7B quantization model.

[0063] The Prompt is fed into the 7B quantization model for conditional generation.

[0064] LLM begins streaming the text: "Recommended Jasmine Apple Tea, pure and light, NFC freshly squeezed juice, refreshing and delicious."

[0065] The triple-validation module first segments the generated text in the background according to sentences or by accumulating a certain length. After segmentation, it performs triple validation based on real-time facts, super-database, and language. All three validations pass.

[0066] Step S5: Streaming Text-to-Speech Mechanism The generated text fragments are fed into the streaming text-to-speech (TTS) module in real time. The TTS module synthesizes and plays the text simultaneously, allowing customers to hear the sound before the system fully generates a response, compressing the first word / first sound interaction delay to less than 1 second.

[0067] Step S8: Adaptive Evolution of the End-to-Center Collaborative Model If the verification in step 4 passes and the user does not report any errors, the system will not process the upload, thereby significantly saving bandwidth and protecting privacy.

[0068] Example 1: Scenario Setting 2_Order Successful: Scenario Setting: The customer interacts continuously in front of the shopping guide terminal of the Digua Development Board (edge ​​computing device) equipped with this system.

[0069] Multimodal dual-channel wake-up and context acquisition: The customer continues to interact in front of the terminal, inputting with voice: "Okay, place your order." At the same time, the terminal's visual channel (camera) captures the customer's facial features in real time.

[0070] The system uses facial recognition to match the user's ID and retrieves recent historical interaction records (ID, QA). Since this is not the customer's first interaction, the facial ID is asynchronously matched with locally cached historical interaction records, successfully linking to the previous session. Here, the QA is [{'role':'user','content':'Give me a milk tea'},{'role':'assistant','content':'Recommended jasmine apple tea, pure and light, NFC freshly squeezed juice, refreshing and delicious'}]. The speech stream is converted into the current text (current Q, "Okay, place order") using an ASR model. Extract previously recorded user attribute tags (currently "User is an adult").

[0071] Front-end call based on global intent parsing engine: Technical action: The lightweight intent parsing model (rather than the large model) of the edge front-end extracts Voice_Text, and the extraction result is {'ORDER':['Order successful'],'TYPE':['Tea drink'],'DRINK':['Jasmine apple tea'],'ALCOHOL':[],'SUGAR':[]}, which identifies the current intent as the intent to place an order.

[0072] Super Library Validation: The system immediately calls the local "Super Library (Business Rules and Facts Knowledge Base)" for preliminary validation. The validation logic is as follows: compare the "Adult" tag in User_Tags with the "Alcoholic Beverages" attribute in Current_Order.

[0073] Verification result: Verification passed (the user is an adult and complies with the rules for selling alcoholic beverages), and the intent is clear and unambiguous.

[0074] The streaming text-to-speech mechanism directly triggers the "short-circuit routing" mechanism, generating a standardized reply text stream: "Your selected product has been pushed to you. It has a complex fruity aroma and a light taste. Verify the beverage information and complete the payment. Your delicious drink is about to arrive." The text stream is fed into the TTS (Text-to-Speech) module in real time via a "streaming asynchronous mechanism," playing as it is generated. Simultaneously, the on-screen UI is rendered asynchronously and a payment QR code is pushed.

[0075] The customer successfully paid by scanning the QR code, and the order status changed to "In Production".

[0076] Example 1: Scenario Setting 3 - Increased Load on Heterogeneous Computing Units: Assume deployment on an outdoor smart terminal (e.g., an edge computing development board based on a heterogeneous architecture, such as the Digua Development Board). The underlying daemon process collects the device status in real time, sets the safety threshold for the comprehensive load index to 80, and the recovery threshold to 60 (introducing a hysteresis interval to prevent the "ping-pong effect" of frequent degradation / recovery at the threshold edge).

[0077] Initially in "full-blooded state" (Level 0): running the Qwen2.5-7B model, with BERT-LSTMLarge+ rules as the intent engine, unlimited KVCache, RAGTop-K=10, and TTS sampling rate of 24kHz.

[0078] Phase 1: Load Surge and Gradual Degradation Process: In high-temperature outdoor environments during summer, the device continuously processes multiple rounds of complex voice interactions. The underlying daemon process collects the five-dimensional hardware monitoring data in real time and calculates a comprehensive load index through weighted averages. When the comprehensive load index rises to 92 (greater than the safety threshold of 80), the system triggers a gradual degradation mechanism: Triggering Level 1 Degradation (Large Model Parameter Degradation): Action: The daemon process issues a command to hot-swap the LLM from Qwen2.5-7B to Qwen2.5-1.5B.

[0079] Assessment: After the switch, the daemon waited 500ms to recollect metrics and found that the overall load index dropped to 83.

[0080] Judgment: 83 is still greater than the safety threshold of 80, proceed to the next level.

[0081] Triggering Level 2 Degradation (Intent Model Degradation): Action: Downgrade the intent engine from BERT-LSTMLarge to the Base version.

[0082] Assessment: After waiting 500ms and re-collecting, the overall load index dropped to 72.

[0083] Judgment: 72 is less than the safety threshold of 80, and the indicator has fallen back to the safe range. Further downgrades are stopped, and the system remains in Level 2 operation. In this downgraded state, core large-scale dialogue models (such as the 1.5B parameter model) and basic intent recognition continue to function normally. The user experience is only characterized by a slight increase in inference latency or a decrease in the length of the text sequence output by the large model (number of tokens), effectively preventing system crashes due to Out of Memory (OOM).

[0084] Phase Two: Extreme Operating Conditions and Deep Degradation After running at Level 2 for a period of time, direct sunlight caused a sharp increase in the device's physical temperature, and memory fragmentation worsened due to prolonged operation. The overall load index spiked again to 95. The system triggered a further degradation from the current level (Level 2). Trigger Level 3 downgrade (generation restriction downgrade): Action: Truncate the length of the KVCache in the current 1.5B model and forcibly limit the maximum number of tokens generated in a single transaction to 64.

[0085] Assessment: After re-collecting the indicators, the overall load index dropped to 82.

[0086] Judgment: 82 is still greater than 80, proceed to the next level.

[0087] Triggering Level 4 downgrade (RAG deep downgrade): Action: Reduce the number of Top-K semantic retrievals in the RAG module from 10 to 5.

[0088] Assessment: After re-collecting the indicators, the overall load index dropped to 77.

[0089] Judgment: 77 is less than 80, stop the downgrade. The system remains at Level 4. At this point, the model's responses become shorter and the number of references cited decreases, but the core dialogue logic remains closed-loop.

[0090] Phase Three: Extreme Survival and the Last Line of Defense If the device's cooling fan suddenly fails, the temperature reaches the hardware's critical value, and the overall load index reaches the limit of 98, the system triggers the final stage of degradation: Triggering Level 5 Degradation (Audio Rendering Degradation): Action: Reduce the audio output sampling rate of the TTS module from 24kHz to 16kHz (or reduce the bit depth).

[0091] Assessment: After re-collecting the indicators, the overall load index dropped to 55.

[0092] Judgment: 55 is less than 80, stop the downgrade. The system enters Level 5 (extreme keep-alive state). At this time, the sound quality of the voice reply is slightly reduced, but the device successfully avoids hardware-level crashes or system failures caused by overheating.

[0093] Phase Four: Improvement in Operating Conditions and Dynamic Recovery Process When the user stops interacting and the device is moved to a cool place, causing the device temperature and heterogeneous computing unit utilization to drop, the underlying daemon process detects that the overall load index has been below the recovery threshold of 60 for 10 consecutive seconds (e.g., currently dropped to 45), and the system triggers the recovery mechanism: Recovery strategy: To ensure a smooth system transition, the recovery process is carried out in the reverse order of degradation (i.e., 5→4→3→2→1), or the highest availability level is matched directly according to the current load index.

[0094] Execution: First, restore TTS to 24kHz. After assessing that the load is stable, restore Top-K to 10, and so on, until the overall load index allows. Then, reload Qwen2.5-7B and BERT-LSTMLarge, and the system is fully restored to "full power" (Level 0).

[0095] In summary, this invention addresses the core pain points of edge devices, such as weak computing power, small memory, poor stability, high interaction latency, high model iteration costs, and significant privacy risks. Through multi-dimensional technological breakthroughs including architectural innovation, algorithm optimization, resource scheduling upgrades, and data security iterations, it achieves high-quality, high-stability, low-latency, and iterative deployment of lightweight large-scale edge models. The core technological effects can be categorized into four dimensions, as detailed below: I. Precise control of illusions in scenarios with limited computing power, achieving a dynamic balance between quality and computing power. Addressing industry pain points such as the easy generation of false content by lightweight small models at the edge and the inability of fixed RAG retrieval strategies to adapt to edge computing power, a combination of adaptive retrieval and multi-layer post-verification is adopted to make up for the quality shortcomings of small model generation.

[0096] 1. Adaptive optimization of computing resources: The local adaptive RAG module can dynamically adjust the number of Top-K searches based on the real-time load of edge hardware. Combined with a lightweight semantic matching algorithm, it abandons the drawbacks of the traditional fixed search mode and solves the dilemma of slow search speed and insufficient effective information recall. It achieves the optimal dynamic balance between search recall rate and system hardware load, and is compatible with various low-computing-power edge devices.

[0097] 2. Low-cost suppression of model illusions: A triple verification subunit of local facts, super library, and language is built. Lightweight rules and lightweight models are used to complete post-verification of generated content without consuming high computing power. This significantly reduces the occurrence rate of illusions and format errors in small edge models, greatly improves the accuracy, compliance, and actual business usability of the model output content, and makes up for the inherent quality defects of lightweight large models.

[0098] Second, break through the bottleneck of edge resources and achieve extremely stable operation of devices under all working conditions. In response to the problems of small memory and limited computing power of edge devices, traditional GC mechanisms are prone to memory fragmentation, program lag, OOM crashes, and system crashes under high temperature and frequency reduction conditions. The memory scheduling and hardware adaptation mechanism is reconstructed to completely solve the problem of stability of edge model operation.

[0099] 1. Completely eliminate memory thrashing and crash risks: Introducing streaming asynchronous and explicit memory scheduling modules, through three core units—zero-copy shared memory, circular asynchronous buffer channel, and explicit memory pool forced release that bypasses the operating system's GC—it eradicates the random pauses and memory fragmentation problems caused by traditional memory management, significantly improves memory utilization and turnover efficiency, and increases the continuous fault-free runtime (MTBF) of large edge models by orders of magnitude, completely solving the core pain point of OOM crashes in edge deployments.

[0100] 2. Achieve Degradation and Survival Under Extreme Operating Conditions: An innovative five-dimensional, progressively adaptive degradation protection mechanism based on edge hardware status awareness breaks the fragile "either running or crashing" mode of traditional systems. When devices experience extreme situations such as high temperature throttling, memory shortages, or computing power preemption, the system can smoothly degrade according to the priority of model parameters, intent accuracy, generation limitations, retrieval depth, and audio rendering, ensuring uninterrupted service and availability without crashes under harsh hardware environments, greatly improving system robustness.

[0101] Third, we reconstruct the end-to-end interactive architecture to achieve real-time interactive experience for edge devices. To address the problems of high redundancy overhead, high end-to-end latency, fragmented interaction, and easy avalanche in high concurrency of the traditional end-to-end serial architecture, we achieve extreme compression of interaction latency and a significant increase in system throughput through asynchronous streaming architecture, parallel inference, and dynamic scheduling of computing power.

[0102] 1. Significantly reduce module data transfer latency: Relying on a ring-shaped asynchronous buffer channel and zero-copy shared memory units, direct transmission of memory pointers for multiple data streams is achieved, bypassing cross-process and cross-thread serialization overhead and GC scan pauses. Data transfer between modules is reduced from milliseconds to microseconds, with latency reduced by more than 70%, freeing up memory bus bandwidth for edge devices and avoiding buffering anomalies caused by mismatch in data production and consumption rates.

[0103] 2. Extremely Compressed First Voice Interaction Latency: A deeply coupled streaming asynchronous architecture with TTS is constructed to achieve parallel processing of LLM text generation, speech synthesis, and audio playback, breaking rigid serial dependencies. At the same time, the token and TTS synthesis granularity are dynamically adjusted according to hardware load to adapt to different scenarios with high and low loads, compressing the end-to-end first voice latency (TTFA) to less than 1 second, eliminating the interaction silence period, and achieving a seamless dialogue experience.

[0104] 3. Several-fold improvement in system throughput and concurrency: A lightweight front-end intent short-circuit interception module is added. For high-frequency, simple intents that account for 60%-70%, the response is directly handled by the local rule engine, bypassing the large model inference and RAG retrieval process. The interception time for a single intent is only 0.008 seconds. Idle computing power is dynamically allocated to complex tasks, completely solving the problem of inverted computing resources. This results in a 3-5 times increase in system throughput (QPS) and energy efficiency, and significantly enhances stability in high-concurrency scenarios.

[0105] IV. Accurate and secure data feedback, building an automated and self-healing model iteration closed loop. Addressing the pain points of traditional full data upload, such as high cost, low training efficiency, and high risk of privacy leakage, as well as the high memory consumption, lack of automatic verification, and high operation and maintenance costs of edge model updates, we have created a low-cost, high-security, and fully automated end-to-cloud collaborative iteration system.

[0106] 1. Accurately select high-value samples to improve model iteration efficiency: Abandon the full / random data upload mode, and rely on the triple verification module and user feedback tags to only select difficult example data with abnormal model inference and verification failure to be fed back to the cloud, which greatly reduces redundant data transmission and storage computing power costs, allowing cloud LoRA training to focus on the blind spots of model knowledge and improve training targeting and convergence speed.

[0107] 2. Targeted desensitization on the client side for ultimate data compliance: A client-side privacy desensitization sandbox is built, and localized desensitization processing is performed only on a small number of difficult-to-access data. This reduces the scope of privacy data flow from all interactive data to a very small number of abnormal data, significantly shrinking the privacy attack surface, eliminating the risk of user privacy leakage in normal interaction scenarios, and meeting data security compliance requirements.

[0108] 3. Seamless Updates During Off-Peak Hours + Automatic Self-Healing, Zero-Maintenance Iteration: Based on historical interaction traffic data, off-peak periods are predicted, dynamically capturing the idle window of no interaction on devices, and completing hot model updates within the idle window. This avoids business interruption issues and avoids the memory doubling defect of traditional double-buffered updates, preventing edge device OOM crashes. After the model is updated, standard dataset A / B verification is automatically performed. If the indicators are not met, a seamless rollback is performed, forming a fully automatic closed loop of "update-verification-rollback". No manual maintenance is required, completely eliminating online failures caused by the deployment of inferior models and achieving continuous and safe model iteration.

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0111] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An interaction method for a local large model on the edge side, characterized in that, Includes the following steps: The system collects the user's facial image to retrieve the user's recent historical interaction records; at the same time, it converts the collected user's voice stream into current text, extracts intent, and controls the routing distribution unit to perform short-circuit judgment based on the intent recognition result in order to generate attribute label routes. Based on the attribute tag routing, and using the recent historical interaction records and the current text as a basis, the fragments are retrieved and recalled in the local knowledge base to construct a Prompt context with constraint instructions; Based on the Prompt context, by obtaining user attribute tags, adaptive local LLM inference is performed, and then fact verification, super library verification, and language verification are performed in sequence to generate a streaming text fragment. The text fragment is then converted and played back using a lightweight acoustic model on an edge computing device.

2. The interaction method of a local large model on the edge side according to claim 1, characterized in that: When acquiring a user's facial images and voice stream, a dual-channel wake-up mechanism of visual wake-up and voice wake-up is deployed locally. Through the wake-up operation, the user's facial images and voice stream are acquired. The visual channel performs face detection and lip motion capture using a lightweight CV model, while the voice channel continuously extracts speech acoustic features using a low-power audio engine.

3. The interaction method of a local large model on the edge side according to claim 1, characterized in that: When performing short-circuit judgment, if the identified business is the preset first type of business, and the business variables are in the standard database and have passed inventory verification, and the user attribute tags do not trigger compliance interception, then the execution instruction is directly generated and the voucher is pushed, and the process short-circuit ends. Otherwise, execution is blocked, and the recent historical interaction records, the current text, and the attribute tags are routed to the next process to obtain the Prompt context with constraint instructions.

4. The interaction method of a local large model on the edge side according to claim 1, characterized in that: When constructing a Prompt context with constraint instructions, based on the attribute tag routing, a local lightweight semantic matching algorithm is used to retrieve relevant fragments from the local knowledge base according to the recent historical interaction records and the current text; and the retrieved fragments are concatenated with the recent historical interaction records and the current text to construct a Prompt context with constraint instructions.

5. The interaction method of a local large model on the edge side according to claim 4, characterized in that: During the retrieval process, the load and memory availability of the heterogeneous computing units are obtained in real time. Based on the load and memory availability, the Top-K value of the retrieval recall is dynamically calculated. According to the Top-K value, a first threshold is set when resources are sufficient, and the second threshold is adaptively downgraded when the load is high.

6. The interaction method of a local large model on the edge side according to claim 1, characterized in that: During adaptive local LLM inference, the Prompt context and the user attribute tags are fed into the local lightweight LLM model for inference. The model is generated based on the execution conditions of the user attribute tags. In this process, the load and memory availability of the heterogeneous computing units are obtained in real time, and the target inference model is dynamically selected from the candidate model library based on the load and memory availability. When resources are sufficient, the local first model is set, and under high load, it is adaptively downgraded to the local second model. If the operation is still laggy, a generation limit downgrade is adopted: the length of the LLM KVCache is truncated, and the maximum token limit for a single generation is reduced.

7. The interaction method of a local large model on the edge side according to claim 1, characterized in that: After performing adaptive local LLM inference, fact verification is performed: the local LLM-generated text is cross-validated with the knowledge fragments recalled by the local RAG, and factual illusions are marked and corrected. Through super-database verification, the local LLM output text is sent back to the local global lightweight intent parsing engine to extract the product business variables mentioned therein, compare them with the standard product database, and if a product is found not in the database, an interception is triggered, and it is corrected or regenerated. Language validation checks the consistency between the local LLM response language and the user's input language. If they do not match, the local lightweight translation model is invoked for alignment.

8. The interaction method of a local large model on the edge side according to claim 1, characterized in that: When performing speech conversion and playback, the system receives text segments streamed from the local large language model. When a text segment meets the preset sentence segmentation conditions, it is used as the current text to be synthesized. The synthesized text is processed in the front end, including text regularization, polyphonic character disambiguation and prosody prediction, to generate phoneme sequences and corresponding prosodic features; Phoneme sequences and prosodic features are input into a lightweight acoustic model pre-deployed on an edge computing device to generate Mel spectrum features. The lightweight acoustic model is a lightweight neural network model that has undergone knowledge distillation, model pruning, or quantization to adapt to the computing power limitations of the edge computing device. Mel spectral features are converted into time-domain audio waveform data using a lightweight vocoder; Write time-domain audio waveform data to the audio buffer and control the audio playback thread to read and play audio data blocks in a streaming manner.

9. The interaction method of a local large model on the edge side according to claim 1, characterized in that: When performing local large model interaction on the edge side, a DMA-based zero-copy mechanism is used to build a shared memory space to achieve cross-module data transfer and avoid data copying involving the CPU. A ring-shaped asynchronous buffer channel consisting of LLM streaming output and TTS block consumption is constructed. When the token output by LLM reaches a sentence, it is directly supplied to the TTS module for block-synthesized speech, so as to realize simultaneous speech output and synthesis. After a single interactive task is completed, the default garbage collection mechanism of the operating system is bypassed, and the underlying explicit release instruction is called. Based on the pre-allocated memory block linked list and reference counting mechanism, the storage resource pool, inference context and RAG vector cache of the heterogeneous computing unit are forcibly cleared.

10. The interaction method of a local large model on the edge side according to claim 1, characterized in that: When performing local large-scale model interactions on the edge, the underlying daemon process collects real-time data on the utilization rate of heterogeneous computing units, remaining memory, device temperature, memory fragmentation rate, and task queue length of the device, and calculates a comprehensive load index through weighted averages. When the comprehensive load index exceeds a safe threshold, the load is progressively downgraded in a fixed order from most to least impactful on computing power. If the index falls back to a safe range after each downgrade, the subsequent downgrade stops; otherwise, it proceeds to the next level. A privacy-de-identifying sandbox is built on the edge computing device to filter out abnormal interaction data that fails the triple verification of facts, super library and language, as well as high-value difficult case data containing user negative feedback indicators; After the selected data is localized and anonymized in a privacy sandbox, it is transmitted to the central server through a secure encrypted channel when the data reaches a certain quantity or when a certain amount of time has passed since the last model update. Using the central server, the received de-identified data is subjected to lightweight incremental training using low-rank adaptive technology, and the incremental weights generated by the training are compressed by 8-bit quantization to generate model incremental weights adapted to the computing power of edge devices, which are then distributed to the edge computing devices.

Images