End-edge collaborative lightweight voice interaction large model optimization method and system

Abstract

The application discloses an end-edge collaborative lightweight voice interaction large model optimization method and system, belongs to the technical field of data mining, data analysis and artificial intelligence, and aims to solve the technical problems of low integration, high resource demand, insufficient robustness and limited adaptability of existing webpage control technology. The technical scheme is as follows: natural language instruction analysis: capturing user natural language instructions on the terminal device by using an ASR model, converting voice signals into text data, and performing semantic recognition and intention recognition on an edge server by using a generative large model to generate structured task instructions; scene vocabulary extraction and ASR model fine-tuning: constructing a scene-specific vocabulary library based on a control scene, fine-tuning a pre-trained ASR model, and improving the voice recognition accuracy in a specific scene; optimizing semantic understanding based on an RAG model; optimizing instruction generation based on a prompt word technology; instruction verification and error correction; automatic webpage operation; model optimization and inference acceleration.

Description

Technical Field

[0001] This invention relates to the fields of data mining, data analysis and artificial intelligence, specifically to a lightweight method and system for optimizing large-scale voice interaction models with edge-end collaboration. Background Technology

[0002] With the rapid development of artificial intelligence, data mining, and data analysis technologies, Automatic Speech Recognition (ASR), Generative Large Models (LLM), and automated web page control technologies have been widely applied in fields such as intelligent customer service, web page data processing, automated testing, and enterprise process automation. While ASR, LLM, and Selenium tools have laid the foundation for intelligent and automated task processing, existing technologies have the following significant shortcomings:

[0003] ①Low integration: Voice command parsing and web page operation modules usually run independently, lacking a unified end-to-end collaboration mechanism, resulting in a gap between command understanding and execution, reducing automation efficiency.

[0004] ② High resource requirements: Existing systems rely heavily on high-performance computing resources, resulting in high model complexity and deployment costs, making it difficult to operate efficiently in edge device scenarios.

[0005] ③ Insufficient robustness: The lack of real-time feedback and optimization mechanisms makes it difficult for the system to dynamically adjust task execution strategies, resulting in insufficient accuracy and stability in complex or dynamic web page environments.

[0006] ④ Limited adaptability: Existing methods do not adequately support diverse scenarios (such as multilingual instructions and cross-platform web page operations), which limits their promotion in global and cross-industry applications. Summary of the Invention

[0007] The technical objective of this invention is to provide a lightweight voice interaction large-scale model optimization method and system with end-edge collaboration to solve the problems of low integration, high resource requirements, insufficient robustness, and limited adaptability of existing web page control technologies.

[0008] The technical objective of this invention is achieved as follows: a lightweight large-scale voice interaction model optimization method with end-edge collaboration, the specific method of which is as follows:

[0009] Natural Language Command Parsing: On the edge device, the ASR (Speech Recognition) model is used to capture the user's natural language commands, convert the speech signal into text data, and perform semantic recognition and intent recognition through a generative large model on the edge server, thereby generating structured task commands;

[0010] Scene vocabulary extraction and ASR model fine-tuning: Based on the control scene, a scene-specific vocabulary library is built, and the pre-trained ASR model is fine-tuned to improve the speech recognition accuracy in specific scenes;

[0011] Optimizing semantic understanding based on the RAG model: A knowledge base containing web page control instructions and operation rules is built on the edge server. The RAG model is combined with the BERT embedding model to perform text vectorization and instruction template retrieval, generating semantically rich structured instructions.

[0012] Optimize instruction generation based on prompt word technology: Design scenario-specific prompt word templates on edge servers, dynamically adjust prompt word content, and guide generative large models to generate standardized structured instructions;

[0013] Command verification and error correction: Perform semantic consistency verification on standardized structured commands on the edge server. If there are semantic deviations or incompleteness, retrieve supplementary information or re-optimize prompt words through the RAG model.

[0014] Automated web page operations: On the edge device, structured instructions are executed through browser automation tools to realize web page data crawling, web page control, page navigation and dynamic interaction, and the execution results are sent back to the edge server for optimization through the React feedback mechanism;

[0015] Model optimization and inference acceleration: Generative large models are converted to ONNX format and quantized with INT8, and ASR models are converted to MNN format and quantized with INT8, improving inference speed and reducing resource requirements.

[0016] As a preferred approach, the ASR model adopts the Whisper model, the generative large model adopts the Qwen model, and the end-side device and the edge server interact with text data and structured instructions in real time through efficient APIs to reduce instruction transmission latency.

[0017] The device is used for lightweight voice processing and command execution; the edge server is used for high-performance semantic analysis and command optimization, and the end-edge collaboration is achieved through low-latency API.

[0018] As a preferred option, a scenario-specific vocabulary library extracts high-frequency scenario-specific words by analyzing user interaction data and stores them on the edge device;

[0019] The edge server uses the recording dataset of control commands to incrementally train the ASR model, adjusts the acoustic model and language model, reduces the false recognition rate in background noise environments, and periodically synchronizes and fine-tunes the ASR model to the edge device.

[0020] As a preferred option, the knowledge base is stored in the FAISS vector database. The RAG model, combined with the BERT embedding model, vectorizes the text recognized by ASR, retrieves relevant instruction templates or context information from the knowledge base, and the Qwen big model generates structured instructions that conform to the logic of the control scenario.

[0021] Preferably, the prompt word template adopts JSON or XML format, which includes action type, target object and operation parameters; through zero-sample or few-sample learning methods, the prompt words are dynamically adjusted to adapt to complex instruction requirements and ensure the executability of instructions.

[0022] As a preferred approach, instruction verification checks semantic consistency using the Qwen large model, and the error correction mechanism uses RAG retrieval to supplement missing parameters or re-trigger the prompt word optimization process, thereby improving the robustness of instruction parsing and adapting to diverse user inputs.

[0023] Even better, browser automation tools use Selenium, which supports multilingual commands and cross-platform web page operations;

[0024] React's feedback mechanism sends the execution results back to the edge server, where Qwen's large model analyzes and dynamically adjusts the subsequent instruction generation strategy.

[0025] A lightweight, edge-coordinated large-scale voice interaction model optimization system, comprising:

[0026] End-side devices are used for voice capture, command execution, and result feedback.

[0027] Edge servers are used for semantic analysis, knowledge retrieval, and instruction generation.

[0028] The communication interface is used to enable data interaction between the device and the edge server through a low-latency API, including text transmission, structured command issuance, and execution result feedback. The device and the edge server work together, with the device used for lightweight processing and the edge server used for high-performance computing to optimize system real-time performance and resource efficiency.

[0029] As a preferred embodiment, the edge device includes a speech parsing module, an ASR module (MNN format, INT8 quantization), a Selenium web page control module, and a React feedback module.

[0030] The speech parsing module is used to capture users' natural language commands on the edge device using an ASR model, convert speech signals into text data, and perform semantic analysis and intent recognition on the edge server using a generative big model to generate structured task commands.

[0031] The ASR module is used to improve the accuracy of speech recognition in specific scenarios. The ASR model is converted into MNN format and quantized with INT8 to improve inference speed and reduce resource requirements.

[0032] The Selenium web control module executes structured instructions to achieve web data scraping, web page control, page navigation, and dynamic interaction;

[0033] The React Feedback module is used to send execution results back to edge servers for optimization.

[0034] More preferably, the edge server includes a Qwen large model (ONNX format, INT8 quantization), a FAISS vector database, a RAG module, a prompt word optimization module, and an instruction verification and error correction module;

[0035] The Qwen large model is used to generate structured instructions that conform to the logic of the control scenario; the Qwen large model is converted to ONNX format and quantized using INT8.

[0036] The FAISS vector database is used to store a structured knowledge base of web page control instructions, operating rules, and frequently asked questions built on edge servers;

[0037] The RAG module is used to vectorize the text recognized by ASR through the BERT embedding model and retrieve relevant instruction templates or context information from the knowledge base;

[0038] The prompt word optimization module is used to design scenario-specific prompt word templates for the Qwen large model on the edge server. Based on the complexity of user commands and scenario requirements, the edge server dynamically adjusts the prompt word content to ensure the accuracy and executability of the generated commands.

[0039] The instruction verification and error correction module is used to input the generated structured instructions into the Qwen large model through the edge server for semantic consistency checks, verifying whether the instructions conform to the logic or expectations of the control scenario: if semantic deviations or incomplete instructions are detected, the edge server retrieves supplementary information through RAG, or re-triggers the prompt word optimization process to generate more accurate instructions.

[0040] The lightweight voice interaction large model optimization method and system with end-edge collaboration of the present invention have the following advantages:

[0041] (I) Addressing the issues of low integration, high resource requirements, insufficient robustness, and limited adaptability in existing web page control technologies, this invention achieves efficient conversion from voice commands to web page operations by deeply integrating Asr, Qwen large models, and Selenium, combined with RAG technology, prompt word optimization, command verification, and error correction. The edge processing handles lightweight voice capture and command execution, while the edge server is responsible for high-performance semantic analysis and optimization. Combined with ONNX and MNN format INT8 quantization models, inference is accelerated, reducing power consumption by 30%, increasing inference speed by 2-3 times, and reducing the false recognition rate by 10%. Simultaneously, edge-end collaboration enhances the system's real-time performance, robustness, and multilingual and cross-platform adaptability, making it widely applicable to scenarios such as intelligent customer service, web page data processing, and automated testing.

[0042] (II) This invention first converts the model into an INT8 quantized model in ONNX and MNN formats. Then, it parses the user's natural language instructions using the converted ASR and generative large model to generate structured task instructions. Subsequently, it uses an automated web page control module combined with Selenium tools to accurately execute operations such as data crawling, web page control, page navigation, and dynamic interaction. Finally, through the React feedback mechanism, the execution results are sent back to the large model for verification and optimization, ensuring high accuracy and robustness of the task, significantly improving the intelligence level of automated tasks, reducing the cost of manual intervention, and is widely applicable to scenarios such as web page data processing, data analysis, and intelligent customer service, with broad application prospects.

[0043] (III) This invention deeply integrates speech recognition (ASR) model, Qwen large model and automated web page control technology, and realizes efficient and intelligent automated task processing through end-edge collaborative architecture, which significantly improves the system's integration, resource efficiency, robustness and adaptability, and is suitable for various scenarios such as web page data processing, data analysis, intelligent customer service and automated testing;

[0044] (iv) This invention solves the problems of insufficient intelligence, control efficiency and adaptability of existing web page control technology by working together with edge computing. The end-edge collaboration mechanism decomposes voice processing, semantic analysis and instruction execution to end devices and edge servers, optimizes real-time performance and resource utilization, adapts to the needs of scenarios with high noise and complex instructions, and improves the real-time performance and resource efficiency of the system in the scenario.

[0045] (v) The Qwen large model of the present invention is converted to ONNX format and quantized with INT8. Through quantization-aware training (QAT) or post-training quantization (PTQ), the inference speed is improved by 2-3 times and the accuracy loss is controlled within 1%. The Whisper model of the present invention is converted to MNN format and quantized with INT8. The inference speed is improved by 2.5 times, the false recognition rate is reduced by 10%, and the power consumption is reduced by 30%, which is suitable for the low power consumption requirements of edge devices.

[0046] (vi) This invention supports multilingual command output and cross-platform web page operation, adapting to global application scenarios; through end-edge collaborative architecture, combined with ONNX and MNN format INT8 quantization model, power consumption is reduced by 30%, improving the real-time performance and robustness of the system in lobby scenarios, and is suitable for web page data processing, data analysis, intelligent customer service and automated testing. Attached Figure Description

[0047] The invention will be further described below with reference to the accompanying drawings.

[0048] Appendix Figure 1 A flowchart illustrating the process of optimizing a lightweight voice interaction large-scale model for end-edge collaboration. Detailed Implementation

[0049] The following detailed description of the lightweight voice interaction large model optimization method and system of the present invention, with reference to the accompanying drawings and specific embodiments, is provided.

[0050] Example 1:

[0051] This embodiment provides a lightweight optimization method for large-scale voice interaction models with end-edge collaboration, as detailed below:

[0052] S1. Natural Language Command Parsing: On the edge device, the ASR (Speech Recognition) model is used to capture the user's natural language commands, convert the speech signal into text data, and perform semantic recognition and intent recognition through a generative large model on the edge server, thereby generating structured task commands;

[0053] S2. Scene vocabulary extraction and ASR model fine-tuning: Based on the control scene, a scene-specific vocabulary library is built, and the pre-trained ASR model is fine-tuned to improve the speech recognition accuracy in specific scenes.

[0054] S3. Optimize semantic understanding based on RAG model: Build a knowledge base containing web page control instructions and operation rules on the edge server, and use the RAG model combined with the BERT embedding model to perform text vectorization and instruction template retrieval to generate semantically rich structured instructions.

[0055] S4. Optimize instruction generation based on prompt word technology: Design scenario-specific prompt word templates on the edge server, dynamically adjust the prompt word content, and guide the generative large model to generate standardized structured instructions;

[0056] S5. Command Verification and Error Correction: Perform semantic consistency verification on standardized structured commands on the edge server. If there are semantic deviations or incompleteness, supplementary information is retrieved through the RAG model or the prompt words are re-optimized.

[0057] S6. Automated Webpage Operations: Execute structured instructions through browser automation tools on the edge device to achieve webpage data crawling, webpage control, page navigation and dynamic interaction, and send the execution results back to the edge server for optimization through the React feedback mechanism;

[0058] S7. Model Optimization and Inference Acceleration: Generative large models are converted to ONNX format and quantized using INT8, and ASR models are converted to MNN format and quantized using INT8, improving inference speed and reducing resource requirements.

[0059] In step S1 of this embodiment, the ASR model adopts the Whisper model, the generative large model adopts the Qwen model, and the terminal device and the edge server interact with text data and structured instructions in real time through efficient APIs to reduce instruction transmission latency.

[0060] The device is used for lightweight voice processing and command execution; the edge server is used for high-performance semantic analysis and command optimization, and the end-edge collaboration is achieved through low-latency API.

[0061] In step S2 of this embodiment, the scene-specific vocabulary library extracts high-frequency scene words by analyzing user interaction data and stores them on the terminal device.

[0062] The edge server uses the recording dataset of control commands to incrementally train the ASR model, adjusts the acoustic model and language model, reduces the false recognition rate in background noise environments, and periodically synchronizes and fine-tunes the ASR model to the edge device.

[0063] In step S3 of this embodiment, the knowledge base is stored in the FAISS vector database. The RAG model, combined with the BERT embedding model, vectorizes the text recognized by ASR, retrieves relevant instruction templates or context information from the knowledge base, and the Qwen big model generates structured instructions that conform to the logic of the control scenario.

[0064] In step S4 of this embodiment, the prompt word template adopts JSON or XML format and includes action type, target object and operation parameters; through zero-sample or few-sample learning method, the prompt words are dynamically adjusted to adapt to complex instruction requirements and ensure the executability of instructions.

[0065] In step S5 of this embodiment, the instruction verification checks semantic consistency through the Qwen large model, and the error correction mechanism uses RAG retrieval to supplement missing parameters or re-trigger the prompt word optimization process to improve the robustness of instruction parsing and adapt to diverse user input.

[0066] In this embodiment, the browser automation tool in step S6 uses Selenium, which supports multilingual commands and cross-platform web page operations.

[0067] React's feedback mechanism sends the execution results back to the edge server, where Qwen's large model analyzes and dynamically adjusts the subsequent instruction generation strategy.

[0068] To improve the system's real-time performance and resource efficiency in the scenario, this embodiment optimizes and accelerates the inference process of the core model, emphasizing end-edge collaboration, as detailed below:

[0069] ① Large Model Optimization: The Qwen large model on the edge server was converted to ONNX format, and the model structure was optimized to improve cross-platform compatibility and inference performance. INT8 quantization technology was used to compress the ONNX model, significantly reducing computational complexity and memory usage while maintaining the accuracy of semantic analysis and intent recognition. Furthermore, through quantization-aware training (QAT) or post-training quantization (PTQ), inference speed was improved by approximately 2-3 times, and model accuracy loss was controlled within 1%.

[0070] ② Speech Recognition and Synthesis Optimization: The Whisper speech recognition and synthesis models on edge devices are converted to MNN format to adapt to the lightweight inference framework and optimize deployment efficiency on edge devices. INT8 quantization is applied to the MNN model to further reduce computational and storage requirements, improving the real-time performance of speech recognition and synthesis. Experiments show that quantization improves the model's inference speed by approximately 2.5 times, and reduces the false recognition rate by approximately 10% in high-noise environments.

[0071] ③ Advantages of end-edge collaboration: End devices run lightweight MNN models to process real-time speech recognition and synthesis; edge servers run high-performance ONNX models to process complex semantic analysis and instruction generation. The two work together to significantly improve system real-time performance and resource efficiency, reduce power consumption by about 30%, and are suitable for large-scale deployment.

[0072] The specific details of scene vocabulary extraction and ASR model fine-tuning in step S2 of this embodiment are as follows:

[0073] ① Vocabulary extraction: Based on the control scenario, high-frequency scenario words (such as "search", "fill in", "query", "open" etc.) are extracted from user interaction data to build a scenario-specific vocabulary library, which is stored on the terminal device to support fast access;

[0074] ② Model fine-tuning: The pre-trained Whisper speech recognition model is fine-tuned using a scene vocabulary database. Supervised learning is employed, with labeled speech-text pairs as input, to optimize the model's speech recognition accuracy in controlled scenarios. Incremental training is performed on an edge server using a recording dataset of control commands to adjust the model's acoustic and language models, improve the recognition rate of specific words, and reduce the false recognition rate in noisy environments. The fine-tuned model is periodically synchronized to edge devices to ensure recognition consistency.

[0075] ③ Advantages of edge-end collaboration: End devices store vocabulary and lightweight models, while edge servers complete complex fine-tuning tasks, balancing computational load and recognition accuracy.

[0076] The specific steps in step S3 of this embodiment, which combine RAG technology to optimize semantic understanding, are as follows:

[0077] ① Knowledge base construction: A structured knowledge base containing web page control instructions, operation rules and frequently asked questions is built on the edge server and stored in the FAISS vector database to support efficient retrieval;

[0078] ② Retrieval stage: The edge device transmits the text recognized by ASR to the edge server through a low-latency network. The RAG model vectorizes the text through the BERT embedding model and retrieves relevant instruction templates or context information from the knowledge base.

[0079] ③ Generation stage: The Qwen big model on the edge server combines the search results to generate semantically rich structured instructions, ensuring that the instructions conform to the logic and semantics of the control scenario; RAG technology enhances the semantic understanding ability of the model through external knowledge, significantly improving the parsing accuracy of complex instructions (such as multi-step web page operations); end-edge collaboration makes up for the lack of computing resources on the end device through the high-performance retrieval and generation of the edge server.

[0080] ④ Advantages of edge-end collaboration: End devices are responsible for lightweight text transmission, while edge servers perform high-performance semantic processing, optimizing latency and accuracy;

[0081] The specific generation of the prompt word technology optimization instruction in step S4 of this embodiment is as follows:

[0082] ① Prompt word design: Design scenario-specific prompt word templates for the Qwen large model on the edge server, such as "Convert the following voice commands into web page control tasks: {ASR output}, with the output format being {structured JSON}";

[0083] ② Dynamic Adjustment: Based on the complexity of user commands and scenario requirements, the edge server dynamically adjusts the prompt content (e.g., adding contextual constraints or task priorities) to ensure the accuracy and executability of generated commands; using zero-shot or few-shot learning methods, the model is guided to generate high-quality structured commands through carefully designed prompts; combined with command templates (e.g., JSON format: {"action":"navigate","target":"URL","parameters":{}}), the output structure is standardized to facilitate subsequent processing by the web page control module;

[0084] ③ Advantages of edge-end collaboration: The edge server is responsible for optimizing prompts and generating instructions, while the end device only needs to receive standardized instructions, reducing the complexity of local processing.

[0085] The instruction verification and error correction in step S5 of this embodiment are as follows:

[0086] ① Semantic verification: The edge server inputs the generated structured instructions into the Qwen big model for semantic consistency check to verify whether the instructions conform to the logic and expectations of the control scenario;

[0087] ② Error correction mechanism: If semantic deviation or incomplete instruction is detected (such as missing key parameters), the edge server retrieves supplementary information through RAG or re-triggers the prompt word optimization process to generate more accurate instructions; the verification and error correction mechanism ensures the robustness of instruction parsing and adapts to diverse user input (such as accent differences and ambiguous expressions).

[0088] ③ Advantages of edge-end collaboration: Edge servers perform complex verification and error correction, while end devices only need to execute the final instructions, improving the overall efficiency of the system.

[0089] This embodiment also includes output structured instructions, as follows:

[0090] ① Command formatting: The edge server outputs the optimized command in a standardized JSON or XML format, including the action type, target object and operation parameters (such as {"action":"fill_form","target":"input_field","value":"data"}), and transmits it to the end device via API;

[0091] ② Interface Adaptation: The end-side device seamlessly connects structured instructions to the automated web control module (such as Selenium) via API to execute web operations; this invention supports multilingual instruction output, adapting to global application scenarios; and reduces instruction transmission latency through an efficient end-to-edge communication interface design;

[0092] ③ Edge-edge collaboration advantages: Edge servers generate high-quality instructions, while end devices focus on execution, optimizing real-time performance and resource allocation.

[0093] Example 2:

[0094] This embodiment provides a lightweight voice interaction large model optimization system with end-edge collaboration, the system comprising:

[0095] End-side devices are used for voice capture, command execution, and result feedback.

[0096] Edge servers are used for semantic analysis, knowledge retrieval, and instruction generation.

[0097] The communication interface is used to enable data interaction between the device and the edge server through a low-latency API, including text transmission, structured command issuance, and execution result feedback. The device and the edge server work together, with the device used for lightweight processing and the edge server used for high-performance computing to optimize system real-time performance and resource efficiency.

[0098] The end-side device in this embodiment includes a speech parsing module, an ASR module (MNN format, INT8 quantization), a Selenium web page control module, and a React feedback module;

[0099] The speech parsing module is used to capture users' natural language commands on the edge device using an ASR model, convert speech signals into text data, and perform semantic analysis and intent recognition on the edge server using a generative big model to generate structured task commands.

[0100] The ASR module is used to improve the accuracy of speech recognition in specific scenarios. The ASR model is converted into MNN format and quantized with INT8 to improve inference speed and reduce resource requirements.

[0101] The Selenium web control module executes structured instructions to achieve web data scraping, web page control, page navigation, and dynamic interaction;

[0102] The React Feedback module is used to send execution results back to edge servers for optimization.

[0103] The edge server in this embodiment includes a Qwen large model (ONNX format, INT8 quantization), a FAISS vector database, a RAG module, a prompt word optimization module, and an instruction verification and error correction module;

[0104] The Qwen large model is used to generate structured instructions that conform to the logic of the control scenario; the Qwen large model is converted to ONNX format and quantized using INT8.

[0105] The FAISS vector database is used to store a structured knowledge base of web page control instructions, operating rules, and frequently asked questions built on edge servers;

[0106] The RAG module is used to vectorize the text recognized by ASR through the BERT embedding model and retrieve relevant instruction templates or context information from the knowledge base;

[0107] The prompt word optimization module is used to design scenario-specific prompt word templates for the Qwen large model on the edge server. Based on the complexity of user commands and scenario requirements, the edge server dynamically adjusts the prompt word content to ensure the accuracy and executability of the generated commands.

[0108] The instruction verification and error correction module is used to input the generated structured instructions into the Qwen large model through the edge server for semantic consistency checks, verifying whether the instructions conform to the logic or expectations of the control scenario: if semantic deviations or incomplete instructions are detected, the edge server retrieves supplementary information through RAG, or re-triggers the prompt word optimization process to generate more accurate instructions.

[0109] As attached Figure 1 As shown, the specific working process of this system is as follows:

[0110] Step 1, Voice Input (Edge-side): The user inputs natural language commands through the edge device, and the voice signal is captured;

[0111] Step 2, ASR processing (edge-side): The Whisper model (MNN format, INT8 quantization) converts speech into text data, improving the accuracy of recognition in noisy environments;

[0112] Step 3, Text Transmission (End-to-Edge): Transmit text data to the edge server via a low-latency API;

[0113] Step 4, RAG Retrieval (Marginal): The RAG model combines the FAISS knowledge base and the BERT embedding model to retrieve command templates and context;

[0114] Step 5, Prompt word optimization (edge): Dynamically adjust prompt words to guide the Qwen large model (ONNX format, INT8 quantization) to generate structured instructions;

[0115] Step Six: Instruction Verification (Edge): The Qwen large model verifies the semantic consistency of instructions, and corrects errors using RAG if necessary;

[0116] Step 7, Command Issuance (Edge-End): Structured commands (JSON / XML format) are transmitted to the edge device via API;

[0117] Step 8, Webpage Operations (Client-Side): Selenium executes commands to perform operations such as webpage data scraping and webpage control;

[0118] Step 9, Result Feedback (End-to-Edge): The React mechanism sends the webpage operation results back to the edge server;

[0119] Step 10, Strategy Optimization (Edge): Analyze feedback from the edge server and dynamically optimize the prompt words and command generation strategy.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight, edge-coordinated method for optimizing a large-scale voice interaction model, characterized in that: The method is as follows: Natural Language Command Parsing: On the edge device, the ASR model is used to capture the user's natural language commands, convert the speech signal into text data, and perform semantic recognition and intent recognition through a generative big model on the edge server, thereby generating structured task commands; Scene-specific vocabulary extraction and ASR model fine-tuning: Based on the control scene, a scene-specific vocabulary library is built, and the pre-trained ASR model is fine-tuned to improve the speech recognition accuracy in specific scenes; specifically: ① Vocabulary extraction: Based on the control scene, high-frequency scene words are extracted from user interaction data to build a scene-specific vocabulary library, which is stored on the edge device for quick access; ② Model fine-tuning: The pre-trained Whisper speech recognition model is fine-tuned using the scene vocabulary library. A supervised learning method is adopted, with labeled speech-text pairs as input, to optimize the speech recognition accuracy of the model in the control scene; Incremental training is performed on the recording dataset of control commands on the edge server to adjust the acoustic and language models of the model, improve the recognition rate of specific words, and reduce the false recognition rate in background noise environments. The fine-tuned model is periodically synchronized to the edge device to ensure recognition consistency; ③ Edge-edge collaboration advantages: The edge device stores the vocabulary and lightweight model, while the edge server completes the complex fine-tuning task, balancing the computational load and recognition accuracy. Optimizing semantic understanding based on the RAG model: A knowledge base containing web page control instructions and operation rules is built on the edge server. The RAG model is combined with the BERT embedding model to perform text vectorization and instruction template retrieval, generating semantically rich structured instructions. Specifically: ① Knowledge base construction: A structured knowledge base containing web page control instructions, operation rules and frequently asked questions is built on the edge server and stored in the FAISS vector database to support efficient retrieval; ② Retrieval Stage: The edge device transmits the text recognized by ASR to the edge server through a low-latency network. The RAG model vectorizes the text using the BERT embedding model and retrieves relevant instruction templates or contextual information from the knowledge base. ③ Generation Stage: The Qwen large model on the edge server combines the retrieval results to generate semantically rich structured instructions, ensuring that the instructions conform to the logic and semantics of the control scenario. RAG technology enhances the model's semantic understanding ability through external knowledge, improving the parsing accuracy of complex instructions. Edge-edge collaboration compensates for the insufficient computing resources of the edge device through the high-performance retrieval and generation of the edge server. ④ Advantages of Edge-Edge Collaboration: The edge device is responsible for lightweight text transmission, while the edge server completes high-performance semantic processing, optimizing latency and accuracy. Optimize instruction generation based on prompt word technology: Design scenario-specific prompt word templates on edge servers, dynamically adjust prompt word content, and guide generative large models to generate standardized structured instructions; Specifically: ① Prompt word design: Design prompt word templates for the Qwen large model on the edge server, in the form of: "Convert voice commands into web page control tasks: {ASR output}, output format is {structured JSON}"; ② Dynamic adjustment: The edge server dynamically adjusts the prompt word content according to the complexity of user commands and scenario requirements to ensure the accuracy and executability of generated commands; use zero-shot or few-shot learning methods to guide the model to generate high-quality structured commands through carefully designed prompt words; combine with command templates to standardize the output structure; ③ End-edge collaboration advantages: The edge server is responsible for prompt word optimization and command generation, and the end device only needs to receive standardized commands; Command Verification and Error Correction: Standardized structured commands are semantically validated on the edge server. If semantic deviations or incompleteness are found, supplementary information is retrieved through the RAG model or the prompt words are re-optimized. Specifically: ① Semantic Verification: The edge server inputs the generated structured commands into the Qwen large model for semantic consistency checks to verify whether the commands conform to the logic and expectations of the control scenario; ② Error Correction Mechanism: If semantic deviations or incomplete commands are detected, the edge server retrieves supplementary information through RAG or re-triggers the prompt word optimization process to generate more accurate commands. The verification and error correction mechanism ensures the robustness of command parsing and adapts to diverse user inputs; ③ End-Edge Collaboration Advantages: The edge server performs complex verification and error correction, while the end device only needs to execute the final command; The output of structured instructions includes: ① Instruction formatting: The edge server outputs optimized instructions in a standardized JSON or XML format, including action type, target object, and operation parameters, and transmits them to the edge device via API; ② Interface adaptation: The edge device seamlessly connects the structured instructions to the automated web control module via API to execute web operations; it supports multilingual instruction output to adapt to global application scenarios; and reduces instruction transmission latency through an efficient edge-end communication interface design; ③ Edge-end collaboration advantages: The edge server generates high-quality instructions, and the edge device focuses on execution, optimizing real-time performance and resource allocation; Automated web page operations: On the edge device, structured instructions are executed through browser automation tools to realize web page data crawling, web page control, page navigation and dynamic interaction, and the execution results are sent back to the edge server for optimization through the React feedback mechanism; Model optimization and inference acceleration: Generative large models are converted to ONNX format and quantized using INT8, and ASR models are converted to MNN format and quantized using INT8 to improve inference speed and reduce resource requirements; specifically: ① Large model optimization: Qwen large models on edge servers are converted to ONNX format to optimize the model structure; and INT8 quantization technology is used to compress ONNX models to reduce computational complexity and memory consumption, while maintaining the accuracy of semantic analysis and intent recognition; And quantify through quantitative perception training or post-training quantization; ② Speech recognition and synthesis optimization: The Whisper speech recognition model and speech synthesis model on the edge device are converted into MNN format to adapt to the lightweight inference framework and optimize the deployment efficiency on the edge device; and INT8 quantization is applied to the MNN model to further reduce the computation and storage requirements of the model and improve the real-time performance of speech recognition and synthesis; ③ Edge-end collaboration advantages: The edge device runs the lightweight MNN model to process real-time speech recognition and synthesis; The edge server runs a high-performance ONNX model, handling complex semantic analysis and instruction generation.

2. The lightweight voice interaction large model optimization method with end-edge collaboration according to claim 1, characterized in that, The ASR model uses the Whisper model, the generative large model uses the Qwen model, and the edge devices and edge servers use efficient APIs to interact with text data and structured instructions in real time, reducing instruction transmission latency. The device is used for lightweight voice processing and command execution; the edge server is used for high-performance semantic analysis and command optimization, and the end-edge collaboration is achieved through low-latency API.

3. The lightweight voice interaction large model optimization method with end-edge collaboration according to claim 1, characterized in that, The browser automation tool uses Selenium and supports multilingual commands and cross-platform web page operations; React's feedback mechanism sends the execution results back to the edge server, where Qwen's large model analyzes and dynamically adjusts the subsequent instruction generation strategy.

Images