End-side-cloud three-in-one active chatting robot system for high-emotional quotients

Abstract

An end-side-cloud three-in-one active chat robot system for high-emotional merchants comprises an end side, a local edge server and a cloud end, and the end side is used for collecting multi-modal data of a user and performing local lightweight real-time processing and response execution; the local edge server is used for receiving and fusing the multi-modal features and the context information from the end side, and carrying out sentiment calculation, dialogue management and active trigger decision making with medium complexity; the cloud end is used for operating a super-large-scale model and providing global knowledge management, long-term user portrait storage and model training optimization; and the end side, the local edge server and the cloud end carry out cooperative communication through an encrypted channel to form a distributed intelligent processing architecture. According to the method, global optimization is realized by integrating end-side lightweight sensing, edge multi-modal fusion and cloud long-term memory. A composite finite state machine (FSM) active questioning mechanism is combined with sentiment calculation, and is different from traditional rule type triggering. Off-line and on-line fusion scheduling and multi-agent role playing are combined to be applied to a high-emotional-quotient interaction scene, and the simulation is improved. Multi-modal emotion perception circulation is introduced, and the problem of single text emotion misjudgment is solved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, specifically a high-EQ proactive chatbot system based on a collaborative architecture of edge servers, local edge servers, and cloud computing. It has multimodal perception, emotion computing, and context awareness capabilities, and is suitable for scenarios such as intelligent customer service, health care, education, and entertainment. Background Technology

[0002] Current chatbot technology has the following limitations: Passive response and low realism: Traditional chatbots rely on user-triggered responses, lack proactive interaction capabilities, and their answers are mechanical and rigid.

[0003] Insufficient emotional understanding: Most systems analyze emotions based solely on text, ignoring multimodal data (such as facial expressions and tone of voice), resulting in weak empathy.

[0004] Low efficiency of edge-cloud collaboration: In weak network environments, cloud response latency is high, while the capabilities of pure edge-side models are limited; Existing online-offline fusion solutions (such as patent CN115795017A) improve offline performance through knowledge distillation, but do not solve the problem of multimodal proactive interaction.

[0005] Contextual break: The lack of long-term memory and personalized adaptation in multi-turn dialogues. Although Microsoft's patent introduces "contextual feedback of expiration indicators", it does not cover edge-cloud collaboration scenarios. Summary of the Invention

[0006] This technical solution provides a three-in-one high-EQ proactive chatbot system integrating edge, device, and cloud, including the device side, local edge server, and cloud, wherein: The terminal side is used to collect the user's multimodal data and perform local lightweight real-time processing and response execution; The local edge server is used to receive and fuse multimodal features and contextual information from the edge side to perform moderately complex sentiment computing, dialogue management, and proactive triggering decisions. The cloud is used to run ultra-large-scale models and provides global knowledge management, long-term user profile storage, and model training optimization. The endpoint, local edge server, and cloud communicate collaboratively through an encrypted channel, forming a distributed intelligent processing architecture.

[0007] Furthermore, the end side includes: The multimodal sensing module is used to collect voice, image, and physiological signal data through microphones, cameras, and sensors; The lightweight processing module is configured to run a lightweight model locally for wake word detection, speech recognition, preliminary facial expression judgment, and text sentiment analysis. The context management module is used to collect user location, time, and application status information; The emotional execution module is used to perform speech synthesis with emotional intonation and digital human facial expression rendering based on received instructions.

[0008] Furthermore, the local edge server includes: The multimodal fusion module is used to integrate text, voice tone, facial expressions, and contextual information to generate a comprehensive user profile. The dialogue management module is used to maintain the dialogue state and select dialogue strategies based on the user state profile and scenario complexity. The proactive triggering module is used to analyze user behavior patterns based on a composite finite state machine to determine the timing and content of proactive interactions. Edge cache pools are used to store models and knowledge bases delivered from the cloud for quick local access.

[0009] Furthermore, the cloud includes: A massive model pool for deploying and training language models and multimodal emotion models with hundreds of billions of parameters; A global knowledge base is used to store and update general knowledge, domain knowledge, and cultural trend information. Long-term memory units are used to securely store and retrieve user cross-session interaction summaries and preferences in vectorized form; The model factory is used to continuously optimize models and generate lightweight models through distillation technology, which are then distributed to the edge and endpoint.

[0010] Furthermore, the system employs a dynamic scheduling strategy, including: Calculate the intent confidence and entity confidence of the user request on the client side; The complexity of the current scene is evaluated on the edge server as a function of the number of intents, entity diversity, and frequency of historical interactions; The confidence level and complexity are weighted and calculated. If the result is lower than a set threshold, the request is forwarded to the cloud for processing; otherwise, the processing is completed at the edge.

[0011] Furthermore, the working method of the active triggering module includes: The first-level state machine parses the multimodal context and generates abstract user behavior states; The second-level state machine selects the most suitable one from a set of multiple high emotional intelligence question templates based on the user's behavior state. The selected question template is fed into the language model to generate natural language sentences, and context-related sentiment tone suggestions are added.

[0012] Furthermore, the system is applied in healthcare scenarios, wherein: The multimodal data includes cough sounds, fall impact sounds, heart rate, and blood oxygen data; The proactive decision-making includes generating a care inquiry and suggesting contacting an emergency contact when abnormal health data is detected.

[0013] Furthermore, the system is applied to educational companionship scenarios, wherein: The edge cache pool stores knowledge data specific to a particular role; The emotional execution module is configured to render corresponding voice tones and digital human images based on the character's personality, thereby achieving immersive role-playing interaction.

[0014] Furthermore, the system is applied to intelligent vehicle scenarios, wherein: The multimodal data includes the driver's facial expressions, heart rate data, and voice tone; The proactive decision-making includes proactively triggering music playback or rest suggestion commands when road rage or fatigue is detected, in order to regulate the driver's emotions.

[0015] Furthermore, the system is applied to commercial shopping guide scenarios, wherein: The multimodal data includes the customer's gaze duration and micro-expressions when viewing the product; The edge server integrates the multimodal data, product inventory information, and user historical preferences to generate personalized proactive marketing messages.

[0016] Furthermore, the system is applied to cross-language communication scenarios, wherein: The sentiment computing module is used to analyze the speaker's emotional tone and generate cross-language sentiment tags; The emotional execution module is used to synthesize speech by mimicking the emotional tone of the source language when translating target language text.

[0017] This invention addresses the aforementioned problems through a three-tier architecture (edge-cloud), multimodal emotion computing, composite finite state machine (FSM), and proactive triggering mechanism.

[0018] This system is divided into three levels: client-side, edge server, and cloud, which work together to achieve proactive, high-EQ interaction. On the device side, multimodal data acquisition (speech, facial expression, physiological sensors), lightweight real-time processing (local ASR / TTS, initial facial expression judgment), context collection (time, location, APP status), lightweight model (TensorFlowLite), and hardware acceleration (NPU / GPU).

[0019] Edge servers, multimodal information fusion (text + facial expression + voice), medium-complexity reasoning (affective computing, dialogue management), proactive decision-making, model caching and updating; containerized deployment (Docker / Kubernetes), stream processing (Apache Kafka), medium-sized LLM (7B-20B parameters).

[0020] Cloud-based: Ultra-large model training and inference (LLM with hundreds of billions of parameters), global knowledge management, long-term user profile storage, model distillation and distribution. Distributed training (DeepSpeed), vector database, GPU / TPU clusters.

[0021] The multimodal emotion perception loop calculates initial emotion tags (such as facial expression classification and voice emotion polarity) locally on the device, and the edge server fuses the multimodal data to generate a comprehensive user state profile (e.g., neutral text + sad facial expression → judged as "forced smile"). The emotion AI model (trained in the cloud + deployed at the edge) generates empathetic responses and drives the device's TTS and Avatar output through emotional tone suggestions.

[0022] The offline-online fusion and dynamic scheduling strategy has been improved to a multimodal scenario: the offline small model calculates the intent / entity confidence score, which is then weighted by the scenario complexity: offline confidence = f(scenario complexity, intent confidence, entity confidence). When the score is below the threshold, the cloud large model is requested to ensure that active interaction is still possible in a weak network environment.

[0023] Proactive triggering and long-term memory are used to generate proactive questions based on a composite finite state machine (FSM): The first-level FSM analyzes the user's behavioral state (e.g., being awake late at night and sighing) to generate a rough question; the second-level FSM refines it into a specific question (e.g., "You seem very tired today, would you like to chat?"). User history dialogue summaries are stored in the cloud (vectorized) to achieve cross-conversation personalization.

[0024] Edge-cloud collaborative workflow example: proactively comforting users who are feeling down, detecting sighs and frowns on the device (local lightweight model), and sending anonymized data to the edge; fusing multimodal data at the edge to confirm the emotional state and triggering a proactive greeting (generated by edge LLM); in complex scenarios, requesting the cloud to generate a deep empathetic response and sending it to the device for execution.

[0025] This invention achieves global optimization by integrating edge-side lightweight sensing, edge multimodal fusion, and cloud-based long-term memory. It combines a composite finite state machine (FSM) proactive questioning mechanism with emotion computing, distinguishing it from traditional rule-based triggering.

[0026] By combining "online-offline fusion scheduling" with "multi-agent role-playing," this approach is applied to high-EQ interaction scenarios to enhance realism. A multimodal emotion perception loop is introduced to address the problem of misjudging emotions in single-text interactions (e.g., trembling voice + stiff facial expression → realistic emotion assessment). Average response latency is <800ms (edge-local decision-making), representing a 60% speedup compared to pure cloud solutions; user satisfaction is improved. Attached Figure Description

[0027] Figure 1 A system architecture diagram of a preferred embodiment of the present invention; Figure 2 A preferred embodiment of the multimodal emotion perception loop flowchart of the present invention; Figure 3 A flowchart of an offline-to-online fusion scheduling algorithm according to a preferred embodiment of the present invention; Figure 4 A schematic diagram of the working principle of the active triggering module of a preferred embodiment of the present invention. Detailed Implementation

[0028] Definitions: Reviewing finite state machines: FSM (Compound Finite-State Machine) likely refers to a more complex system composed of multiple hierarchical or modular state machines, used to handle the variable decision-making processes that require contextual understanding in high-EQ interactions.

[0029] Figure 1 Description: The overall architecture of this invention comprises three layers: the edge device, the local edge server, and the cloud. Solid arrows represent data flow, including the process of the edge device uploading multimodal perception data and contextual information to the edge server, the edge server uploading complex requests and anonymized data to the cloud, and the cloud distributing models and knowledge to the edge, and the edge distributing response commands to the edge device. This architecture achieves optimized allocation and efficient collaboration of perception, computation, and decision-making capabilities.

[0030] The endpoint, local edge server, and cloud, among which: On the client side, it is used to collect users' multimodal data and perform local lightweight real-time processing and response execution; The local edge server is used to receive and fuse multimodal features and contextual information from the edge device to perform moderately complex sentiment computing, dialogue management and proactive triggering decisions. The cloud is used to run ultra-large-scale models, providing global knowledge management, long-term user profile storage, and model training optimization. The edge servers and the cloud communicate collaboratively through encrypted channels, forming a distributed intelligent processing architecture.

[0031] Preferably, the end side includes: The multimodal sensing module is used to collect voice, image, and physiological signal data through microphones, cameras, and sensors; The lightweight processing module is configured to run a lightweight model locally for wake word detection, speech recognition, preliminary facial expression judgment, and text sentiment analysis. The context management module is used to collect user location, time, and application status information; The emotional execution module is used to perform speech synthesis with emotional intonation and digital human facial expression rendering based on received instructions.

[0032] Preferably, the local edge server includes: The multimodal fusion module is used to integrate text, voice tone, facial expressions, and contextual information to generate a comprehensive user profile. The dialogue management module is used to maintain the dialogue state and select dialogue strategies based on user state profiles and scenario complexity. The proactive triggering module is used to analyze user behavior patterns based on a composite finite state machine to determine the timing and content of proactive interactions. Edge cache pools are used to store models and knowledge bases delivered from the cloud for quick local access.

[0033] Preferred cloud-based features include: A massive model pool for deploying and training language models and multimodal emotion models with hundreds of billions of parameters; A global knowledge base is used to store and update general knowledge, domain knowledge, and cultural trend information. Long-term memory units are used to securely store and retrieve user cross-session interaction summaries and preferences in vectorized form; The model factory is used to continuously optimize models and generate lightweight models through distillation technology, which are then distributed to the edge and endpoint.

[0034] Preferably, the system employs a dynamic scheduling strategy, including: Calculate the intent confidence and entity confidence of the user request on the client side; The complexity of the current scene is evaluated on the edge server as a function of the number of intents, entity diversity, and frequency of historical interactions. The confidence level and complexity are weighted and calculated. If the result is lower than a set threshold, the request is forwarded to the cloud for processing; otherwise, the processing is completed at the edge.

[0035] Preferably, the working method of the actively triggered module includes: The first-level state machine parses the multimodal context and generates abstract user behavior states; The second-level state machine selects the most suitable one from a set of pre-set high emotional intelligence question templates based on the user's behavior state. The selected question template is fed into the language model to generate natural language sentences, and context-related sentiment tone suggestions are added.

[0036] Edge development: Integrate sensor data acquisition using native Android / iOS frameworks.

[0037] Deploy lightweight models (such as MobileBERT) to handle local sentiment analysis.

[0038] Edge server deployment: Containerized services (affective computing, dialogue management modules) are managed using Kubernetes. Distilled medium-sized LLMs (such as DistilBERT+GPT-2 level models) are deployed.

[0039] Cloud-based training and inference: Training multimodal sentiment models based on LLMs with hundreds of billions of parameters (such as the GPT-4 architecture). Using vector databases (such as Pinecone) to store users' long-term memories.

[0040] Dynamic scheduling strategy: Define the scene complexity formula: Complexity = f(number of intents, entity diversity, frequency of historical interactions) Set a confidence threshold (e.g., 0.7) and trigger a cloud request when the confidence level falls below the threshold.

[0041] Figure 2 Description: This invention describes the emotion perception and interaction loop process. The loop begins in the multimodal perception stage, where the system collects multi-dimensional user data. It then enters the emotion state fusion calculation stage (typically performed on an edge server), where the perceived data is comprehensively analyzed and judged. Based on this state, the system invokes AI models at different levels to make high-EQ response decisions and generate responses. The generated response instructions are executed emotionally on the device side, ultimately affecting the user and completing an interaction. User feedback then initiates a new perception loop, thereby achieving continuous emotion understanding and interaction optimization. Long-term profiles and a global knowledge base stored in the cloud support this loop.

[0042] Figure 3 Description: This invention describes the dynamic scheduling decision-making process. After receiving a user request, the system first performs lightweight preprocessing and calculates the confidence level on the device side (using a mobile phone as an example in this diagram). Subsequently, the core scheduling algorithm performs two levels of judgment: first, it determines whether the combination of scene complexity and confidence level satisfies the conditions for edge processing; if not, it further judges the data sensitivity. For non-sensitive data, it is uploaded to the cloud for deep inference to obtain the best results; for sensitive or simple tasks, edge processing is prioritized to ensure response speed and privacy. This process achieves the optimal balance between performance, accuracy, and privacy.

[0043] Example 1: Intelligent Health Care Assistant 24 / 7 health and emotional care for elderly people living alone.

[0044] Perception phase: The microphone on the elderly's end continuously collects ambient sounds (such as coughing sounds, falling and impact sounds), the camera (after user authorization) captures facial expressions at regular intervals, and the wearable device uploads heart rate and blood oxygen data via Bluetooth.

[0045] Decision-making and generation phase: On the edge: The local lightweight model performs real-time analysis and detects suspected violent coughing sounds and abnormally high heart rate. It immediately sends the desensitized feature data (not the original audio) to the edge server.

[0046] Edge server: The multimodal fusion module integrates voice features, physiological data, and temporal context (late at night) to determine that the discomfort is "suspected to be caused by a cold." The dialogue management module triggers the proactive care process, calling the edge LLM to generate a greeting: "Sir, you seem to be coughing a lot. Have you caught a cold? Would you like me to contact your child or the community doctor for you?" (Optional) Cloud: If the elderly person activates the detailed consultation mode, the edge server will upload the anonymized data to the cloud. The cloud big model generates a more professional consultation dialogue based on the medical knowledge base and sends it to the edge for execution.

[0047] Execution phase: The device delivers a greeting in a gentle, concerned tone, and the virtual avatar on the screen displays a worried expression. If the elderly person confirms via voice that they need help, the system can automatically dial a pre-set emergency contact number.

[0048] Multimodal fusion (sound + physiological data) accurately determines health status.

[0049] A proactive care mechanism, rather than a passive response.

[0050] Privacy protection: The original audio is processed on the client side, and only the anonymized features are uploaded.

[0051] Example 2: Immersive Educational Companion Robot Children learn history through interactive educational experiences, such as engaging in dialogues with historical figures.

[0052] Perception Stage: The child selects the "Conversation with Li Bai" mode. The phone's camera is turned on to capture the child's facial expressions (curiosity, confusion) and voice questions.

[0053] Decision-making and generation phase: On the edge: Complete local speech recognition (ASR) and upload the text "Li Bai, which poem do you like the most?" and the "curious" emoji tag to the edge.

[0054] Edge Server: The dialogue management module determines that the current scenario is "Li Bai Poetry Appreciation". Since its knowledge base has cached Li Bai-related poems and background knowledge distributed from the cloud, it directly completes the reasoning at the edge and generates a response that matches Li Bai's character: "Haha, I love 'Bring in the Wine' the most. Don't you see the Yellow River's waters coming from the sky...", and adds the emotional tag "bold and unrestrained".

[0055] In the cloud: In the background, the cloud continuously uses the latest historical materials to optimize and upgrade the "Li Bai" character model, and uses model distillation technology to regularly distribute the lightweight model to the edge server to update the cache.

[0056] Execution phase: The client receives text and sentiment tags, and the TTS engine reads the poem in a bold and unrestrained tone. The avatar is also transformed into the image of Li Bai, accompanied by actions such as waving sleeves and drinking wine, providing an immersive experience.

[0057] A knowledge update mechanism that integrates edge, cloud, and endpoint processes ensures the freshness of content.

[0058] Edge caching and computing enable zero-latency interaction in role-playing.

[0059] Multimodal output (emotional TTS + Avatar actions) enhances immersion.

[0060] Example 3: Intelligent In-Vehicle Emotion Regulation System Inside the cockpit, the driver's emotions are monitored and adjusted in real time to ensure driving safety.

[0061] Perception phase: The in-car camera analyzes the driver's facial expressions (capturing frowns and yawns), the steering wheel sensor monitors heart rate fluctuations, and the microphone analyzes the tone of voice (whether it contains profanity).

[0062] Decision-making and generation phase: Mobile / In-vehicle Devices: The local model quickly determines the emotion as "road rage" or "fatigue driving." Due to safety concerns and extreme sensitivity to latency, processing is prioritized on edge servers.

[0063] Edge server: The sentiment computing module confirms the status. The dialogue management module triggers the corresponding proactive adjustment strategy. For "road rage," a soothing response is generated: "Calm down, listen to some soothing music?"; for "fatigue," a warning response is generated: "You have been driving continuously for 2 hours, we suggest you rest at the next service area." Execution phase: The vehicle system executes commands, such as playing music or navigating to a service area, while simultaneously announcing the commands in a calm and firm voice to avoid distractions caused by the system interaction itself.

[0064] In high-latency-sensitive scenarios, leverage the low-latency advantages of edge computing.

[0065] Expanding the application boundaries of emotion perception for proactive safety intervention.

[0066] A tiered response strategy is implemented, with different interaction commands corresponding to different emotional states.

[0067] Example 4: Intelligent Customer Service for High-End Retail Smart shopping assistants in luxury stores provide services to customers via tablets.

[0068] Perception phase: When a customer picks up a bag, the tablet camera captures the duration of their gaze and micro-expressions (joy, hesitation).

[0069] Decision-making and generation phase: Tablet device: The local model initially judged the expression as "interested but hesitant".

[0070] Edge servers (deployed within the mall): Integrating facial expression analysis, product information (inventory, price), and customer past preferences (anonymized historical records retrieved from the cloud), the system determines that the core concern might be "price" or "versatility." It then proactively triggers a conversation: "You have excellent taste; this is a limited edition. Would you like me to tell you about its design story and styling options?" Cloud-based: Provides a large knowledge base including the latest global product information and celebrity fashion photos, which can be accessed by edge servers.

[0071] Execution phase: The avatar on the tablet delivers a professional and elegant presentation, simultaneously displaying text, images, and video materials to facilitate the transaction.

[0072] Engage in proactive, emotional marketing by integrating business context (product information, user history).

[0073] A perfect combination of global knowledge in the cloud and real-time edge perception.

[0074] It enhances the "emotional intelligence" of AI customer service, making it more like a top sales expert.

[0075] Example 5: Cross-language Emotional Communication Assistant In international business meetings, we provide assistance to both parties in translating not only words and phrases, but also emotions and feelings.

[0076] Perception phase: The device collects the speech, tone of voice, and facial expressions of the speaker from Party A.

[0077] Decision-making and generation phase: Mobile / Conference Device: Complete local speech recognition and upload the text and emotional features (e.g., passionate, sad).

[0078] Edge server: First, accurate text translation is performed. Then, the sentiment processing module instructs the dialogue generation module to add sentiment annotations to the translated text, such as: "(In an excited and firm tone) I believe this plan must be implemented this week!" Cloud-based: Provides a powerful multilingual translation model and a cross-cultural emotional knowledge base (e.g., subtle differences in a certain expression across different cultures) to ensure the accuracy of emotional translation.

[0079] Execution phase: The TTS system reads the translated text aloud in another language, mimicking a similar enthusiastic tone, so that the B party can not only understand the content, but also feel the other party's emotional attitude, greatly improving communication efficiency.

[0080] By deeply integrating affective computing with machine translation, we can achieve communication that is "accurate, expressive, and elegant".

[0081] A cloud-based cross-cultural knowledge base ensures the universality and accuracy of emotional interpretation.

[0082] This expands the depth of chatbot applications in professional fields.

[0083] Example 6: A Daily Health Management Companion for Patients with Chronic Diseases Proactive care for diabetic patients in terms of daily diet, medication, and emotional management.

[0084] After the user finishes eating, the phone camera takes a picture of the plate (for rough food identification).

[0085] The user said via voice: "I feel a little dizzy today."

[0086] The Bluetooth-connected blood glucose meter automatically uploads the latest blood glucose measurement (high).

[0087] Decision-making and generation phase: On the edge: The local model quickly classifies food images (identifying high-carbohydrate foods), converts speech to text, and packages it with blood glucose data before sending it to the edge server.

[0088] Edge server: The multimodal fusion module combines data on "high-carbohydrate foods," "dizziness," and "high blood sugar," determining that "a sudden spike in postprandial blood sugar may cause discomfort." The proactive trigger module then activates, generating supportive messages and suggestions: "Your blood sugar is high; the pasta you just ate may have caused a rapid rise in blood sugar. Please remember to take your medication on time, drink plenty of water, and rest. Would you like me to schedule a doctor's appointment for you tomorrow?" Execution phase: The mobile phone will broadcast the above statement in a gentle and reminding tone, and after the user confirms, it will redirect to the appointment registration interface of the medical APP.

[0089] It demonstrates how to identify health abnormalities using multimodal data (blood glucose physiological signals, voice, and images) and proactively generate caring inquiries, including suggestions to contact a doctor.

[0090] Example 7: AR-based Interactive Museum Guide In the museum, visitors can interact with virtual historical figures and ask questions through AR glasses or a mobile app.

[0091] Perception phase: When visitors stop in front of the "Terracotta Warriors of Qin Shi Huang" exhibit, their mobile phone GPS and AR image recognition work together to confirm the area where they are located.

[0092] Decision-making and generation phase: Edge server: The dialogue management module confirms the scenario as "Terracotta Warriors Guided Tour". Since its edge cache pool already has a pre-installed "Qin Shi Huang" character knowledge package (including historical facts, character personality, and related cultural relic information) downloaded from the cloud, there is no need to request the cloud.

[0093] The role-playing engine on the edge server directly generated a proactive greeting in a tone befitting Qin Shi Huang's domineering presence: "My army, as fierce as tigers and wolves, is it awe-inspiring? What questions do you have?" Execution phase: The emotional execution module on the device / AR glasses receives the text and the emotional tag "majestic". It uses AR technology to overlay a virtual image of Qin Shi Huang onto the real exhibition hall and synthesizes and broadcasts the sentence with a deep and majestic voice tone.

[0094] It demonstrates how edge caching of specific character (Qin Shi Huang) knowledge drives the client to render the character's personalized voice and image (ARAvatar), achieving immersive interaction.

[0095] Example 8: Fatigue Monitoring and Intervention System for Public Transportation Drivers 1. Application Scenarios: Used for real-time status monitoring of professional drivers of buses, long-distance trucks, etc., to ensure public safety.

[0096] 2. Specific implementation process: Perception phase: The camera in the cockpit continuously monitors the driver's face and detects multiple yawns and excessive eyelid closure (slow blinking).

[0097] Decision-making and generation phase: Mobile / In-vehicle system: The local lightweight model processes the video stream in real time, identifies "micro-sleep" characteristics, and immediately triggers an alarm.

[0098] Edge server: Upon receiving the alert, the dialogue management module identifies it as "fatigue driving" with the highest priority. To avoid cloud latency, it generates the most direct intervention command directly at the edge: "Warning! You are driving while fatigued! Please stop and rest immediately in a safe area!" Execution phase: The system broadcasts a warning in a very loud and serious voice tone, while the screen flashes a red warning light and automatically sets the navigation destination to the nearest service area.

[0099] It demonstrates how to identify fatigue through facial expressions and proactively trigger strong intervention commands (warnings, suggestions to rest) rather than gentle conversation.

[0100] Example 9: Wealth Management Advisors at High-End Banks In the bank's VIP room, as customers talk with their account managers, a smart system provides real-time decision support via tablet computers.

[0101] Perception phase: When a customer hears "Fund Product A", the tablet camera captures their slight frown and leaning back as micro-expressions.

[0102] Decision-making and generation phase: Edge server: The multimodal fusion module combines micro-expressions (hesitation / resistance), product information (A is high risk), and anonymized user history preferences retrieved from the cloud (this customer has previously chosen low-risk products) to determine the customer's resistance to high-risk products.

[0103] The module proactively generates suggestions and displays them to the account manager (or allows direct conversation with the client via tablet): "Mr. Wang seems quite sensitive to volatility. We have a 'stable portfolio' with good historical drawdown control. Would you like to learn more about it?" Execution phase: New product information pops up on the tablet screen for account managers to refer to or to show directly to customers.

[0104] It demonstrates how to integrate micro-expressions (gaze / facial expressions), product information (risk level), and user history preferences to generate precise, personalized proactive marketing messages.

[0105] Example 10: A Synchronized Emotion Translation System for International Video Conferences Provides emotionally resonant real-time translation during online strategic negotiation meetings between Party A and Party B.

[0106] Perception phase: The spokesperson from Party A speaks faster and louder, and the camera captures that he leans forward and has a serious expression.

[0107] Decision-making and generation phase: Edge server: The emotion computing module analyzes voice signals (tone, speech rate) and visual signals (facial expressions, posture) to determine that the other party's emotion is "strong and resolute".

[0108] While translating the text, add sentiment metadata to the sentence: "<Sentence: Determined, Intensity: 8 / 10> We must reach an agreement this quarter." Execution phase: When the TTS module of Party B broadcast the Chinese translation "We must reach an agreement this quarter", it did not simply state the message in a flat manner, but rather imitated the firm and powerful tone of the US side, so that the Chinese representative could accurately feel the other party's attitude and sense of urgency.

[0109] It perfectly demonstrates how to analyze the emotional tone of the source language and inject it into the speech synthesis of the target language, achieving "emotional synchronization" that goes beyond literal translation.

[0110] This invention achieves global optimization by integrating edge-side lightweight sensing, edge multimodal fusion, and cloud-based long-term memory. It combines a composite finite state machine (FSM) proactive questioning mechanism with emotion computing, distinguishing it from traditional rule-based triggering.

[0111] By combining "online-offline fusion scheduling" with "multi-agent role-playing," and applying it to high-EQ interaction scenarios, the realism of the simulation can be enhanced.

[0112] Introducing a multimodal emotion perception loop solves the problem of misjudging emotion in single text (e.g., trembling voice + stiff facial expression → accurate emotion assessment). Average response latency <800ms (edge ​​local decision-making), 60% faster than pure cloud solutions; improved user satisfaction. Intelligent customer service: Real-time identification of customer emotions at the edge, proactively providing solutions (e.g., in the finance and telecommunications industries); Health care: Monitor the physiological data of elderly users on the device and trigger proactive inquiries when abnormalities occur (such as fall detection + low mood). Education and entertainment: multi-agent role-playing (such as dialogues between historical figures), cloud-generated scripts, and edge-coordinated multi-role interactions.

[0113] Figure 4 Description: The proactive triggering module of this invention combines the working mechanism of a Finite State Machine (FSM). The first-level FSM performs high-level abstraction on the fused original context (such as time, detected user behavior, and facial expressions) to parse out the macroscopic user state (such as "feeling down late at night"). Based on this state, the second-level FSM selects the most appropriate and specific question template from multiple options (such as choosing "concern about health" instead of "simple greeting") and ultimately generates a natural and emotionally intelligent proactive greeting. This two-level structure separates state judgment from content generation, enabling the system to accurately understand the scenario and flexibly provide a wide variety of responses.

[0114] The core decision-making module of the composite finite state machine (FSM) is mainly configured and runs on a local edge server.

[0115] I. Core Deployment Location: Local Edge Server The first level of FSM requires multimodal information fusion (such as combining text, facial expressions, voice tone, and context). This data comes from multiple sensors on the device and must be processed in a place that can aggregate all the data. A single device cannot efficiently complete this cross-modal fusion analysis.

[0116] While the decision-making process of FSM does not require the massive computing power of cloud-based models with hundreds of billions of parameters, it still exceeds the computing power range of lightweight models on the edge. Edge servers are capable of running medium-sized models (such as LLMs with 7B parameters), which is sufficient to support the complex state judgments and policy selections of FSM.

[0117] Proactive interaction prioritizes real-time performance. If FSM is placed in the cloud, network latency can severely impact the proactive experience (e.g., a greeting is sent only after the user has left). Deploying it at the edge ensures extremely fast response times (milliseconds).

[0118] FSM processes the fused feature data, not the raw audio and video streams. Placing FSM at the edge rather than in the cloud prevents sensitive raw data from leaving the local area network, better meeting data privacy and compliance requirements.

[0119] Although the core is at the edge, the edge and cloud also play indispensable supporting roles, forming a collaborative workflow: 1. End-side role: Front-end perception and preprocessing: The lightweight model on the edge acts as the "perceptual nerve ending" of the first-level FSM. It first performs local processing to generate preliminary, low-dimensional feature labels (e.g., `expression = sadness (confidence 85%)`, `sigh detected`), and then uploads these anonymized feature data to the edge server instead of the raw data stream.

[0120] Action execution: Receive and execute instructions issued by the edge server FSM after decision-making (e.g., play a sentence or display an emoticon).

[0121] 2. The role of the cloud: Model training and optimization: The cloud is responsible for training the model used in the second-level FSM. For example, a model is trained to learn "which response strategy has the highest success rate under what user state". The trained model is then distilled and distributed to the edge server.

[0122] Knowledge base support: When generating specific responses, the Level 2 FSM may need to query the latest knowledge base (such as news and cultural information). These query requests are sent to the cloud, and after the cloud returns the results, the edge FSM then incorporates them into the response generation.

[0123] Handling extremely complex situations: For extremely rare or complex scenarios that edge FSM cannot handle, the entire state judgment and decision request may be uploaded to the cloud for deep reasoning.

[0124] III. Workflow Example (FSM at the Edge) Taking "detecting users' low mood and proactively comforting them" as an example, this illustrates how FSM works in a three-tier architecture: 1. End-side (sensing): The microphone detected a sigh, and the local model flagged the event as `sigh`.

[0125] The camera detected a frown, and the local model labeled the expression as `sad`.

[0126] The tagged structured event data is sent to the edge server.

[0127] 2. Edge Server (Decision-Making FSM Core): Level 1 FSM: Receives event data from the endpoint, combines it with the time context (`Time: 02:00AM`), and performs a fusion judgment. Outputs an abstract state: `State: Late night, feeling down`.

[0128] Level 2 FSM: Based on the above status, select the "Caring for Health" policy from the policy library. Invoke the mid-sized LLM deployed at the edge to generate a natural language sentence from the policy template: "You seem a little tired today, would you like to talk to me?", with a `gentle tone`.

[0129] The generated response text and tone instructions are sent back to the endpoint.

[0130] 3. End-side (Execution): The system receives instructions, reads them aloud in a "gentle" tone using the local TTS engine, and displays a concerned expression on the screen.

[0131] 4. Cloud-based (backend support): The strategy selection model of the second-level FSM is regularly optimized using a large amount of dialogue data, and the new model is then distributed to the edge server for updates. The composite finite state machine does not exist in isolation at any one layer. Instead, its core decision-making logic is deployed on an edge server, coordinating resources from both the edge and the cloud to jointly complete proactive interactive tasks with high emotional intelligence. This deployment method is one of the essences of the architecture design of this invention.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end cloud trinity high emotional quotient proactive chat robot system, characterized in that, It includes an end side, a local edge server and a cloud side, wherein: The end side is used to collect multi-modal data of users and perform local lightweight real-time processing and response execution; The local edge server is used to receive and fuse multi-modal features and context information from the end side, perform medium complexity sentiment calculation, dialogue management and active triggering decision; The cloud side is used to run super large scale models, provide global knowledge management, long-term user portrait storage and model training optimization; The end side, the local edge server and the cloud side perform collaborative communication through an encrypted channel to form a distributed intelligent processing architecture.

2. The system of claim 1, wherein, The end side includes: A multi-modal perception module for collecting voice, image and physiological signal data through microphones, cameras and sensors; A lightweight processing module configured to run lightweight models locally to perform wake-up word detection, speech recognition, expression preliminary judgment and text emotion analysis; A context management module for collecting user location, time and application state information; An emotional execution module for performing voice synthesis with emotional intonation and digital human expression rendering according to received instructions.

3. The system of claim 1, wherein, The local edge server includes: A multi-modal fusion module for fusing text, voice intonation, expression and context information to generate a comprehensive user state portrait; A dialogue management module for maintaining dialogue state and selecting dialogue strategies according to the user state portrait and scene complexity; An active triggering module for analyzing user behavior patterns based on a composite finite state machine to determine the timing and content of active interaction; An edge cache pool for storing models and knowledge bases downloaded from the cloud for local fast calling.

4. The system of claim 1, wherein, The cloud side includes: A super large scale model pool for deploying and training trillion parameter level language models and multi-modal sentiment models; A global knowledge base for storing and updating general knowledge, domain knowledge and cultural trend information; A long-term memory unit for securely storing and retrieving user cross-session interaction summaries and preferences in vectorized form; A model factory for continuously optimizing models and generating lightweight models through distillation technology to be delivered to the edge and the end side.

5. The system of claim 1, wherein, The system adopts a dynamic scheduling strategy, including: Calculating the intent confidence and entity confidence of user requests at the end side; Evaluating the complexity of the current scene at the edge server, which is a function of the number of intents, entity diversity and historical interaction frequency; Weighted calculation of the confidence and complexity, if the result is lower than the set threshold, the request is forwarded to the cloud for processing, otherwise the processing is completed at the edge.

6. The system of claim 3, wherein, The working method of the active triggering module includes: A first level state machine analyzes multi-modal context to generate an abstract user behavior state; A second level state machine selects the most suitable one from a plurality of high emotional intelligence question templates according to the user behavior state; The selected question template is sent to the language model to generate a natural language sentence and add context related emotional intonation suggestions.

7. The system of any one of claims 1-6, wherein, The system is applied to health care scenarios, wherein: The multi-modal data includes coughing sound, fall impact sound, heart rate and blood oxygen data; The active triggering decision includes automatically generating a care inquiry and suggesting contacting an emergency contact when abnormal health data is identified.

8. The system of any one of claims 1-6, wherein, The system is applied to an education accompanying scene, wherein: The edge cache pool stores specific role knowledge data; The emotional execution module is configured to render corresponding voice tone and digital human image according to the role personality, and realize immersive role-playing interaction.

9. The system of any one of claims 1-6, wherein, The system is applied to an intelligent vehicle scene, wherein: The multi-modal data includes driver facial expressions, heart rate data, and voice tone; The active triggering decision includes actively triggering music playing or rest suggestion instructions to adjust the driver's mood when road rage or fatigue is identified.

10. The system of any one of claims 1-6, wherein, The system is applied to a commercial guide scene, wherein: The multi-modal data includes customer gaze time and micro-expression on goods; The edge server fuses the multi-modal data, goods inventory information, and user historical preferences to generate personalized active marketing rhetoric.

Images