Active multi-modal education agent system and method

Abstract

The invention discloses an active multi-mode education agent system and method, and belongs to the technical field of artificial intelligence education. The system adopts an end-cloud collaborative architecture and comprises two parts, namely end-side equipment and a cloud micro-service system. The end side device is responsible for collecting multi-modal data (emotion data, behavior data and content data) of a learner in real time and carrying out preprocessing and feature extraction. The cloud micro-service system comprises an API gateway service, a state reasoning service, an active response generation service, a multi-modal content generation service and a TTS service, and is used for processing and analyzing the multi-modal data and generating a personalized active response. According to the method, the confusion state of a learner can be actively perceived, the cognitive state is inferred based on a multi-modal data fusion algorithm, personalized education content is generated according to the cognitive state, a perfect service degradation strategy is provided, and the reliability and learning effect of the system are improved.

Description

Technical Field

[0002] This invention relates to the field of artificial intelligence education technology, specifically to a proactive multimodal educational intelligent agent system and method based on an edge-cloud collaborative architecture, used to perceive learner status in real time, infer cognitive status, and generate personalized proactive responses. Background Technology

[0003] With the development of artificial intelligence technology, educational AI systems are being applied more and more widely in the education field. However, current educational AI systems face the following technical challenges:

[0004] 1. Passive response mode: Existing systems mainly adopt a passive response mode, which only provides help when learners actively ask questions. They cannot proactively perceive the learner's state of confusion, resulting in learners not being able to get help in a timely manner when they encounter difficulties.

[0005] 2. Limitations of single-modal perception: Most systems only interact based on a single modality of text or speech, lacking the fusion perception of multimodal data such as learners' emotions and behaviors, and thus failing to fully understand the learners' true state.

[0006] 3. Lack of edge-cloud collaboration: Existing systems are either entirely deployed in the cloud (high response latency) or entirely deployed on the edge (limited computing power), failing to fully leverage the advantages of both the edge and the cloud.

[0007] 4. Lack of personalization: It is unable to generate personalized educational content based on learners' real-time cognitive, emotional, and behavioral data, resulting in poor educational outcomes.

[0008] 5. Low service reliability: The lack of a sound service degradation strategy means that the entire system is prone to crashing when a service component fails, affecting the user experience.

[0009] Therefore, a novel proactive multimodal educational intelligent agent system is needed, which can proactively perceive the learner's state, achieve efficient state reasoning and response generation based on an edge-cloud collaborative architecture, and generate personalized educational content according to the learner's state. Summary of the Invention

[0010] The purpose of this invention is to provide an active multimodal educational intelligent agent system and method, solving the technical problems of passive response, single-modal perception, and lack of edge-cloud collaboration in existing educational AI systems, and achieving:

[0011] 1. Real-time perception of learners' multimodal states (emotions, behaviors, cognition)

[0012] 2. Achieving efficient state reasoning and response generation based on an edge-cloud collaborative architecture.

[0013] 3. Proactively generate personalized educational content based on learner status.

[0014] 4. Possesses a comprehensive service degradation strategy to improve system reliability.

[0015] Technical solution

[0016] The technical solution of this invention includes an active multimodal educational intelligent agent system and method, as detailed below:

[0017] System Architecture:

[0018] This system adopts an edge-cloud collaborative architecture, mainly consisting of two parts: edge devices and a cloud-based microservice system.

[0019] 1. End-side equipment:

[0020] - Perception Module: Collects learners' emotional data (facial expressions, tone of voice), behavioral data (gaze duration, motor stability, touch behavior), and content data (learning content).

[0021] - Lightweight processing module: Performs preprocessing and lightweight analysis on the perceived data, extracting feature vectors.

[0022] - Communication module: Communicates with cloud services in real time, uploads processed data, and receives responses from the cloud.

[0023] 2. Cloud-based microservice system:

[0024] - API Gateway Service: Unified entry point, authentication and authorization, request rate limiting, service routing

[0025] - State reasoning service: Reasoning about cognitive states (confusion, focus, fatigue, etc.) based on multimodal data.

[0026] - Proactive Response Generation Service: Generates personalized responses based on cognitive state.

[0027] - Multimodal content generation service: Generates multimodal content such as text and images.

[0028] - TTS service: Generate voice response

[0029] Data flow:

[0030] 1. The edge-side sensing module collects learners' multimodal data in real time.

[0031] 2. The lightweight processing module on the edge performs data preprocessing and feature extraction.

[0032] 3. Data is uploaded to the cloud API gateway via the communication module.

[0033] 4. After authenticating and rate-limiting the request, the API gateway routes it to the state reasoning service.

[0034] 5. State reasoning service: Reasoning about cognitive states based on multimodal data.

[0035] 6. The proactive response generation service generates personalized responses based on cognitive state, invoking multimodal content generation and TTS services.

[0036] 7. The generated response is returned to the end device via the API gateway.

[0037] 8. The endpoint device displays response content, including text, voice, and visual cues.

[0038] Core Algorithm:

[0039] 1. Multimodal data fusion algorithm: Employs a weighted fusion algorithm to fuse sentiment data, behavioral data, and content data to generate a comprehensive feature vector.

[0040] 2. Cognitive State Inference Model: Based on a machine learning model, this model infers the learner's cognitive state from a comprehensive feature vector.

[0041] 3. Proactive Response Decision-Making Mechanism: Based on the learner's cognitive state, learning content, and learning progress, decide whether proactive intervention is needed and, if so, what intervention methods to use.

[0042] Service degradation strategy:

[0043] When a microservice component fails, the system automatically switches to degrade mode:

[0044] - Switch to InMemoryRedis memory emulation implementation when Redis service fails.

[0045] - When the state reasoning service fails, use preset rules for state reasoning.

[0046] - When the multimodal content generation service fails, use a local preset template to generate content.

[0047] - When the TTS service fails, only a text response is returned.

[0048] Technical effect

[0049] The technical effects of this invention include:

[0050] 1. Proactive Perception and Intervention: Able to proactively perceive learners' confusion and provide timely assistance to improve learning efficiency.

[0051] 2. Multimodal Fusion: By comprehensively considering multimodal data on emotion, behavior, and content, learners can gain a more holistic understanding of their learning status.

[0052] 3. Edge-Cloud Collaboration Advantages: Combining low latency on the edge with powerful computing capabilities in the cloud, it enables efficient real-time response.

[0053] 4. Personalized Education: Generates personalized educational content based on the learner's real-time status to improve learning outcomes.

[0054] 5. High reliability: Employing a microservice architecture and degradation strategies improves system availability and fault tolerance.

[0055] 6. Scalability: Modular design facilitates future feature expansion and performance optimization. Detailed Implementation

[0056] Hardware configuration

[0057] Components Configuration (including the original configuration and higher configurations) use End-side equipment RK3588 processor + Android 12 system Real-time perception and lightweight processing Camera 1080P HD camera Collect facial expression and gaze data microphone Dual microphone array Collect voice data sensor Accelerometer, light sensor Collect behavioral and environmental data cloud server 8-core CPU + 16GB RAM Running microservice components

[0058] Software Configuration

[0059] Software components Minimum version use Python 3.10 Developing microservices FastAPI / difyAPI / langchain API 0.100.0 Build API services Uvicorn 0.23.0 ASGI server Redis 7.0+ caching and rate limiting Pydantic 2.0+ Data model validation Python-Jose 3.3.0 JWT Certification

[0060] System Deployment

[0061] 1. Edge Deployment: Deploy the sensing module, lightweight processing module, and communication module onto the RK3588+Android device.

[0062] 2. Cloud Deployment: Deploy each microservice component using Docker containers.

[0063] 3. Service Startup: Use the start_services.py script to start all cloud services.

[0064] 4. Monitoring Configuration: Configure Prometheus + Grafana to monitor the system's running status.

[0065] Use cases

[0066] Scenario 1: Math Problem Solving Tutoring

[0067] 1. Learners view a math problem on their device: 2x + 3 = 7

[0068] 2. The end-effector sensed the learner's confused expression and prolonged gaze.

[0069] 3. The data was uploaded to the cloud, and the state reasoning service calculated the perplexity to be 0.86 and the focus to be 0.8.

[0070] 4. Proactively respond to generate service care, allowing users to decide whether to generate step-by-step instructions.

[0071] 5. Use the multimodal content generation service to generate narration content and the TTS service to generate audio.

[0072] 6. The endpoint displays the response: "Don't worry, let's solve this equation step by step: 2x + 3 = 7...", while simultaneously showing a visual cue.

[0073] Scenario 2: Fatigue Reminder

[0074] 1. Learners who study for extended periods exhibit fatigued facial expressions and reduced motor stability as perceived by their endoscopes.

[0075] 2. The state reasoning service calculated the fatigue level to be 0.85.

[0076] 3. Proactively respond to the service decision to generate a rest reminder.

[0077] 4. The device displays the response: "You've been studying for a long time. We suggest you take a 5-minute break before continuing!" Attached Figure Description

[0078] Figure 1 EduMind Active Multimodal Educational Intelligent Agent System Architecture Diagram

[0079] Figure 2 EduMind Active Multimodal Educational Intelligent Agent Data Flow Diagram

[0080] Figure 3 Cognitive State Reasoning Model Diagram

[0081] Comparison with existing technologies

[0082] Comparison Dimensions This invention Existing technology Advantages Response mode Active passive Promptly identify learners' difficulties and proactively offer assistance. Perception modality Multimodal (emotion + behavior + content) Single-mode or dual-mode To gain a more comprehensive understanding of learners' status Architecture Design End-to-cloud collaboration Pure cloud or pure client-side Low latency, high performance, and resource optimization Personalization Real-time personalization Preset Personalization The educational content is dynamically adjusted based on real-time status. reliability High (microservices + degradation strategy) Medium to low (monochrome architecture) Service decoupling, strong fault tolerance

[0083] Variations and extensions of the invention

[0084] Variations and extensions of the present invention include:

[0085] 1. Expand to other subjects: In addition to mathematics, it can be extended to other subjects such as physics, chemistry, and language arts.

[0086] 2. Supports more end devices: In addition to RK3588+Android, it supports other platforms such as iOS and Windows.

[0087] 3. Introduce more sensory modalities: such as electroencephalogram (EEG) signals, heart rate, and other physiological signals.

[0088] 4. Enhance personalized models: Combine learners' historical data to build more accurate personalized models.

[0089] 5. Supports group learning scenarios: Extends to classroom teaching scenarios, supporting multiple learners to use simultaneously.

[0090] Summarize

[0091] This invention provides a proactive multimodal educational intelligent agent system and method based on an edge-cloud collaborative architecture. By integrating multimodal data on emotion, behavior, and content, it infers the learner's cognitive state in real time and generates personalized proactive responses. This system possesses the advantages of edge-cloud collaboration, high reliability, and strong scalability, effectively improving learning efficiency and experience, and has broad application prospects.

Claims

1. An active multi-modal educational agent system, characterized in that, The application relates to an intelligent learning system, comprising: an end-side device for collecting multi-modal data of a learner in real time, wherein the multi-modal data comprises emotional data, behavioral data and content data; and a cloud micro-service system in communication connection with the end-side device, for processing and analyzing the multi-modal data to generate personalized active responses; wherein the cloud micro-service system comprises: an API gateway service as a unified entrance for authentication and authorization, request flow control and service routing; a state inference service for inferring a cognitive state of the learner based on the multi-modal data, wherein the cognitive state comprises confusion degree, concentration degree and fatigue degree; an active response generation service for generating personalized active responses according to the cognitive state; a multi-modal content generation service for generating multi-modal educational content; and a TTS service for generating voice responses. The end-side device comprises: a perception module for collecting emotional data, behavioral data and content data of the learner; a lightweight processing module for pre-processing and feature extraction of the multi-modal data; and a communication module for communication with the cloud micro-service system. The emotional data comprises facial expression data and voice tone data; the behavioral data comprises gaze duration data, motion stability data, touch behavior data and environmental data; and the content data comprises learning content text and formula data. The state inference service adopts a multi-modal data fusion algorithm to perform weighted fusion on the emotional data, the behavioral data and the content data to generate a comprehensive feature vector, and then infers the cognitive state based on a preset machine learning model. In the multi-modal data fusion algorithm, the emotional data weight is 0.3, the behavioral data weight is 0.4 and the content data weight is 0.

3. The active response generation service decides whether active intervention is needed and intervention methods according to the cognitive state, the learning content and the learning progress of the learner, wherein the intervention methods comprise step-by-step explanation, encouragement and rest prompting. The cloud micro-service system further comprises a service degradation module for automatically switching to a degradation mode when a certain micro-service component fails, wherein the degradation mode comprises: switching to an InMemoryRedis memory simulation implementation when a Redis service fails; using preset rules for state inference when the state inference service fails; using a local preset template to generate content when the multi-modal content generation service fails; and returning only a text response when the TTS service fails. The application further relates to an intelligent learning method, comprising the following steps: an end-side device collects multi-modal data of a learner in real time, wherein the multi-modal data comprises emotional data, behavioral data and content data; the end-side device pre-processes and extracts features of the multi-modal data; the end-side device uploads the processed data to a cloud micro-service system; an API gateway service of the cloud micro-service system performs authentication and authorization, request flow control and service routing on the request; a state inference service of the cloud micro-service system infers a cognitive state of the learner based on the multi-modal data; an active response generation service of the cloud micro-service system generates personalized active responses according to the cognitive state; and the active response generation service returns the active responses to the end-side device. ​ 2. The active multi-modal educational agent system of claim 1, wherein, ​ ​ ​ ​ 3. The active multi-modal educational agent system of claim 2, wherein, ​ 4. The active multi-modal educational agent system of claim 1, wherein, ​ 5. The active multi-modal educational agent system of claim 4, wherein, ​ 6. The active multi-modal educational agent system of claim 1, wherein, ​ 7. The active multi-modal educational agent system of claim 1, wherein, ​ ​ ​ ​ ​ 8. An active multi-modal education agent method, comprising: ​ ​ ​ ​ ​ ​ ​ The active response generation service calls the multi-modal content generation service to generate multi-modal educational content, and calls the TTS service to generate a voice response; The cloud micro-service system returns the generated response to the end-side device; The end-side device presents the response content, including text, voice, and visual cues.

9. The active multi-modal educational agent method of claim 8, wherein, The step of inferring the cognitive state by the state inference service comprises: The affective data, behavior data, and content data are weighted and fused to generate a comprehensive feature vector; The comprehensive feature vector is input into a preset machine learning model to infer the confusion level, concentration level, and fatigue level of the learner.

10. The active multi-modal educational agent method of claim 8, wherein, The step of generating personalized active responses by the active response generation service comprises: When the confusion level of the learner is greater than 0.7 and the concentration level is greater than 0.5, a high-priority step-by-step explanation is generated; When the fatigue level of the learner is greater than 0.8, a medium-priority rest reminder is generated; In other cases, no active response is generated.

Images