Learning path real-time adaptive adjustment method and system based on multi-modal behavior perception

Abstract

The invention discloses a learning path real-time adaptive adjustment method and system based on multi-modal behavior perception, and belongs to the crossing field of education technology and artificial intelligence. The method comprises the steps that a terminal layer collects multi-modal data such as vision, voice, physiology and interaction behaviors of a learner in real time and carries out local privacy processing; the edge layer adaptively fuses multi-modal features through a dynamic attention mechanism, and generates millisecond path adjustment suggestions by using a lightweight element reinforcement learning model and combining with a local knowledge graph; and the cloud layer aggregates a global information optimization model through federal learning, optimizes a multi-target path strategy by using a meta-reinforcement learning framework, and regularly and safely synchronizes optimization parameters to the edge layer. According to the invention, through a'end-edge-cloud 'three-level collaborative architecture, the defects of a traditional system in the aspects of real-time performance, multi-modal fusion precision and privacy protection are effectively overcome, low-delay, high-accuracy and high-security learning path dynamic personalized adjustment is realized, and the learning efficiency and the user experience are remarkably improved.

Description

Technical Field

[0001] This invention relates to the intersection of educational technology and artificial intelligence, and particularly to a personalized learning support technology. Specifically, this invention provides a method and system for real-time adaptive adjustment of learning paths based on multimodal behavior perception. By collecting and fusing learners' visual, speech, physiological, and behavioral multimodal data in real time, and while protecting privacy, it utilizes an edge-cloud collaborative computing architecture and meta-reinforcement learning technology to achieve millisecond-level dynamic and personalized adjustment of learning paths. Background Technology

[0002] With the development of educational informatization, personalized learning has become key to improving the quality of education. Its core lies in dynamically adjusting the learning content, sequence, and difficulty based on the learner's real-time status, i.e., achieving adaptive learning paths. However, existing adaptive learning path systems face three major challenges in practical applications.

[0003] First, real-time performance is severely lacking. Most systems employ a centralized cloud processing model. Learner status data (such as video and audio) needs to be uploaded to a remote server for analysis before the decision is returned to the terminal. This process typically results in delays of hundreds of milliseconds or even seconds. Research shows that feedback delays exceeding 200 milliseconds can significantly distract learners, disrupt learning immersion, and thus reduce learning efficiency.

[0004] Second, multimodal data fusion is inefficient. Learning states are a comprehensive reflection of multiple dimensions, including cognition, emotion, and behavior, requiring the integration of multimodal information such as facial expressions, voice tone, physiological signals, and interactive behaviors for comprehensive judgment. Existing methods often employ simple feature concatenation or post-processing decision fusion, failing to effectively capture the complex nonlinear relationships and complementarities between modalities. For heterogeneous data with varying feature scales and sampling rates, traditional methods often suffer from poor fusion results due to feature space mismatch and noise sensitivity, limiting the accuracy of state recognition.

[0005] Third, there are risks of privacy breaches and behavioral interference. Multimodal data, especially biometric features such as facial and voice data, constitutes sensitive personal information. Uploading raw data to the cloud poses a risk of leakage. Furthermore, ubiquitous sensing devices can easily trigger learners' "observation anxiety," leading to unnatural behavior, loss of ecological validity in the collected data, and consequently affecting the accuracy of path adjustment.

[0006] In recent years, edge computing and federated learning technologies have provided new ideas for addressing the above problems. Edge computing pushes computing tasks to the edge of the network, which can significantly reduce latency. Federated learning allows models to be trained collaboratively without leaving the device, which helps protect privacy. However, existing educational application solutions that combine these technologies still have obvious limitations: (1) Edge devices have limited computing power and lack efficient, lightweight multimodal fusion algorithms that can run in real time; (2) There is a lack of close coordination between cloud-based global model optimization and edge-based local real-time decision-making, resulting in lagging policy updates and difficulty in quickly adapting to individual differences and dynamic scenarios; (3) The system optimization goal is singular, often focusing only on knowledge mastery and ignoring the balance of multiple dimensions such as learning efficiency, emotional experience, and cognitive load.

[0007] Therefore, there is an urgent need for a novel adaptive learning path method and system that can deeply integrate multimodal perception, edge intelligence, privacy computing and multi-objective optimization technologies to achieve low-latency and high-precision personalized learning guidance while ensuring privacy and security. Summary of the Invention

[0008] (a) Purpose of the invention The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for real-time adaptive adjustment of learning paths based on multimodal behavior perception. This method aims to achieve millisecond-level response, high-precision judgment, and strong privacy protection of learning paths through an innovative three-level collaborative architecture of "terminal-edge-cloud," a dynamic attention multimodal fusion mechanism, and a global optimization framework that integrates federated learning and meta-reinforcement learning. This comprehensively improves the adaptability, effectiveness, and security of personalized learning.

[0009] (II) Technical Solution To achieve the above objectives, the present invention adopts the following technical solution: On the one hand, a method for real-time adaptive adjustment of learning paths based on multimodal behavior perception is provided, including the following steps: S1: Multimodal Data Acquisition and Local Privacy Processing on the Terminal. Multimodal raw data, including learners' visual, speech, physiological signals, and platform interaction behaviors, is simultaneously acquired through terminal devices (such as smart tablets, embedded cameras, microphones, and wearable sensors). Locally on the terminal, the raw data undergoes preprocessing (such as facial feature point detection, speech segmentation, and signal filtering) and feature extraction. Crucially, local differential privacy perturbations are applied to raw data containing sensitive biometric information (such as facial image frames and raw audio clips) to generate desensitized feature vectors that meet privacy protection requirements; generalized desensitization processing is also performed on behavioral log data.

[0010] S2: Edge-based Dynamic Attention Multimodal Fusion. The desensitized feature vectors of each modality are transmitted to a nearby edge server via a low-latency network (such as Time-Sensitive Networking (TSN)). The edge server runs a lightweight dynamic attention multimodal fusion network. This network designs a feature encoding branch for each modality and includes a dynamic attention weight generator. This generator calculates a set of normalized attention weights in real time based on the current input modal features and their context. Finally, a comprehensive feature vector representing the learner's current cognitive and emotional state is generated through weighted summation. .

[0011] S3: Real-time edge decision-making based on lightweight meta-reinforcement learning. The edge server is pre-loaded with a subject knowledge graph and a lightweight meta-reinforcement learning model (such as a compressed deep Q-network). This model uses the state representation generated in step S2. Taking the current learning node as input, the output is an "action," namely, an immediate path adjustment suggestion. Suggestion types may include: recommending specific learning resources, adjusting the difficulty of subsequent tasks, suggesting rest periods, or changing the learning pace. This decision-making process is completed at the edge, ensuring extremely low latency (target <200ms).

[0012] S4: Cloud-based Federated Learning and Meta-Reinforcement Learning Global Optimization. The cloud server does not access the raw data. Instead, it receives encrypted model parameter updates (such as gradients or model weights) from numerous edge servers through a federated learning framework, performing secure aggregation to train and update a more powerful global model. Simultaneously, a meta-reinforcement learning optimizer runs in the cloud. Guided by a multi-objective reward function (comprehensively considering learning efficiency, mastery, satisfaction, cognitive load, etc.), it optimizes strategies based on experience across learners and learning tasks, enabling the system to quickly adapt to new scenarios.

[0013] S5: Secure synchronization and closed-loop update of cloud-edge parameters. The cloud encrypts and compresses the optimized global model parameters and distributes them to each edge server through a high-efficiency communication channel (such as 5G URLLC). The edge servers use these parameters to update their local models, improving the accuracy of the next decision. The edge servers also periodically upload their local model parameters to the cloud to participate in a new round of federated aggregation, thus forming a continuously evolving closed-loop system.

[0014] On the other hand, a system for implementing the above method is provided, which adopts a three-level collaborative architecture of end-edge-cloud: Terminal layer: Composed of learning devices integrating multimodal sensors and local processing units, responsible for data acquisition, preliminary processing and privacy protection.

[0015] Edge layer: Consists of servers deployed at the edge of the regional network, with core workloads including dynamic attention fusion module, lightweight meta-reinforcement learning decision module, and local knowledge graph.

[0016] The cloud layer consists of a high-performance server cluster located in a data center, with its core workload being federated learning aggregation services and a meta-reinforcement learning global optimizer. The three layers are connected via wired / wireless networks, enabling the secure and efficient flow of data, control commands, and model parameters.

[0017] (III) Beneficial Effects Compared with the prior art, the present invention has the following significant advantages: 1. Achieved millisecond-level real-time adjustment: By deploying the core state awareness and decision reasoning tasks on the edge, the end-to-end path adjustment latency was significantly reduced from the 450ms level of traditional cloud solutions to about 156ms, meeting learners' psychological expectations for immediate feedback and improving learning continuity and immersion.

[0018] 2. Improved accuracy of multimodal state perception: The proposed dynamic attention fusion mechanism can adaptively weight modal information according to the context, effectively solving the problems of modality imbalance and noise interference. Experiments show that its learning state recognition F1 score reaches 0.89, which is about 15% higher than the traditional feature concatenation method.

[0019] 3. A strong privacy protection barrier has been built: A full-link privacy protection solution of "terminal local differential privacy + edge feature encrypted transmission + cloud-based federated learning" is adopted to ensure that original sensitive data does not leave the terminal and that model training does not require centralized data. Testing has shown that this effectively reduces the success rate of member inference attacks to below 10%.

[0020] 4. Achieved intelligent multi-objective optimization: Through the cloud-based meta-reinforcement learning framework, the system can balance multi-dimensional objectives such as learning efficiency, knowledge acquisition, and user experience, and can learn from a large amount of cross-scenario experience to quickly generate appropriate strategies for new learners or new content, achieving a higher level of personalization.

[0021] 5. Enhanced system applicability and robustness: Edge computing enables the system to provide smooth localized services even in weak network or fluctuating network environments. In a pilot project with a bandwidth of 10Mbps in a rural school, the system still maintained 98% smoothness of interaction, effectively promoting the equitable access to high-quality educational resources. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall system architecture provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the entire process of multimodal data processing and path adjustment provided in the embodiments of the present invention.

[0025] Figure 3 This is a schematic diagram of the dynamic attention multimodal fusion network structure used in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the collaborative optimization mechanism of federated learning and meta-reinforcement learning involved in the embodiments of the present invention. Detailed Implementation

[0027] The technical solution of the present invention will now be described in more detail and completely with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0028] Example 1: Configuration and Interaction of System Core Components refer to Figure 1 The overall system architecture of the present invention includes a terminal layer 101, an edge layer 102, and a cloud layer 103.

[0029] The terminal layer 101 includes the learner's smart terminal device, such as a learning tablet or a smart desk. This device integrates: High-definition camera (used to capture facial video streams).

[0030] Microphone array (for directional voice acquisition).

[0031] Bluetooth interface (for connecting wrist-worn physiological sensors, such as ECG PPG and skin conductance GSR).

[0032] The local processing unit (such as the ARM Cortex-A series processor) runs the data acquisition and preprocessing software.

[0033] The edge layer 102 consists of edge servers deployed within the classroom local area network or the district / county education data center. These servers are equipped with GPU accelerator cards (such as NVIDIA Jetson AGX Orin) and deploy the following software modules: Dynamic attention fusion engine: achieving, for example Figure 3 The network shown.

[0034] Lightweight DQN decision maker: The number of parameters is approximately 2MB, obtained from the cloud through knowledge distillation.

[0035] Local Neo4j graph database: stores the knowledge point graph of the current course.

[0036] Secure communication agent: Responsible for encrypted communication with the terminal and the cloud.

[0037] The cloud layer 103 consists of a high-performance computing cluster located in a public cloud or a dedicated educational cloud, and mainly runs: Federated Learning Coordination Server: Developed based on the FATE framework, it manages the registration, task distribution, and secure aggregation of edge nodes.

[0038] Meta-reinforcement learning training cluster: The Ray framework is used to train meta-policy models in a distributed manner (such as an extension based on the TD3 algorithm).

[0039] Model repository and distribution service: Manages different versions of global models and is responsible for encrypted distribution.

[0040] Example 2: Specific Implementation of Dynamic Attention Multimodal Fusion Network refer to Figure 3 This section details the working mechanism of the Dynamic Attention Multimodal Fusion Network 302.

[0041] Suppose there are four modalities: visual (V) 301a, speech (A) 301b, physiological (P) 301c, and behavioral (B) 301d. The desensitization feature vector for each modality... As input.

[0042] First, each feature vector is mapped to a unified feature space through an independent feature encoding branch (e.g., a two-layer fully connected network with ReLU activation), resulting in... .

[0043] Then, these encoded features are concatenated to obtain... .

[0044] then, The weights are fed into the dynamic attention weight generator 303. This generator is a small neural network that can be designed as follows: Attention Score: For each mode Calculate a fraction Attention weights:

[0045] in, and These are learnable parameters. This is how they are obtained. The sum is 1, and varies with the input. Dynamic changes.

[0046] Finally, a weighted summation is performed at feature fusion layer 304 to obtain the comprehensive state representation: .

[0047] During the training phase, the network was trained end-to-end using a large amount of multimodal data labeled with state tags such as "focused", "confused", and "bored". The loss function combined the cross-entropy loss of state classification with a regularization term that encourages weight diversity.

[0048] Example 3: Cloud-based optimization process combining federated learning and meta-reinforcement learning refer to Figure 4 This explains how the cloud layer collaboratively optimizes the global model.

[0049] Federal learning phase ( Figure 4 (Left half) The cloud coordination server directs data to the registered edge nodes. Deploy the current global multimodal fusion model The parameters.

[0050] Each edge node performs local data (which has been anonymized) on... After several rounds of training, the local model update is obtained. .

[0051] Each node uses a homomorphic encryption algorithm to... Encrypt to obtain And upload it to the cloud.

[0052] The cloud aggregator 402 securely aggregates encrypted updates (utilizing the additivity of homomorphic encryption) to obtain... ,in The weights are dynamically calculated based on the amount of node data or the quality of updates.

[0053] Decrypt the aggregated results in the cloud (or directly update the global model in its encrypted state), and update the global model. .

[0054] Meta-reinforcement learning stage ( Figure 4 (Right half) The meta-reinforcement learning optimizer 403 maintains a meta-policy model. .

[0055] It treats a set of different learning tasks (e.g., different students or different course chapters) as different reinforcement learning environments.

[0056] For each task, it uses the current Quickly initialize a task-specific strategy And adapt a small number of steps on the simulation environment or historical data of the task.

[0057] Multi-objective cumulative rewards based on the adapted strategy in their respective tasks ,calculate The gradient is calculated and updated.

[0058] Optimized It is used to initialize or fine-tune lightweight decision models distributed to the edge.

[0059] Example 4: Complete Workflow in an Online Mathematics Learning Scenario refer to Figure 2 Take, for example, a junior high school student learning the "quadratic equation formula" on a tablet computer.

[0060] Step S201: Data Acquisition and Local Processing. The student learns from the instructional video. The tablet's camera captures a slight frown and frequent blinking (visual); the microphone does not record speech; the heart rate wristband shows a slight decrease in heart rate variability (physiological); the platform records that the student paused the video three times (behavioral). The tablet's local program uses the OpenFace library to extract 17-dimensional facial motion unit features, adds Laplacian noise (LDP) to the original image frames; extracts frequency domain features from the physiological signals; and encodes the behavior into vectors. All feature vectors are generated locally.

[0061] Step S202: Data transmission. The encrypted feature vector is transmitted to the edge server in the classroom within 15ms via a Wi-Fi 6 network.

[0062] Step S203: Edge Fusion and State Calculation. The edge server runs a dynamic attention network. Since the current stage is video viewing with no audio, the network automatically assigns higher weights to visual and physiological modalities. The fused state... The diagnosis was "moderate confusion with mild cognitive fatigue".

[0063] Step S204: Real-time path decision. The lightweight DQN model of the edge server receives the state. The model then queries the knowledge graph and finds that the current knowledge point, "derivation of the quadratic formula," is a key and error-prone point. The model's decision output action is: "Push a 2-minute animation explaining the quadratic formula from a geometric perspective, and insert a low-difficulty coefficient identification exercise after the animation."

[0064] Step S205: Suggestion Implementation and Feedback. The suggestion is displayed in the sidebar of the student's tablet within 156ms of its generation. The student clicks to watch the animation, then completes the exercises and answers correctly.

[0065] Step S206: Local Model Update and Cloud Synchronization (Backend). The edge server records the "state-action-reward" trajectory of this success for periodic experience replay updates of the local DQN model. Every 5 minutes, the edge server uploads encrypted incremental model parameters to the cloud. Every 30 minutes, the cloud aggregates updates from a large number of edge nodes nationwide, trains the global model, and optimizes the strategy through meta-reinforcement learning to better handle the common scenario of "confusion while watching videos." The optimized model parameters are then periodically distributed to the edge server, completing the closed loop.

[0066] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0067] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by program instructions and related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, a magnetic disk, or an optical disk, etc.

Claims

1. A method for real-time adaptive adjustment of learning paths based on multimodal behavior perception, characterized in that, Includes the following steps: S1: Collect learners' multimodal raw data in real time through terminal devices, and perform preprocessing, feature extraction and privacy processing locally on the terminal devices to obtain desensitized feature vectors for each modality; the multimodal raw data includes visual data, speech data, physiological signal data and learning interaction behavior data; S2: The desensitized feature vector is transmitted to the edge computing node; the edge computing node uses a dynamic attention multimodal fusion network to perform weighted fusion on the desensitized feature vector to generate a comprehensive state representation that reflects the learner's current cognitive and emotional state; the dynamic attention multimodal fusion network dynamically generates normalized attention weights based on the real-time quality and contextual relevance of each modality feature. S3: The edge computing node, based on the comprehensive state representation and the pre-set local knowledge graph, performs real-time reasoning through a lightweight meta-reinforcement learning decision model, generates immediate path adjustment suggestions, and feeds them back to the terminal device for execution. S4: The cloud server securely aggregates encrypted model parameter updates from multiple edge computing nodes through a federated learning framework to train and update the global multimodal fusion model and the global policy model; at the same time, the cloud server uses a meta-reinforcement learning framework to iteratively optimize the global learning path adjustment strategy with a multi-objective reward function that integrates learning efficiency, knowledge mastery, user satisfaction and cognitive load as the optimization objective. S5: The cloud server encrypts the optimized global model parameters and sends them to each edge computing node; the edge computing node uses the sent parameters to update its local model, forming a closed-loop optimization system of "edge real-time decision-making - cloud global evolution - parameter secure synchronization".

2. The method according to claim 1, characterized in that, In step S1, the privacy processing specifically involves: applying a local differential privacy mechanism to the biometric raw data containing face images and raw audio, and perturbing it by adding Laplacian noise or Gaussian noise that satisfies ε-differential privacy to generate a desensitized modal feature vector. The learning interaction behavior log data is generalized and k-anonymized.

3. The method according to claim 1, characterized in that, The dynamic attention multimodal fusion network mentioned in step S2 includes: The feature encoding branch corresponding to each modality is used to map the input desensitized feature vector to a high-dimensional representation space; The dynamic attention weight calculation module takes the output of each feature encoding branch as input and passes it through a fully connected layer and... A small neural network with activation functions is used to calculate a set of attention weights related to the current input. ,in Represents a modal index, and ; Feature fusion layer, used to perform weighted summation operation: ,in For the first Feature encoding output for each modality This is the final comprehensive state representation.

4. The method according to claim 3, characterized in that, The input to the dynamic attention weight calculation module also includes a context state vector, which is encoded by one or more pieces of information such as learning stage, activity type, and historical state sequence, and is used to guide attention weights to tilt toward modalities that are more relevant to the current learning context.

5. The method according to claim 1, characterized in that, The lightweight meta-reinforcement learning decision model described in step S3 is obtained by compressing a standard deep Q-network or policy gradient network. The compression methods include: knowledge distillation, using a cloud-trained teacher network to guide the training of the marginal student network; parameter quantization, converting network weights and activation values ​​from 32-bit floating-point numbers to 8-bit integers; and network pruning, removing connections or entire neuron channels whose absolute weight values ​​are below a threshold.

6. The method according to claim 1, characterized in that, The federated learning framework described in step S4 adopts a performance-adaptive federated averaging algorithm. When aggregating parameters, it dynamically calculates and allocates aggregation weights based on the amount of local data reported by each edge computing node, the model update quality index, and the node availability reliability, so as to improve the robustness and convergence efficiency of the global model.

7. The method according to claim 1, characterized in that, The multi-objective reward function described in step S4 Represented as: ;in, Calculated based on the number of knowledge points mastered per unit of time. Accuracy and consolidation are calculated based on practice and testing. Calculation based on positive interaction events and negative feedback Calculation of cognitive load and behavioral retardation index based on physiological signal estimation; weighting parameters It can be preset based on the experience of education experts or dynamically adjusted through online Bayesian optimization.

8. The method according to claim 1, characterized in that, In step S5, the parameter transmission between the cloud server and the edge computing node uses a binary encoding format based on Google Protocol Buffers to compress the parameter tensor, and transmits it through a 5G network's ultra-reliable low-latency communication slice or a dedicated time-sensitive network channel. During the transmission process, the parameters are encrypted using a homomorphic encryption algorithm based on RLWE.

9. The method according to claim 1, characterized in that, The instant path adjustment suggestions generated in step S3 include, in specific ways, recommending learning resources in a specific format, adjusting the difficulty level of subsequent learning tasks, suggesting changes in the learning pace, planning review intervals, and triggering prompts for peer support or teacher intervention.

10. A real-time adaptive adjustment system for a learning path based on multimodal behavior perception for implementing the method of any one of claims 1-9, characterized in that, The architecture adopts a three-tier collaborative structure of endpoint-edge-cloud, including: The terminal layer consists of a learning terminal that integrates a multimodal data acquisition sensor and a local processing unit, and is used to execute step S1. The edge layer, consisting of network edge servers or smart gateways deployed close to the terminal layer, incorporates the dynamic attention multimodal fusion network, the lightweight meta-reinforcement learning decision model, and a local knowledge graph database, and is used to execute steps S2 and S3. The cloud layer consists of a high-performance computing cluster deployed in a data center, which includes a federated learning aggregator and a meta-reinforcement learning optimizer, and is used to execute the cloud functions in steps S4 and S5. The terminal layer, edge layer, and cloud layer are connected through wired and wireless communication networks, forming a closed-loop system in which data, control, and parameter flows work together.

Images