Localized deployment intelligent interactive classroom system and method based on edge intelligence architecture

Abstract

This invention discloses a locally deployed intelligent interactive classroom system and method based on an edge intelligence architecture, belonging to the field of digital intelligent teaching technology. It deploys a lightweight multimodal fusion model at the edge, accurately identifying students' cognitive states in milliseconds, providing a basis for personalized intervention. Employing a physically closed architecture, federated learning, differential privacy, and TEE hardware-level protection, it meets data compliance requirements and eliminates the risk of data leakage. Utilizing deep reinforcement learning, it dynamically optimizes learning paths based on real-time status and historical data, achieving personalized instruction and improving learning effectiveness and experience. Leveraging the WebAssembly sandbox, it allows third-party developers to access extended applications while ensuring data security, contributing to the construction of a smart classroom ecosystem.

Description

Technical Field

[0001] This invention belongs to the field of digital and intelligent teaching technology, specifically relating to a locally deployed intelligent interactive classroom system and method based on edge intelligence architecture. Background Technology

[0002] The field of educational informatization is currently undergoing a profound transformation from "digitalization" to "intelligentization," with the core objective of using artificial intelligence technology to achieve personalized teaching and provide highly accurate individualized feedback. However, achieving this goal faces the severe challenge of increasingly stringent data privacy protection regulations. For example, China's Personal Information Protection Law explicitly requires that sensitive data, such as educational data, be stored and processed locally.

[0003] Traditional smart classroom systems generally adopt an "edge-cloud" architecture, transmitting all raw video and audio data collected in the classroom, including multimodal data containing biometrics and personal identification information, to the cloud for centralized processing and analysis. This architecture has two inherent and irreconcilable flaws: First, transmitting raw data to the cloud violates legal requirements for local data storage, and centralized data storage, once attacked, will lead to large-scale and irreversible privacy leaks; second, the cloud processing model, which relies on network transmission, is limited by network latency and bandwidth bottlenecks, making it difficult to achieve real-time and accurate analysis of students' classroom learning status. The feedback is often delayed and cannot be used to guide the current teaching process.

[0004] To alleviate these problems, some exploratory edge computing solutions have emerged in the industry, attempting to offload some computing tasks to gateways or servers within the classroom. However, these solutions generally suffer from the following technical bottlenecks: First, most can only process single-modal data (e.g., analyzing student head-up rate solely through cameras, or calculating accuracy solely through quiz machines), failing to integrate multi-dimensional information such as video micro-expressions, speech semantics, and answering behavior. Therefore, they struggle to accurately assess students' complex and implicit cognitive states, such as "confusion," "understanding," or "boredom." Second, there is a lack of systematic privacy protection design in model optimization and deployment. Gradient uploads during local model fine-tuning may still leak individual information; during inference, data in memory is also at risk of being stolen by malware. Third, adjustments to teaching strategies are usually based on pre-set simple rules (e.g., pushing easy questions after three consecutive incorrect answers), rather than in-depth mining and dynamic modeling of real-time learning data, resulting in very limited personalized teaching effectiveness. Furthermore, most existing systems are closed, making it difficult to securely introduce innovative teaching applications from third-party developers, creating an ecosystem barrier for functional iteration. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this application provides a locally deployed intelligent interactive classroom system and method based on an edge intelligence architecture.

[0006] Firstly, this application proposes a locally deployed intelligent interactive classroom system based on an edge intelligence architecture, comprising:

[0007] The multimodal data acquisition and preprocessing module is deployed in the local physical space of the classroom. It is used to collect students' video, audio and answer behavior data in real time through heterogeneous sensor networks, and to clean, extract features and anonymize the data locally to ensure that the original sensitive data does not leave the classroom.

[0008] A lightweight multimodal edge AI inference engine is deployed on an edge server in the classroom to receive preprocessed multimodal features. Through a spatiotemporal separation attention mechanism and a gated cross-attention fusion architecture, it analyzes and outputs students' fine-grained cognitive state tensors in real time.

[0009] A two-layer privacy protection mechanism, including a federated learning differential privacy gradient perturbation module during the model training phase and an encrypted computing module based on a trusted execution environment during the model inference phase, is used to build end-to-end data privacy protection from data collection to model application.

[0010] The dynamic personalized learning path generation module, based on a knowledge tracking model enhanced by deep reinforcement learning networks and attention mechanisms, dynamically adjusts the difficulty of teaching content and the distribution of knowledge points according to students' real-time cognitive state and historical learning sequence, generating personalized learning paths.

[0011] An open plugin sandbox architecture is used to securely connect to third-party teaching applications in the form of memory-isolated WebAssembly modules, and to anonymize and audit access to sensitive data through a secure interface gateway to ensure that core data is not illegally obtained.

[0012] In some embodiments, the multimodal data acquisition and preprocessing module includes:

[0013] The video acquisition unit consists of four 1080P wide-angle cameras supporting H.265 hardware encoding deployed at the four corners of the classroom. It transmits data through an independent video virtual local area network and adopts a dynamic frame sampling strategy to adaptively adjust the frequency of face detection and key frame extraction based on the range of student head movements.

[0014] The audio acquisition unit consists of a 6-array digital microphone. It uses beamforming technology to locate and separate the sound source, acquires the original waveform with 48kHz / 24bit precision, and performs voiceprint separation and speech enhancement processing.

[0015] The answer terminal unit is equipped with a touch screen and a pressure-sensitive pen to record students' answers, response time, handwriting trajectory and pressure changes, and standardizes them into a structured data format that includes timestamps.

[0016] The preprocessing unit is used to perform face detection and background blurring on the video stream, speech activity detection and feature extraction on the audio stream, and format unification and normalization on the interactive data.

[0017] In some embodiments, the lightweight multimodal edge AI inference engine includes:

[0018] The video feature extraction branch adopts the lightweight MobileViT model optimized for edge devices. This model integrates a spatiotemporal separation attention mechanism, decomposes the 3D convolution into spatial self-attention and temporal self-attention, and combines a dynamic frame sampling strategy to extract spatiotemporal features such as students' micro-expressions, head posture, and gaze direction.

[0019] The audio feature extraction branch adopts the knowledge-distilled TinyBERT model. The input of this model is the multi-scale spliced ​​Mel frequency cepstral coefficient features, and the effective speech units are accurately segmented by the speech activity detection algorithm based on conditional random fields to understand semantic content and tone features.

[0020] The text behavior analysis branch is used to analyze the interactive data of the answer terminal, including normalizing the response time through statistical methods and encoding the pressure fluctuation pattern during the writing process through Gram matrix to reflect the student's thinking state.

[0021] The cross-modal fusion module employs a gated cross-attention mechanism, which dynamically calculates the importance weights of video, audio, and text modalities in the current context using learnable gating values, and then fuses the weighted multimodal features to generate a comprehensive feature vector for cognitive state discrimination.

[0022] In some embodiments, the federated learning differential privacy gradient perturbation module in the two-layer privacy protection mechanism includes: a differential privacy gradient pruning unit, used to perform L2 norm pruning on the model gradient calculated by the local client, scaling gradients exceeding a preset threshold to within the threshold, so as to limit the contribution of a single sample to the model update;

[0023] An adaptive noise injection unit is used to dynamically adjust the scale of Laplacian or Gaussian noise added to the clipped gradient according to the current training epoch. Larger noise is used in the early stage of training to enhance privacy protection, and the noise decays exponentially with each epoch, while a minimum noise threshold is set to maintain basic protection.

[0024] The secure aggregation unit employs a double-mask protocol, which enables each client to obfuscate the gradient with a pair of random vectors that sum to zero before uploading. After aggregating all encrypted gradients, the server can restore the true gradient sum without being able to snoop on the original gradient information of a single client.

[0025] In some embodiments, the trusted execution environment encrypted computing module in the two-layer privacy protection mechanism includes:

[0026] The encrypted model loading unit is used to convert the pre-trained model parameters into a format compatible with the trusted execution environment and encrypts them using sealed storage technology to ensure that the model exists in encrypted form on both the hard drive and in memory, and is decrypted only when the computation is performed within the CPU enclave.

[0027] The secure computing unit is used to receive input data transmitted through a secure channel within the enclave, and to obfuscate and protect the intermediate feature map tensors generated during the forward propagation of the model. By generating a random mask matrix and performing element-wise operations with the feature map, it prevents attackers from reconstructing the real information through memory sniffing.

[0028] The hardware key derivation unit uses a unique key derived from the trusted execution environment hardware, combined with national cryptographic algorithms such as SM4, to encrypt and decrypt data within the enclave, ensuring that the key is not exposed.

[0029] In some embodiments, the dynamic personalized learning path generation module includes:

[0030] The multimodal state encoder adopts an improved cross-modal Transformer structure, which maps video, audio and text features to the same space through linear transformation and then performs layer normalization. It also calculates cross-modal correlation through a multi-head attention mechanism to enhance the expression of key cognitive state information and generate the student state code at the current moment.

[0031] The deep reinforcement learning strategy network is constructed based on the PPO-Clip algorithm. It takes the state code as input, the question difficulty adjustment amount and the knowledge point weight vector as the output action space, and the reward function is an index that comprehensively considers the answer accuracy, learning efficiency and cognitive load. The teaching strategy is stably optimized through the pruning mechanism.

[0032] The LSTM-Attention knowledge tracking and prediction model constructs a bidirectional LSTM network to process students' learning history sequences, calculates the importance weight of each historical moment to the current prediction through a temporal attention mechanism, and finally predicts the probability of students mastering future knowledge points through a multilayer perceptron, which can be used to push pre-study resources in advance or adjust the teaching pace.

[0033] In some embodiments, the LSTM-Attention knowledge tracking prediction model further includes:

[0034] A bidirectional LSTM unit is used to process the student's historical learning sequence from both forward and backward directions, and to concatenate the hidden states from the two directions to capture the complete contextual dependencies in the sequence.

[0035] The temporal attention weight calculation unit applies a linear transformation and a sigmoid activation function to the concatenated hidden state at each time step to generate normalized attention weights, which are used to measure the impact of the learning event at that time step on the prediction of future mastery.

[0036] The weighted summation and prediction unit sums the hidden states at each time step according to their corresponding attention weights, inputs the resulting context vector into the multilayer perceptron, and finally outputs the probability that the student has mastered a specific knowledge point at a future time point.

[0037] In some embodiments, the open plug-in sandbox architecture includes:

[0038] The module encapsulation and static verification unit is used to receive third-party plugins submitted in WebAssembly binary format. Before loading, it performs type checking, memory boundary checking and control flow integrity verification. Modules that pass the verification are granted a unique and unforgeable capability token.

[0039] The security interface gateway enables controlled data access based on the CPI protocol, including a de-identification proxy for real-time face blurring of the raw video stream, a query rewriting engine that rewrites plugin requests into standardized SQL templates, and adds Laplace noise to statistical query results to meet differential privacy requirements.

[0040] The resource isolation execution unit creates an independent lightweight container for each verified plugin instance, restricts its access to only temporary directories through file system virtualization, restricts its outbound connections through network whitelisting, and precisely controls its CPU and memory resource usage through cgroups technology.

[0041] The behavior auditing and circuit breaker unit monitors the system call frequency and memory access patterns of plugins in real time. When abnormal behavior is detected, it automatically triggers dynamic permission downgrade and revokes the plugin's access permissions to sensitive data.

[0042] In some embodiments, the student cognitive state tensor output by the lightweight multimodal edge AI inference engine includes at least four fine-grained states: focus, confusion, understanding, and fatigue, and the end-to-end inference latency is controlled to within 100 milliseconds.

[0043] Secondly, this application proposes a method for locally deploying intelligent interactive classrooms based on an edge intelligence architecture, including the following steps:

[0044] Locally in the classroom, multimodal raw data is collected through a heterogeneous sensor network, and then preprocessed by cleaning, feature extraction, and identity anonymization on an edge server.

[0045] The preprocessed multimodal features are input into a lightweight edge AI inference engine, which analyzes and outputs the student's fine-grained cognitive state in real time through a spatiotemporal separation attention and gating cross-attention fusion mechanism.

[0046] During the model training phase, the local gradients are differentially privacy-preserving and adaptively noise-perturbed using a federated learning framework, and secure aggregation is performed using a double-mask protocol. During the model inference phase, the model and data are loaded into a trusted execution environment, and all computation processes are encrypted and obfuscated for protection.

[0047] The module generates a personalized learning path based on deep reinforcement learning and attention knowledge tracking by inputting students’ real-time cognitive state and historical learning sequence into the module, and dynamically outputs the next best teaching action, including adjusting the difficulty of questions or pushing specific learning resources.

[0048] Secure access to third-party plugins is achieved through a memory-isolated WebAssembly sandbox, with all data access requests being anonymized, audited, and controlled via a secure interface gateway.

[0049] Thirdly, this application proposes an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0050] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0051] The beneficial effects of this invention are:

[0052] By deploying a lightweight multimodal fusion model at the edge, millisecond-level accurate identification of students' cognitive states is achieved, providing a reliable basis for personalized intervention. Combined with a physically closed architecture that prevents raw data from leaving the classroom, a federated learning and differential privacy model training mechanism, and TEE hardware-level inference protection, the compliance requirements for educational data are fully met, completely eliminating the risk of data leakage. Abandoning static teaching rules, deep reinforcement learning technology dynamically generates and continuously optimizes learning paths based on each student's real-time learning status and historical data, truly achieving individualized instruction and significantly improving learning outcomes and experience. Through the WebAssembly sandbox architecture, third-party developers are allowed to securely access and extend teaching applications while ensuring core data security, providing unlimited possibilities for future smart classroom functional iterations and ecosystem development. Attached Figure Description

[0053] Figure 1 This is a system structure block diagram of the present invention.

[0054] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0055] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein; rather, these embodiments are provided so that a more thorough understanding of the invention can be achieved and that the full scope of the invention can be conveyed to those skilled in the art.

[0056] Firstly, this application proposes a locally deployed intelligent interactive classroom system based on an edge intelligence architecture, such as... Figure 1 As shown, it includes:

[0057] The multimodal data acquisition and preprocessing module is deployed in the local physical space of the classroom. It is used to collect students' video, audio and answer behavior data in real time through heterogeneous sensor networks, and to clean, extract features and anonymize the data locally to ensure that the original sensitive data does not leave the classroom.

[0058] In some embodiments, the multimodal data acquisition and preprocessing module includes:

[0059] The video acquisition unit consists of four 1080P wide-angle cameras supporting H.265 hardware encoding deployed at the four corners of the classroom. It transmits data through an independent video virtual local area network and adopts a dynamic frame sampling strategy to adaptively adjust the frequency of face detection and key frame extraction based on the range of student head movements.

[0060] The audio acquisition unit consists of a 6-array digital microphone. It uses beamforming technology to locate and separate the sound source, acquires the original waveform with 48kHz / 24bit precision, and performs voiceprint separation and speech enhancement processing.

[0061] The answer terminal unit is equipped with a touch screen and a pressure-sensitive pen to record students' answers, response time, handwriting trajectory and pressure changes, and standardizes them into a structured data format that includes timestamps.

[0062] The preprocessing unit is used to perform face detection and background blurring on the video stream, speech activity detection and feature extraction on the audio stream, and format unification and normalization on the interactive data.

[0063] Specifically, the system uses a camera array, omnidirectional microphones, and smart answer terminals deployed in the classroom to collect real-time video streams of students' faces, voice interaction data between teachers and students, and answer texts. All raw data is transmitted to the edge server through the local network to ensure that the data does not leave the physical classroom environment.

[0064] The localized acquisition and preprocessing system for multimodal data adopts a distributed sensor network architecture, deploying multiple high-precision data acquisition nodes within the classroom. Specific implementation includes:

[0065] (1) The video acquisition node consists of four 1080P@60fps wide-angle cameras, forming a stereoscopic visual coverage network with the four corners of the classroom as the vertices. Each camera has a built-in H.265 hardware encoder to compress the original video stream in real time into HEVC format with a resolution of 2560×1440 and a bit rate of 8Mbps.

[0066] (2) The audio acquisition uses an acoustic positioning system composed of 6 array digital microphones to acquire the original waveform at a sampling rate of 48kHz and a quantization accuracy of 24bit. The independent audio track of each sound source is separated by beamforming technology.

[0067] (3) The intelligent answering terminal is equipped with a capacitive touch screen and a pressure-sensitive pen to record interactive metadata such as the student's handwriting trajectory, pressure changes, and response delay. The data format is a JSON sequence with timestamps. All acquisition devices are connected to the edge computing gateway via gigabit Ethernet and are divided into three independent virtual local area networks using the IEEE 802.1Q VLAN protocol: video VLAN (priority 6), audio VLAN (priority 5), and interactive data VLAN (priority 4) to ensure the transmission quality of critical teaching data. The edge server receiver is equipped with a streaming media processing pipeline to perform face detection (based on the YOLOv5s model), keyframe extraction (retaining 1 I-frame every 0.5 seconds), and background blurring on the video stream; perform voiceprint separation (using the open-source toolkit Kaldi) and speech enhancement (based on the RNNoise algorithm) on the audio stream; and perform standardized conversion on the interactive data, mapping the data from different terminals to a unified format of [student ID, question ID, response time, confidence level] quadruple.

[0068] A lightweight multimodal edge AI inference engine is deployed on an edge server in the classroom to receive preprocessed multimodal features. Through a spatiotemporal separation attention mechanism and a gated cross-attention fusion architecture, it analyzes and outputs students' fine-grained cognitive state tensors in real time.

[0069] In some embodiments, the lightweight multimodal edge AI inference engine includes:

[0070] The video feature extraction branch adopts the lightweight MobileViT model optimized for edge devices. This model integrates a spatiotemporal separation attention mechanism, decomposes the 3D convolution into spatial self-attention and temporal self-attention, and combines a dynamic frame sampling strategy to extract spatiotemporal features such as students' micro-expressions, head posture, and gaze direction.

[0071] The audio feature extraction branch adopts the knowledge-distilled TinyBERT model. The input of this model is the multi-scale spliced ​​Mel frequency cepstral coefficient features, and the effective speech units are accurately segmented by the speech activity detection algorithm based on conditional random fields to understand semantic content and tone features.

[0072] The text behavior analysis branch is used to analyze the interactive data of the answer terminal, including normalizing the response time through statistical methods and encoding the pressure fluctuation pattern during the writing process through Gram matrix to reflect the student's thinking state.

[0073] The cross-modal fusion module employs a gated cross-attention mechanism, which dynamically calculates the importance weights of video, audio, and text modalities in the current context using learnable gating values, and then fuses the weighted multimodal features to generate a comprehensive feature vector for cognitive state discrimination.

[0074] The MobileViT-TinyBERT fusion architecture is used to process video, audio and text data: the video branch analyzes micro-expression and head pose features through MobileViT, the audio branch uses TinyBERT for semantic understanding, the text branch analyzes answering behavior, and dynamically fuses features from various modalities through a cross-modal attention mechanism to output a learned state tensor.

[0075] The lightweight multimodal edge AI inference engine employs an improved MobileViT-TinyBERT hybrid architecture to achieve multimodal feature extraction and fusion, specifically including:

[0076] 1. Spatiotemporal Separation Attention Mechanism: This mechanism decomposes the traditional 3D convolution into two independent parts: spatial attention and temporal attention. First, spatial self-attention is calculated within each frame to capture the spatial structural features within the image; then, temporal self-attention is calculated on the frame sequence to capture the changes in actions and expressions over time; finally, the two are fused to obtain a spatiotemporal feature representation that simultaneously contains spatial details and dynamic information.

[0077] Dynamic frame sampling strategy: The system adaptively adjusts the processing frame rate based on the intensity of the student's head movements. The system detects the change in Euler angles of the head between consecutive frames in real time. When the amplitude of the movement increases, the frame rate is automatically increased to capture more subtle behavioral changes; when the movement is gentle, the frame rate is reduced to save computing resources, but it is always kept between the preset minimum and maximum frame rates.

[0078] 2. The audio semantic understanding module is implemented based on the distilled TinyBERT model:

[0079] Speech Feature Pyramid: MFCC features extracted at different time granularities (such as 25ms, 50ms, 100ms) are concatenated to form multi-scale Mel-frequency cepstral coefficient features in order to capture multi-scale temporal information in speech.

[0080] Context-aware speech segmentation: A speech activity detection algorithm based on Conditional Random Field (CRF) is adopted. This algorithm uses a transition feature function to characterize the dependency relationship between adjacent speech segment labels, and a state feature function to characterize the relationship between the current segment features and the label. The two are weighted by a learnable weight parameter, and finally the most likely speech segment boundary is calculated, thereby accurately segmenting effective speech units.

[0081] 3. Key steps in text behavior analysis and processing of data from answer terminals:

[0082] Response time normalization: Considering the impact of question difficulty on response time, the original response time is subtracted from the mean of historical response times for questions of the same difficulty, then divided by the standard deviation, and then scaled and translated to obtain a standardized response time feature, thereby eliminating the interference caused by differences in question difficulty.

[0083] Handwriting pressure feature encoding: Gram matrix is ​​used to capture the dynamic changes in pressure values ​​during the writing process. This matrix reflects the pattern of pressure fluctuation over time by calculating the correlation between pressure values ​​at different times, such as hesitation and fluency during writing, thereby helping to determine the student's cognitive state.

[0084] 4. Cross-modal fusion employs a gated cross-attention mechanism:

[0085] Intermodal importance weight calculation: For each pair of modalities, their feature vectors are concatenated and input into a learnable linear layer. Then, a gating value between 0 and 1 is generated using the sigmoid function. This gating value dynamically measures the importance of one modality to another in the current context. For example, when a student answers a question, the weights of the audio and text modalities will be relatively increased.

[0086] Weighted fusion: Using features from one modality as the query and features from another modality as the key and value, the association features between the two are calculated through an attention mechanism, multiplied by the aforementioned gating value, and finally the weighted results of all modality pairs are summed to obtain a comprehensive feature vector that integrates information from each modality and has dynamic weights, which is used for subsequent cognitive state judgment.

[0087] A two-layer privacy protection mechanism, including a federated learning differential privacy gradient perturbation module during the model training phase and an encrypted computing module based on a trusted execution environment during the model inference phase, is used to build end-to-end data privacy protection from data collection to model application.

[0088] In some embodiments, the federated learning differential privacy gradient perturbation module in the two-layer privacy protection mechanism includes: a differential privacy gradient pruning unit, used to perform L2 norm pruning on the model gradient calculated by the local client, scaling gradients exceeding a preset threshold to within the threshold, so as to limit the contribution of a single sample to the model update;

[0089] An adaptive noise injection unit is used to dynamically adjust the scale of Laplacian or Gaussian noise added to the clipped gradient according to the current training epoch. Larger noise is used in the early stage of training to enhance privacy protection, and the noise decays exponentially with each epoch, while a minimum noise threshold is set to maintain basic protection.

[0090] The secure aggregation unit employs a double-mask protocol, which enables each client to obfuscate the gradient with a pair of random vectors that sum to zero before uploading. After aggregating all encrypted gradients, the server can restore the true gradient sum without being able to snoop on the original gradient information of a single client.

[0091] During the local federated learning fine-tuning phase, the edge server adds noise that satisfies (ε,δ)-differential privacy to the data before uploading the gradient; during the inference phase, all AI computations are run in a TEE (such as Intel SGX or Chinese national standard SM9), and the model parameters and intermediate data remain encrypted to prevent data leakage.

[0092] Local Federated Learning Fine-Tuning Phase: An improved gradient perturbation federated learning algorithm is used to achieve privacy protection. Specific steps include:

[0093] 1.1 Differential Privacy Gradient Pruning: The gradients calculated for each client are pruned to limit their L2 norm to no more than a preset threshold. If the gradient norm exceeds this threshold, it is scaled down proportionally to within the threshold, thereby controlling the impact of individual samples on model updates and preventing the gradient from containing too much individual information.

[0094] 1.2 Adaptive Laplacian Noise Injection: The scale of added noise is dynamically adjusted according to the current training epoch. Larger noise is used in the early stages of training to provide stronger privacy protection and help explore the parameter space; as the training epochs increase, the noise scale decays exponentially, enabling the model to converge more finely, while ensuring that basic privacy protection is not lost through a minimum noise threshold.

[0095] 1.3 Secure Aggregation Protocol: A double-mask mechanism is used to obfuscate gradients. Each client generates two random vectors, adds the first random vector to the clipped gradient, subtracts the second random vector, and then uploads it to the server. All clients' random vectors satisfy the condition that their sum is zero. Therefore, after aggregating all encrypted gradients, the server can recover the true gradient sum without knowing the original gradient of any individual client, thus achieving secure aggregation.

[0096] In some embodiments, the trusted execution environment encrypted computing module in the two-layer privacy protection mechanism includes:

[0097] The encrypted model loading unit is used to convert the pre-trained model parameters into a format compatible with the trusted execution environment and encrypts them using sealed storage technology to ensure that the model exists in encrypted form on both the hard drive and in memory, and is decrypted only when the computation is performed within the CPU enclave.

[0098] The secure computing unit is used to receive input data transmitted through a secure channel within the enclave, and to obfuscate and protect the intermediate feature map tensors generated during the forward propagation of the model. By generating a random mask matrix and performing element-wise operations with the feature map, it prevents attackers from reconstructing the real information through memory sniffing.

[0099] The hardware key derivation unit uses a unique key derived from the trusted execution environment hardware, combined with national cryptographic algorithms such as SM4, to encrypt and decrypt data within the enclave, ensuring that the key is not exposed.

[0100] Trusted Execution Environment Inference Phase: 2.1 Encrypted Model Loading: Convert the pre-trained model to a TEE-compatible format, encrypt the model parameters using SGX Sealed Storage technology, and keep them in memory in an encrypted state until decryption during computation.

[0101] 2.2 Secure Computation Process: Input data is transmitted directly to the enclave via a secure channel. Within the enclave, the data in memory is encrypted using the national standard SM4 algorithm. The encryption process employs a hardware-derived key, combined with an initialization vector and bitwise XOR operations, to ensure data confidentiality in memory and prevent physical attacks or memory sniffing.

[0102] 2.3 Intermediate Data Protection: All feature map tensors generated during the computation process are obfuscated. The system generates a random mask matrix of the same size as the feature map and multiplies it element-wise with the feature map to obtain the obfuscated tensor. This ensures that even if an attacker obtains the memory data, they cannot reconstruct the true feature information, further enhancing the security of the inference process.

[0103] The dynamic personalized learning path generation module, based on a knowledge tracking model enhanced by deep reinforcement learning networks and attention mechanisms, dynamically adjusts the difficulty of teaching content and the distribution of knowledge points according to students' real-time cognitive state and historical learning sequence, generating personalized learning paths.

[0104] In some embodiments, the dynamic personalized learning path generation module includes:

[0105] The multimodal state encoder adopts an improved cross-modal Transformer structure, which maps video, audio and text features to the same space through linear transformation and then performs layer normalization. It also calculates cross-modal correlation through a multi-head attention mechanism to enhance the expression of key cognitive state information and generate the student state code at the current moment.

[0106] The deep reinforcement learning strategy network is constructed based on the PPO-Clip algorithm. It takes the state code as input, the question difficulty adjustment amount and the knowledge point weight vector as the output action space, and the reward function is an index that comprehensively considers the answer accuracy, learning efficiency and cognitive load. The teaching strategy is stably optimized through the pruning mechanism.

[0107] Based on real-time learning status data, the deep reinforcement learning strategy network dynamically adjusts the difficulty of questions and the distribution of knowledge points, and the LSTM-Attention prediction model analyzes historical learning sequences and predicts future knowledge mastery trends, pushing out pre-study resources in advance.

[0108] The dynamic personalized learning path generation scheme uses a hybrid neural network architecture to achieve real-time optimization of teaching strategies. The specific implementation steps are as follows:

[0109] 1. Multimodal State Encoder: Employs an improved cross-modal Transformer structure to fuse real-time learned features. First, visual, audio, and textual features are mapped to the same feature space through linear transformations, then summed and normalized layer by layer to obtain a preliminary fused state representation. Simultaneously, a multi-head attention mechanism is used to calculate cross-modal correlations: for each modality pair, features from one modality are used as the query and features from the other modality as the key; the similarity between the two is calculated, and then normalized using Softmax to obtain attention weights. These weights reflect the strength of the association between features from different modalities, thereby enhancing the expression of key information.

[0110] 2. Deep Reinforcement Learning Strategy Network: An instructional strategy optimizer based on the PPO-Clip algorithm is designed. The state space represents the encoded learning features, and the action space includes adjustments to question difficulty and knowledge point weight vectors. The reward function is defined as an indicator that comprehensively considers student accuracy, learning efficiency, and cognitive load to encourage efficient instructional actions suitable for the student's current state. Strategy updates employ a pruning mechanism: the probability ratio of the new and old strategies is calculated, this ratio is pruned, multiplied by the dominance function, and finally the expected value is taken. This optimizes the instructional strategy while ensuring stable updates, avoiding learning instability caused by excessive updates.

[0111] The LSTM-Attention knowledge tracking and prediction model constructs a bidirectional LSTM network to process students' learning history sequences, calculates the importance weight of each historical moment to the current prediction through a temporal attention mechanism, and finally predicts the probability of students mastering future knowledge points through a multilayer perceptron, which can be used to push pre-study resources in advance or adjust the teaching pace.

[0112] In some embodiments, the LSTM-Attention knowledge tracking prediction model further includes:

[0113] A bidirectional LSTM unit is used to process the student's historical learning sequence from both forward and backward directions, and to concatenate the hidden states from the two directions to capture the complete contextual dependencies in the sequence.

[0114] The temporal attention weight calculation unit applies a linear transformation and a sigmoid activation function to the concatenated hidden state at each time step to generate normalized attention weights, which are used to measure the impact of the learning event at that time step on the prediction of future mastery.

[0115] The weighted summation and prediction unit sums the hidden states at each time step according to their corresponding attention weights, inputs the resulting context vector into the multilayer perceptron, and finally outputs the probability that the student has mastered a specific knowledge point at a future time point.

[0116] LSTM-Attention Prediction Model: Constructing a bidirectional LSTM network to predict knowledge mastery trends. The model processes the student's learning history sequence using both forward and backward LSTMs, obtaining the forward and backward hidden states at each time step. These are then concatenated to obtain the complete hidden state representation. A temporal attention mechanism is then used to calculate attention weights for each hidden state at each time step through a linear transformation and a sigmoid function. These weights measure the importance of each historical time step to the current prediction. Finally, the hidden states at each time step are weighted and summed according to their attention weights, and the sum is input into a multilayer perceptron (MLP) to predict the probability of mastering future knowledge points. For example, it can predict a student's mastery of a specific knowledge point at a future time, thereby providing advance learning resources or adjusting the teaching pace.

[0117] An open plugin sandbox architecture is used to securely connect to third-party teaching applications in the form of memory-isolated WebAssembly modules, and to anonymize and audit access to sensitive data through a secure interface gateway to ensure that core data is not illegally obtained.

[0118] In some embodiments, the open plug-in sandbox architecture includes:

[0119] The module encapsulation and static verification unit is used to receive third-party plugins submitted in WebAssembly binary format. Before loading, it performs type checking, memory boundary checking and control flow integrity verification. Modules that pass the verification are granted a unique and unforgeable capability token.

[0120] The security interface gateway enables controlled data access based on the CPI protocol, including a de-identification proxy for real-time face blurring of the raw video stream, a query rewriting engine that rewrites plugin requests into standardized SQL templates, and adds Laplace noise to statistical query results to meet differential privacy requirements.

[0121] The resource isolation execution unit creates an independent lightweight container for each verified plugin instance, restricts its access to only temporary directories through file system virtualization, restricts its outbound connections through network whitelisting, and precisely controls its CPU and memory resource usage through cgroups technology.

[0122] The behavior auditing and circuit breaker unit monitors the system call frequency and memory access patterns of plugins in real time. When abnormal behavior is detected, it automatically triggers dynamic permission downgrade and revokes the plugin's access permissions to sensitive data.

[0123] Third-party functionalities are integrated via standardized protocols in the form of WebAssembly modules. The system runs these plugins in a sandbox environment and approves all sensitive data access requests through a secure interface, ensuring that core data is not illegally accessed. The open plugin sandbox architecture employs a modular security design, utilizing a four-layer protection mechanism to achieve controllable access and secure execution of third-party functionalities.

[0124] 1. Module Encapsulation and Validation: All third-party plugins must be submitted in WebAssembly binary format, which provides a memory-isolated sandbox execution environment. The system performs rigorous static validation during load time.

[0125] Type checking: Verifies that the types of all function call parameters and return values ​​match the interface specification.

[0126] Memory boundary check: Ensures that the module only accesses pre-allocated memory regions.

[0127] Control flow integrity: Modules that pass the verification that all redirection target addresses are within the legal range are assigned a unique capability token, in the format of HMAC-SHA256(plugin ID|developer certificate|feature list).

[0128] 2. Security Interface Gateway Design: The Classroom Plugin Interface (CPI) protocol enables controlled data access and includes the following core mechanisms:

[0129] Data anonymization proxy: Performs real-time face blurring on the original video stream, providing only anonymized feature vectors to the plugin.

[0130] Query rewriting engine: Converts plugin data requests into standardized SQL templates, for example:

[0131] SELECT anonymized_feature FROM student_state

[0132] WHERE timestamp > NOW() - INTERVAL '5 MINUTE'

[0133] Differential privacy filter: Adds Lap(Δf / ϵ) noise to the statistical query results, where the sensitivity Δf is dynamically adjusted according to the query type.

[0134] 3. Resource isolation: Each plugin instance runs in an independent lightweight container, including the following isolation measures:

[0135] File system virtualization: The plugin can only access the mounted temporary directory / tmp / plugin_[hash]

[0136] Network whitelist: Restrict outbound connections to predefined API endpoints

[0137] CPU / Memory Quotas: Limit the resource usage of a single plugin to no more than 15% of the total resources via cgroups.

[0138] 4. The behavior auditing and circuit breaker real-time monitoring system records abnormal behavior patterns of the plugin:

[0139] System call frequency detection: Counts file operations or network requests exceeding 50 per second.

[0140] Memory Access Pattern Analysis: Detecting Unconventional Large-Span Memory Reads

[0141] Dynamic permission downgrade: Automatically revoke access permissions for sensitive data when suspicious behavior is detected.

[0142] In some embodiments, the student cognitive state tensor output by the lightweight multimodal edge AI inference engine includes at least four fine-grained states: focus, confusion, understanding, and fatigue, and the end-to-end inference latency is controlled within 100 milliseconds.

[0143] Secondly, this application proposes a method for locally deployed intelligent interactive classrooms based on an edge intelligence architecture, such as... Figure 2 As shown, it includes the following steps:

[0144] S100: Locally in the classroom, multimodal raw data is collected through a heterogeneous sensor network, and then preprocessed by cleaning, feature extraction and identity anonymization on the edge server;

[0145] S200: Input the preprocessed multimodal features into the lightweight edge AI inference engine, and analyze and output the student's fine-grained cognitive state in real time through the spatiotemporal separation attention and gating cross-attention fusion mechanism;

[0146] S300: During the model training phase, the local gradients are differentially privacy-preservingly pruned and adaptively noise-perturbed using a federated learning framework, and secure aggregation is performed using a dual-mask protocol; during the model inference phase, the model and data are loaded into a trusted execution environment, and all computation processes are encrypted and obfuscated for protection.

[0147] S400: Inputs the student's real-time cognitive state and historical learning sequence into the personalized learning path generation module based on deep reinforcement learning and attention knowledge tracking, and dynamically outputs the next best teaching action, including adjusting the difficulty of questions or pushing specific learning resources.

[0148] S500: Securely connects to third-party plugins through a memory-isolated WebAssembly sandbox. All data access requests are anonymized, audited, and controlled via a secure interface gateway.

[0149] The technical solution of this invention aims to construct an end-to-end intelligent interactive classroom system deployed within the physical space of a classroom. The following will elaborate on the specific workflow of this system using a complete junior high school mathematics classroom scenario to ensure the clarity, completeness, and feasibility of the technical solution:

[0150] Application scenario settings:

[0151] In a ninth-grade math class on "Graphs and Properties of Quadratic Functions," the system of this invention was pre-installed in the classroom. The class had 30 students, each equipped with a smart answering terminal. A 1080P high-definition network camera was installed under the ceiling at each of the four corners of the classroom, and a set of 6-array digital microphones was deployed in the teacher's area. All acquisition devices were connected to an edge computing server located on the back wall of the classroom via Cat6 network cables. The teaching objective of this lesson was to enable students to master the vertex coordinates, axis of symmetry, and opening direction of quadratic functions.

[0152] Step 1: Localized Acquisition and Lightweight Preprocessing of Multimodal Learning Data

[0153] Data collection:

[0154] Video Stream: Four cameras begin capturing the entire classroom at 60fps. The cameras' built-in H.265 encoders compress the raw video stream in real time and transmit it to the edge server via a dedicated video VLAN (priority 6) to ensure smooth, lag-free streaming.

[0155] Audio Streaming: A 6-array microphone array captures all sound in the classroom in real time. Its built-in beamforming algorithm initially separates the teacher's questions, student responses, and ambient noise based on the direction of the sound source. High-fidelity 48kHz / 24bit audio stream is transmitted via audio VLAN (priority 5).

[0156] Interactive data: Students' smart devices record every writing and clicking action in real time. For example, when a teacher presents a multiple-choice question, "What are the coordinates of the vertex of the quadratic function y=(x-1)²+2?", the students' answers are recorded in detail.

[0157] Localized preprocessing (executed on the edge server):

[0158] Video Processing: After receiving the video stream, the server immediately calls the pruned YOLOv5s model for face detection. To completely eliminate privacy risks, the system performs irreversible pixelation blurring on all detected face regions, ensuring that only de-identified video features are stored and processed subsequently. Simultaneously, the system employs a dynamic frame sampling strategy: when most students are listening (with minimal head movement), face detection and keyframe extraction are maintained at 5fps; when the teacher announces the start of the exercise and students lower their heads to write (increased head movement), the system automatically increases the processing frame rate to 15fps to capture more subtle facial expressions. Finally, the system generates a feature vector for each student in real time, containing blink frequency, head pose Euler angles, and facial expression encoding, and binds it to the student ID (an internal system mapping ID, not a real student ID).

[0159] Audio Processing: The server uses the RNNoise algorithm to filter out ambient noise such as air conditioner and fan noise, enhancing the speech signal-to-noise ratio. Subsequently, a speech segmentation module based on voiceprint features separates the mixed audio stream into independent audio tracks for each student. When student number 3 stands up to answer a question, their speech is accurately captured. The Voice Activity Detection (VAD) module, based on the Conditional Random Field algorithm, accurately segments the valid speech segment "I think the vertex coordinates are (1, 2)" and extracts its Mel-frequency cepstral coefficients (MFCC) features.

[0160] Interactive data processing: The raw data from the answering terminal is standardized into the format of [Student ID, Question ID, Answer, Response Time (ms), Average Pen Pressure, Number of Hesitations]. The system compares the individual response time with the historical average response time for the same question in real time and calculates the response time deviation.

[0161] After the above steps, all original audio and video streams are securely erased from memory immediately after processing. Only the de-identified multimodal feature data packets associated with student number 3 are retained on the edge server, preparing for the next step of fusion analysis.

[0162] Step 2: A lightweight multimodal edge AI inference engine performs cognitive state analysis;

[0163] The preprocessed multimodal features are fed into a lightweight multimodal edge AI inference engine in real time.

[0164] Video Feature Extraction: The MobileViT-XXS model in the video branch analyzes the feature sequence of student number 3. The model's internal spatiotemporal attention mechanism first captures the frequency of activation of the corrugator supercilii muscle in the last 30 seconds (22 times / min), significantly higher than the student's baseline (10 times / min). Simultaneously, the temporal attention module detects that the student's gaze is fixed on a specific area of ​​the answer terminal screen for an extended period, rather than following the teacher's blackboard writing. The model initially outputs a video feature vector indicating a "confused" tendency.

[0165] Audio Semantic Understanding: The TinyBERT model in the audio branch analyzes the MFCC features of student number 3. It not only captures the "vertex coordinates" of the speech content through the keyword recognition module, but also detects the rising and pausing at the end of sentences through the intonation analysis module. These acoustic features are highly correlated with doubt. The model outputs a high-scoring feature representing the "semantic doubt level."

[0166] Text Behavior Analysis: Text branch analysis of student #3's answer data. Regarding the vertex coordinate multiple-choice question, the response time was 45 seconds, far exceeding the average response time for this type of question (25 seconds), indicating a positive deviation. Simultaneously, handwriting pressure Gram matrix analysis showed drastic fluctuations in pressure during writing, exhibiting a typical "pause-pressure-pause" pattern, which usually signifies mental blocks and hesitation.

[0167] Cross-modal fusion: The feature vectors output from the three branches are fed into the entry-level cross-attention fusion module. At this point, since the system detects that the student is "answering questions," the fusion module calculates the inter-modal importance weights as follows: video 0.3, audio 0.2, and text 0.5. This means that the text and video modalities are assigned higher weights. Finally, the fusion module generates a high-dimensional comprehensive feature tensor and, with a classifier and a confidence score of 0.94, determines that student number 3 is currently in a state of "conceptual confusion," specifically, "unclear understanding of the geometric meaning of the vertex coordinate formula (h, k)." The entire end-to-end inference process takes approximately 85 milliseconds, meeting the requirements for real-time interaction.

[0168] Step 3: Model optimization and secure inference under a two-layer privacy protection mechanism;

[0169] Federated Learning and Differential Privacy Tuning (to be done after class):

[0170] After this lesson, the edge servers in each class, acting as clients of federated learning, fine-tune the general teaching model using the local data collected during the lesson. After calculating the local gradient, the system first performs gradient pruning: limiting the L2 norm of the gradients of all parameters to within 0.3. Next, adaptive noise injection is implemented: since this is the 5th round of federated learning, the system calculates the noise scale added in this round as σ = 0.8 * (0.9^4) ≈ 0.52 according to the preset decay strategy (initial σ = 0.8, decay factor 0.9). Gaussian noise is added to the pruned gradient. Finally, in the secure aggregation phase, each client generates a pair of random vectors and uploads "gradient + noise + random vector 1 - random vector 2" to the central server. Since the sum of the random vectors from all clients is zero, the central server successfully aggregates the new global model without knowing the gradient information of any single classroom.

[0171] Trusted execution environment reasoning (conducted in real-time in class):

[0172] On the second day of class, when the system needed to perform real-time analysis on student number 3, its operation flow was as follows: The encrypted model stored on the hard drive was first loaded into the protected memory of the Intel SGX enclave, and was only decrypted when the CPU performed calculations. All input feature vectors of student number 3 were directly passed to the enclave through a pre-established secure channel. Inside the enclave, the model underwent forward propagation. For each intermediate feature map generated during the calculation process, the system generated a random mask matrix of the same size as that layer and multiplied it element-wise to achieve obfuscation. Even if malware compromised the operating system during inference and attempted a side-channel attack on the server memory, it could only read a string of meaningless noise that had been highly encrypted and obfuscated, and could not reconstruct any real information related to the student's cognitive state.

[0173] Step 4: Generation of dynamic personalized learning paths based on deep reinforcement learning;

[0174] After the system accurately identified that student number 3 was confused about the concept of "vertex coordinates", it immediately triggered personalized intervention.

[0175] State Encoding and Knowledge Tracking: The system first combines Student 3's current cognitive state (conceptual confusion, confidence level 0.94) with five answer records from the past 20 minutes (three correct answers regarding the opening direction, two incorrect answers regarding vertex coordinates) to form a historical sequence, which is then input into the LSTM-Attention knowledge tracking model. The model captures dependencies in the sequence using a bidirectional LSTM and then uses a temporal attention mechanism to discover that the two most recent errors regarding vertex coordinates have the greatest impact on the current state. The model ultimately predicts that without intervention, Student 3's probability of mastering the next knowledge point, "judging the translation of a function graph using vertex coordinates," will plummet to 35%.

[0176] Reinforcement Learning Strategy Decision-Making: The core of the system—a deep reinforcement learning policy network based on PPO-Clip—makes decisions based on the current state (fusion features + predicted probabilities). This network has learned, through simulations and historical classroom data, how to maximize a reward function that comprehensively considers "accuracy improvement," "learning efficiency," and "cognitive load." For student #3's predicament, the policy network outputs an optimal teaching action instruction: temporarily switching the current learning path to a "basic consolidation and micro-lesson tutoring" mode.

[0177] Dynamic execution and feedback loop:

[0178] Action 1 (Difficulty Downgrade): The system pauses the delivery of new comprehensive application problems (difficulty level 0.7) to student number 3 and instead delivers a basic concept discrimination question (difficulty level 0.3): "Which of the following options is the vertex coordinate of the quadratic function y=(x-3)²+5?"

[0179] Action 2 (Push Micro-lesson): The system retrieves and pushes a 2-minute micro-lesson video explaining the "geometric meaning of vertex coordinates" from the local resource library to student number 3's terminal.

[0180] After watching the micro-lesson, Student #3 resumed answering the questions. Their response time decreased to 28 seconds, handwriting pressure fluctuations stabilized, and they answered correctly consecutively in subsequent variation exercises. The system continuously collected this new data, forming a real-time closed loop of "intervention-feedback-reassessment." Five minutes later, the LSTM-Attention model predicted that their mastery probability of "function graph translation" had rebounded to 85%. The policy network detected this change and output instructions again: gradually reduce the difficulty of the learning path back to normal levels and introduce a new knowledge point, "axis of symmetry."

[0181] Step 5: Secure Access to the Open Plugin Sandbox Architecture

[0182] An edtech company developed an innovative "handwriting-based cognitive style analysis" plugin, hoping to conduct a classroom pilot program using this system. The company compiled the plugin into a WebAssembly module and submitted it.

[0183] Loading and Verification: When the system loads this module, it first performs rigorous static verification, including checking the types of all function parameters, ensuring that memory accesses do not exceed preset boundaries, and verifying the legality of control flow jump addresses. Upon successful verification, the system generates a unique HMAC-SHA256-based capability token for the plugin.

[0184] Controlled Data Access: The plugin requests student handwriting pressure data from the system via the CPI protocol. The request first reaches the security interface gateway. After verifying its capability token, the gateway rewrites the request, converting it into a standardized SQL query: "SELECT anonymized_pressure_gram FROM student_state WHERE timestamp > NOW() - INTERVAL '5 MINUTE'". Simultaneously, all returned handwriting Gram matrices have been anonymized by the gateway, removing any direct association with the student ID.

[0185] Isolation and Auditing: The plugin runs in an independent, lightweight container with a virtualized file system, allowing access to only a temporary directory. cgroups limit its CPU usage to no more than 10% of total resources. During operation, a behavior auditing module continuously monitors its behavior. When the system detects that the plugin is attempting to access the file system at an abnormal frequency (>100 times / second), it determines it as a potential threat, immediately triggers a circuit breaker mechanism, dynamically revokes its data access permissions, and logs the information for administrator review, thereby ensuring the security and stability of the core system.

[0186] Thirdly, this application proposes an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0187] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0188] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0189] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0190] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0191] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0194] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0195] The above are merely preferred embodiments of the present invention. It should be noted that any modifications and improvements made by those skilled in the art without departing from the present technical solution should also be considered to fall within the scope of protection claimed by the present solution.

Claims

1. A locally deployed intelligent interactive classroom system based on an edge intelligence architecture, characterized in that, include: The multimodal data acquisition and preprocessing module is deployed in the local physical space of the classroom. It is used to collect students' video, audio and answer behavior data in real time through heterogeneous sensor networks, and to clean, extract features and anonymize the data locally to ensure that the original sensitive data does not leave the classroom. A lightweight multimodal edge AI inference engine is deployed on an edge server in the classroom to receive preprocessed multimodal features. Through a spatiotemporal separation attention mechanism and a gated cross-attention fusion architecture, it analyzes and outputs students' fine-grained cognitive state tensors in real time. A two-layer privacy protection mechanism, including a federated learning differential privacy gradient perturbation module during the model training phase and an encrypted computing module based on a trusted execution environment during the model inference phase, is used to build end-to-end data privacy protection from data collection to model application. The dynamic personalized learning path generation module, based on a knowledge tracking model enhanced by deep reinforcement learning networks and attention mechanisms, dynamically adjusts the difficulty of teaching content and the distribution of knowledge points according to students' real-time cognitive state and historical learning sequence, generating personalized learning paths. An open plugin sandbox architecture is used to securely connect to third-party teaching applications in the form of memory-isolated WebAssembly modules, and to anonymize and audit access to sensitive data through a secure interface gateway to ensure that core data is not illegally obtained.

2. The system according to claim 1, characterized in that: The multimodal data acquisition and preprocessing module includes: The video acquisition unit consists of four 1080P wide-angle cameras supporting H.265 hardware encoding deployed at the four corners of the classroom. It transmits data through an independent video virtual local area network and adopts a dynamic frame sampling strategy to adaptively adjust the frequency of face detection and key frame extraction based on the range of student head movements. The audio acquisition unit consists of a 6-array digital microphone. It uses beamforming technology to locate and separate the sound source, acquires the original waveform with 48kHz / 24bit precision, and performs voiceprint separation and speech enhancement processing. The answer terminal unit is equipped with a touch screen and a pressure-sensitive pen to record students' answers, response time, handwriting trajectory and pressure changes, and standardizes them into a structured data format that includes timestamps. The preprocessing unit is used to perform face detection and background blurring on the video stream, speech activity detection and feature extraction on the audio stream, and format unification and normalization on the interactive data.

3. The system according to claim 2, characterized in that: The lightweight multimodal edge AI inference engine includes: The video feature extraction branch adopts the lightweight MobileViT model optimized for edge devices. This model integrates a spatiotemporal separation attention mechanism, decomposes the 3D convolution into spatial self-attention and temporal self-attention, and combines a dynamic frame sampling strategy to extract spatiotemporal features such as students' micro-expressions, head posture, and gaze direction. The audio feature extraction branch adopts the knowledge-distilled TinyBERT model. The input of this model is the multi-scale spliced ​​Mel frequency cepstral coefficient features, and the effective speech units are accurately segmented by the speech activity detection algorithm based on conditional random fields to understand semantic content and tone features. The text behavior analysis branch is used to analyze the interactive data of the answer terminal, including normalizing the response time through statistical methods and encoding the pressure fluctuation pattern during the writing process through Gram matrix to reflect the student's thinking state. The cross-modal fusion module employs a gated cross-attention mechanism, which dynamically calculates the importance weights of video, audio, and text modalities in the current context using learnable gating values, and then fuses the weighted multimodal features to generate a comprehensive feature vector for cognitive state discrimination.

4. The system according to claim 3, characterized in that: The federated learning differential privacy gradient perturbation module in the two-layer privacy protection mechanism includes: a differential privacy gradient pruning unit, which is used to perform L2 norm pruning on the model gradient calculated by the local client, scaling gradients exceeding a preset threshold to within the threshold, so as to limit the contribution of a single sample to the model update; An adaptive noise injection unit is used to dynamically adjust the scale of Laplacian or Gaussian noise added to the clipped gradient according to the current training epoch. Larger noise is used in the early stage of training to enhance privacy protection, and the noise decays exponentially with each epoch, while a minimum noise threshold is set to maintain basic protection. The secure aggregation unit employs a double-mask protocol, which enables each client to obfuscate the gradient with a pair of random vectors that sum to zero before uploading. After aggregating all encrypted gradients, the server can restore the true gradient sum without being able to snoop on the original gradient information of a single client.

5. The system according to claim 4, characterized in that: The trusted execution environment encrypted computation module in the two-layer privacy protection mechanism includes: The encrypted model loading unit is used to convert the pre-trained model parameters into a format compatible with the trusted execution environment and encrypts them using sealed storage technology to ensure that the model exists in encrypted form on both the hard drive and in memory, and is decrypted only when the computation is performed within the CPU enclave. The secure computing unit is used to receive input data transmitted through a secure channel within the enclave, and to obfuscate and protect the intermediate feature map tensors generated during the forward propagation of the model. By generating a random mask matrix and performing element-wise operations with the feature map, it prevents attackers from reconstructing the real information through memory sniffing. The hardware key derivation unit uses a unique key derived from the trusted execution environment hardware, combined with national cryptographic algorithms such as SM4, to encrypt and decrypt data within the enclave, ensuring that the key is not exposed.

6. The system according to claim 5, characterized in that: The dynamic personalized learning path generation module includes: The multimodal state encoder adopts an improved cross-modal Transformer structure, which maps video, audio and text features to the same space through linear transformation and then performs layer normalization. It also calculates cross-modal correlation through a multi-head attention mechanism to enhance the expression of key cognitive state information and generate the student state code at the current moment. The deep reinforcement learning strategy network is constructed based on the PPO-Clip algorithm. It takes the state code as input, the question difficulty adjustment amount and the knowledge point weight vector as the output action space, and the reward function is an index that comprehensively considers the answer accuracy, learning efficiency and cognitive load. The teaching strategy is stably optimized through the pruning mechanism. The LSTM-Attention knowledge tracking and prediction model constructs a bidirectional LSTM network to process students' learning history sequences, calculates the importance weight of each historical moment to the current prediction through a temporal attention mechanism, and finally predicts the probability of students mastering future knowledge points through a multilayer perceptron, which can be used to push pre-study resources in advance or adjust the teaching pace.

7. The system according to claim 6, characterized in that: The LSTM-Attention knowledge tracking and prediction model further includes: A bidirectional LSTM unit is used to process the student's historical learning sequence from both forward and backward directions, and to concatenate the hidden states from the two directions to capture the complete contextual dependencies in the sequence. The temporal attention weight calculation unit applies a linear transformation and a sigmoid activation function to the concatenated hidden state at each time step to generate normalized attention weights, which are used to measure the impact of the learning event at that time step on the prediction of future mastery. The weighted summation and prediction unit sums the hidden states at each time step according to their corresponding attention weights, inputs the resulting context vector into the multilayer perceptron, and finally outputs the probability that the student has mastered a specific knowledge point at a future time point.

8. The system according to claim 7, characterized in that: The open plug-in sandbox architecture includes: The module encapsulation and static verification unit is used to receive third-party plugins submitted in WebAssembly binary format. Before loading, it performs type checking, memory boundary checking and control flow integrity verification. Modules that pass the verification are granted a unique and unforgeable capability token. The security interface gateway enables controlled data access based on the CPI protocol, including a de-identification proxy for real-time face blurring of the raw video stream, a query rewriting engine that rewrites plugin requests into standardized SQL templates, and adds Laplace noise to statistical query results to meet differential privacy requirements. The resource isolation execution unit creates an independent lightweight container for each verified plugin instance, restricts its access to only temporary directories through file system virtualization, restricts its outbound connections through network whitelisting, and precisely controls its CPU and memory resource usage through cgroups technology. The behavior auditing and circuit breaker unit monitors the system call frequency and memory access patterns of plugins in real time. When abnormal behavior is detected, it automatically triggers dynamic permission downgrade and revokes the plugin's access permissions to sensitive data.

9. The system according to claim 8, characterized in that, The student cognitive state tensor output by the lightweight multimodal edge AI inference engine includes at least four fine-grained states: focused, confused, understanding, and tired, and the end-to-end inference latency is controlled within 100 milliseconds.

10. A method for locally deploying intelligent interactive classrooms based on an edge intelligence architecture, applied to the system described in any one of claims 1 to 9, characterized in that, Includes the following steps: Locally in the classroom, multimodal raw data is collected through a heterogeneous sensor network, and then preprocessed by cleaning, feature extraction, and identity anonymization on an edge server. The preprocessed multimodal features are input into a lightweight edge AI inference engine, which analyzes and outputs the student's fine-grained cognitive state in real time through a spatiotemporal separation attention and gating cross-attention fusion mechanism. During the model training phase, the local gradients are differentially privacy-preserving and adaptively noise-perturbed using a federated learning framework, and secure aggregation is performed using a dual-mask protocol. During the model inference phase, the model and data are loaded into a trusted execution environment, and all computation processes are protected by encryption and obfuscation. The module generates a personalized learning path based on deep reinforcement learning and attention knowledge tracking by inputting students’ real-time cognitive state and historical learning sequence into the module, and dynamically outputs the next best teaching action, including adjusting the difficulty of questions or pushing specific learning resources. Secure access to third-party plugins is achieved through a memory-isolated WebAssembly sandbox, with all data access requests being anonymized, audited, and controlled via a secure interface gateway.

Images