Intelligent classroom teaching optimization method and system combining behavior recognition and Internet of Things

Abstract

This invention provides a method and system for optimizing intelligent classroom teaching by combining behavior recognition with the Internet of Things (IoT). This system involves acquiring student behavior data, audio and video data, and environmental data through a distributed IoT data acquisition system deployed on student terminals. This data is then processed using a multimodal feature fusion network to obtain characteristics of student interaction status. Based on these characteristics, a remote teaching interaction evaluation model is constructed to calculate student attention levels and classroom engagement in real time, generating personalized teaching guidance strategies. This method enables intelligent perception of student status and dynamic optimization of teaching strategies during remote teaching.

Description

Technical Field

[0001] The present invention relates to smart classroom technology, and in particular to a smart classroom teaching optimization method and system combining behavior recognition and the Internet of Things. Background Art

[0002] With the rapid development of internet technology, distance learning has become an integral part of education. During distance learning, teachers struggle to intuitively assess students' learning status and classroom engagement, significantly impacting teaching quality and effectiveness. Currently, distance learning platforms primarily use cameras to capture video data from students and analyze their online learning status using computer vision technology. Furthermore, by collecting data on students' online learning platform operations, they track and analyze their learning behavior, providing teachers with feedback.

[0003] However, existing distance learning optimization methods have the following shortcomings: First, existing methods mainly rely on single video data or operation data for analysis, lacking comprehensive analysis of student behavior data, audio data, and environmental data, and are unable to comprehensively and accurately assess students' actual learning status. Second, existing methods usually use static feature extraction methods and fail to fully consider the temporal variation characteristics of student status, resulting in a lack of temporal consistency in the assessment results of students' attention levels and classroom participation. Third, existing methods often use fixed rule models when generating teaching guidance strategies, failing to make adaptive adjustments based on individual differences and environmental factors of different students, making it difficult to provide teachers with personalized teaching optimization suggestions.

[0004] To address the above problems, the present invention proposes an intelligent classroom teaching optimization method that combines behavior recognition with the Internet of Things. Through multimodal data collection and fusion analysis, it realizes intelligent perception and dynamic optimization of the remote teaching process. Summary of the Invention

[0005] The embodiments of the present invention provide an intelligent classroom teaching optimization method and system combining behavior recognition and the Internet of Things, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides intelligent classroom teaching optimization methods that combine behavior recognition and the Internet of Things, including:

[0008] Remote teaching interaction data is collected through a distributed Internet of Things data collection system, the distributed Internet of Things data collection system including a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal. The remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, the environmental data including lighting intensity data, background noise data, and network status data;

[0009] Constructing a multimodal feature fusion network to process the distance learning interaction data, the multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the time series analysis subnetwork analyzes the temporal variation patterns of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vectors with the environmental data to obtain student interaction status features;

[0010] A remote teaching interaction evaluation model is constructed based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and classroom participation value in real time; according to the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, thereby realizing dynamic optimization and adjustment of the remote teaching process.

[0011] The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the time series analysis subnetwork analyzes the temporal variation patterns of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vectors with the environmental data to obtain student interaction status features including:

[0012] Behavioral data features, audio data features, and video data features are extracted separately through a parallel three-way feature extraction architecture. The behavioral data features are extracted by using a multi-scale convolution layer and a residual connection to extract local features of the facial area, and the head posture data is processed by a dual-stream attention module. The output result of the dual-stream attention module is adaptively weighted by a feature recalibration module to obtain a behavioral feature vector. The audio data features are extracted by obtaining a time-frequency spectrum through a short-time Fourier transform, and a multi-scale one-dimensional convolution is used to extract the local time-frequency features of the time-frequency spectrum. The local time-frequency features are processed by a channel attention mechanism to obtain an audio feature vector. The video data features are extracted by using a spatiotemporal convolution module to process a sixteen-frame image sequence to obtain a video feature vector.

[0013] Inputting the behavior feature vector, the audio feature vector, and the video feature vector into a bidirectional long short-term memory network for temporal analysis, wherein the bidirectional long short-term memory network includes a modality-specific forget gate mechanism, wherein the modality-specific forget gate mechanism uses a first forget gate threshold for the behavior feature vector and a second forget gate threshold for the audio feature vector and the video feature vector, wherein the first forget gate threshold is greater than the second forget gate threshold; and processing the output features of the bidirectional long short-term memory network through a temporal attention mechanism to obtain temporal features;

[0014] The temporal features are input into the attention fusion subnetwork, which includes an intra-modal attention module and an inter-modal attention module. The intra-modal attention module calculates the importance weights of features at different time steps within the same modality, and the inter-modal attention module learns the complementary relationship between features of different modalities. The environmental data is mapped to the feature vector dimensional space through a multi-layer perceptron to obtain environmental features, and a conditional attention mechanism is constructed based on the environmental features to guide the feature fusion process. The output features of the attention fusion subnetwork are enhanced through residual connections and self-attention mechanisms to obtain student interaction status features.

[0015] A distance learning interaction evaluation model is constructed based on the student interaction status characteristics. The distance learning interaction evaluation model calculates the student attention level value and the class participation value in real time, including:

[0016] Receive a student interaction state feature sequence within a ten-second time window and input the student interaction state feature sequence into a temporal convolution module. The temporal convolution module extracts multi-scale temporal dependency features using different convolution kernel sizes. The multi-scale temporal dependency features are passed through a temporal importance evaluation unit to calculate the attention weight of each time step to obtain micro-evaluation features.

[0017] Inputting the micro-evaluation features into a hierarchical long short-term memory network, wherein the bottom network of the hierarchical long short-term memory network processes the micro-evaluation features to generate intermediate attention features, and the top network of the hierarchical long short-term memory network integrates the intermediate attention features to obtain long-term change features;

[0018] Inputting the long-term variation features into a multi-task learning framework, the multi-task learning framework extracts universal interaction features through a multi-head self-attention mechanism, and the universal interaction features are respectively input into an attention level evaluation branch and a class engagement evaluation branch, wherein the attention level evaluation branch includes a spiking neural network module, and the class engagement evaluation branch includes a temporal reasoning module of a graph attention network;

[0019] A learning scenario recognition module is constructed based on the course type and teaching activity form. The output of the learning scenario recognition module is passed through a scenario adaptation layer to obtain feature weights and evaluation thresholds. The feature weights and evaluation thresholds are input into a Bayesian neural network, which uses Monte Carlo dropout to quantify the uncertainty of model predictions and obtain a reliability index.

[0020] The output result of the attention level evaluation branch is passed through a nonlinear mapping function to obtain an attention level value, and the output result of the classroom participation evaluation branch is passed through a cumulative probability distribution function to obtain a classroom participation value. The attention level value and the classroom participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result.

[0021] The method further includes determining an attention evaluation value by a spiking neural network module:

[0022] Normalizing the input universal interaction features to obtain normalized features, mapping the normalized features to the time domain via a logarithmic function to generate a time-series pulse pattern, inputting the time-series pulse pattern into a first neuron layer having 256 integrating and releasing neurons, the integrating and releasing neurons of the first neuron layer converting input current into postsynaptic current via synaptic weights, and the synaptic weights are dynamically adjusted via a spike-timing-dependent plasticity learning rule;

[0023] Inputting the output signal of the first neuron layer into a second neuron layer having one hundred and twenty-eight leaky integral-sparking neurons, constructing lateral inhibitory connections between the leaky integral-sparking neurons in the second neuron layer to form a local competition mechanism, wherein the inhibitory strength of the lateral inhibitory connections decreases as the distance between adjacent neurons increases;

[0024] Inputting the output signal of the second neuron layer into a third neuron layer having sixty-four adaptive leaky integrate-and-send neurons, wherein the firing thresholds of the adaptive leaky integrate-and-send neurons in the third neuron layer are adaptively adjusted according to the neuron activity history;

[0025] Establishing a multi-time-scale membrane potential dynamics model for each adaptive leaky integrate-and-spark neuron in the third neuronal layer, wherein the multi-time-scale membrane potential dynamics model includes a fast synaptic current with a time constant of milliseconds, a slow modulation current with a time constant of hundreds of milliseconds, and an adaptation current with a time constant of seconds;

[0026] Calculate the group average firing rate of the third neuron layer within the time window, map the group average firing rate to the interval of zero to one through the sigmoid function to obtain an initial attention value, and perform exponential sliding average on the initial attention value to obtain a final attention evaluation value, wherein the size of the time window dynamically shrinks as the group average firing rate increases.

[0027] The method further includes determining an engagement representation vector by a temporal reasoning module of a graph attention network:

[0028] Mapping the input universal interaction feature sequence to a graph structure, where the feature vector of each time step corresponds to a node, and the feature dimension of the node is 512. Eight attention heads are used to calculate the edge weights between nodes. The attention heads perform linear transformations on the node features to obtain query vectors and key vectors. Attention coefficients are calculated based on the query vectors and the key vectors, and the outputs of the eight attention heads are concatenated to obtain an edge weight matrix.

[0029] A three-layer graph attention network is constructed. In the first layer, each node is connected to the nodes within two adjacent time steps and relative position encoding is introduced. In the second layer, the connection range is expanded to the adjacent five time steps and features are extracted through multi-scale convolution operations. In the third layer, global connections are achieved and long-range dependency features are extracted through the transformer structure.

[0030] Performing spectral clustering on the node features output by the three-layer graph attention network to obtain multiple node clusters, performing a maximum pooling operation on each of the node clusters to obtain node cluster features, calculating the importance weight of each of the node cluster features, and weighting the node cluster features based on the corresponding importance weights to obtain an engagement representation vector;

[0031] The method further comprises:

[0032] A gated graph neural network is used to update the node information in the graph structure. The degree of retention of historical information is calculated through the update gate, and the integration ratio of new information is calculated through the reset gate. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node to obtain the updated node state.

[0033] Generate a personalized teaching guidance strategy based on the calculation results of the remote teaching interaction evaluation model. The personalized teaching guidance strategy includes suggestions for adjusting the teaching rhythm and optimizing the interaction mode, including:

[0034] A sliding time window is constructed to perform time series analysis on the attention level and classroom participation output by the distance learning interactive evaluation model. The sliding time window has a window size of ten minutes and slides every thirty seconds. The mean, standard deviation, rate of change, and periodic pattern of the attention level and classroom participation within the sliding time window are calculated to obtain the time series variation characteristics of the attention level and the dynamic characteristics of the classroom participation.

[0035] performing cognitive load analysis based on the temporal variation characteristics of the attention level, detecting the trend of attention level changes within three consecutive sliding time windows, triggering rhythm adjustment when the attention level drops by more than 20%, estimating the cognitive load cycle based on the cyclical pattern of the attention level, and dividing the cognitive load cycle into a cognitively active period and a cognitively fatigued period;

[0036] Based on the division results of the cognitive active period and the cognitive fatigue period, an interaction strategy is generated in combination with the dynamic characteristics of the class participation. The optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period. A mapping matrix between the interaction form and the effect of improving the class participation is established. The interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages.

[0037] Constructing an environmental feature vector, performing a multivariate regression analysis on the temporal variation characteristics of the attention level and the dynamic characteristics of the classroom participation with the environmental feature vector, quantifying the influence weight of each environmental factor in the environmental feature vector, and making environmental adaptability corrections to the interaction depth adjustment plan based on the influence weights to obtain a personalized interaction strategy that includes environmental parameter thresholds;

[0038] The execution effect of the personalized interactive strategy at different cognitive stages is input into a decision tree model. Based on the real-time feedback of the attention level and the classroom participation, the strategy generation rules of the decision tree model are continuously optimized through reinforcement learning to achieve dynamic adjustment and optimization of the teaching guidance strategy.

[0039] Based on the division results of the cognitive active period and the cognitive fatigue period, an interaction strategy is generated in combination with the dynamic characteristics of the class participation. The optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period. A mapping matrix between the interaction form and the effect of improving the class participation is established. The interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages, including:

[0040] An exponential decay function was used to model the time series of classroom participation. Classroom participation data was collected within a five-minute sliding time window. The classroom participation decay parameter was obtained through least squares fitting, and a decay curve of classroom participation over time was established.

[0041] Calculate the second-order derivative of the attenuation curve, detect the inflection point where the second-order derivative turns from negative to positive, verify the local minimum constraint within one minute before and after the inflection point, and determine the time difference between the inflection point that meets the constraint and the last interaction time as the optimal interaction interval;

[0042] Construct a mapping matrix between interaction forms and effect indicators, where the effect indicators include the magnitude of improvement in class participation, the duration of improvement in class participation, and the interaction response rate. Assign a time-decay weight to the historical data of each effect indicator, and calculate the weighted average score of different interaction forms on each effect indicator.

[0043] Setting different weight coefficients for the effect indicators based on the current cognitive stage, performing weighted calculation on the weight coefficients and the scores of the mapping matrix to obtain an interaction form score, and selecting the interaction form with the highest score as the interaction strategy for the current cognitive stage;

[0044] Calculating the duration during which the class participation is maintained above the baseline level after each interaction, and inputting the difference ratio between the duration and the expected duration into the depth adjustment function to generate an interaction depth adjustment plan;

[0045] Setting constraints on the interaction depth adjustment scheme to limit the change in interaction depth between two consecutive times to no more than 30%, lowering the upper limit of the depth adjustment to 15% during cognitive fatigue, and resetting to the baseline depth after three consecutive adjustments are ineffective;

[0046] Calculate the weighted combination score of the improvement in classroom participation, classroom participation duration, and interactive response rate for each interaction, and update the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function based on the weighted combination score to achieve dynamic optimization of interactive strategy generation.

[0047] According to a second aspect of the embodiments of the present invention,

[0048] Provides an intelligent classroom teaching optimization system that combines behavior recognition and the Internet of Things, including:

[0049] The first unit is configured to collect remote teaching interaction data through a distributed Internet of Things data collection system. The distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal. The remote teaching interaction data includes student behavior data, audio data, video data, and environmental data. The behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, including lighting intensity data, background noise data, and network status data.

[0050] The second unit is used to construct a multimodal feature fusion network to process the remote teaching interaction data. The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork. The feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively. The time series analysis subnetwork analyzes the time series variation patterns of each feature vector based on the long short-term memory network structure. The attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain student interaction status features.

[0051] The third unit is used to construct a remote teaching interaction evaluation model based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and classroom participation value in real time; generates a personalized teaching guidance strategy based on the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, thereby realizing dynamic optimization and adjustment of the remote teaching process.

[0052] According to a third aspect of the embodiments of the present invention,

[0053] An electronic device is provided, comprising:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] According to a fourth aspect of the embodiments of the present invention,

[0058] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0059] The beneficial effects of this application are as follows:

[0060] 1. Improve the accuracy and comprehensiveness of teaching assessments: This technology collects multimodal data (behavior, audio, video, and environment) and analyzes it using a multimodal feature fusion network. This allows for a more comprehensive and accurate assessment of students' learning status, including attention levels and classroom engagement. Compared to traditional single-dimensional assessment methods (such as relying solely on classroom performance or homework grades), this method can capture more subtle changes in students' emotions and behaviors, thereby more accurately reflecting their true learning situation.

[0061] 2. Personalized teaching guidance: A remote teaching interaction assessment model, built based on student interaction status characteristics, can calculate students' attention levels and engagement in real time and generate personalized teaching guidance strategies, such as suggestions for adjusting teaching pace and optimizing interaction methods. This personalized guidance can help teachers better improve the relevance and effectiveness of their teaching.

[0062] 3. Optimize the remote teaching experience and improve teaching efficiency: By providing real-time feedback on personalized teaching strategies to teachers through IoT communication networks, teachers can adjust teaching methods and pacing, enhance classroom interaction, and increase student interest and motivation. This technology also helps teachers better manage remote classrooms, optimize the remote teaching experience, and ultimately improve teaching efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of an intelligent classroom teaching optimization method combining behavior recognition and the Internet of Things according to an embodiment of the present invention;

[0064] Figure 2 This is a structural diagram of an intelligent classroom teaching optimization system that combines behavior recognition and the Internet of Things according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0066] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0067] Figure 1 This is a flow chart of an intelligent classroom teaching optimization method combining behavior recognition and the Internet of Things according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0068] S101. Collecting remote teaching interaction data through a distributed Internet of Things data collection system, the distributed Internet of Things data collection system including a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal, the remote teaching interaction data including student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, the environmental data including lighting intensity data, background noise data, and network status data;

[0069] S102. Construct a multimodal feature fusion network to process the distance learning interaction data. The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork. The feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from the student behavior data, audio data, and video data, respectively. The time series analysis subnetwork analyzes the temporal variation patterns of each feature vector based on a long short-term memory network structure. The attention fusion subnetwork adaptively fuses the multimodal feature vectors with the environmental data to obtain student interaction status features.

[0070] S103. A remote teaching interaction evaluation model is constructed based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and the classroom participation value in real time; according to the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

[0071] First, a distributed IoT data collection system is deployed on student terminals. The system consists of three modules: behavioral data collection module, audio and video data collection module, and environmental data collection module.

[0072] The behavioral data collection module uses a camera to capture students' facial expressions and head posture. For example, image recognition technology can identify whether a student is frowning, smiling, looking up, or looking down, and convert this data into feature data, such as frown frequency, smile duration, and head tilt angle. For example, if a student frowns three times in one minute, each time for an average of 0.5 seconds, and the student tilts their head downward by an average of 15 degrees, these data will be recorded as part of the behavioral data.

[0073] The audio and video data acquisition module uses microphones and cameras to collect student voice and image information. For example, it can record audio features such as volume, speaking rate, and speaking duration, as well as video features such as whether the student raises their hand or interacts with other students. For example, if a student speaks twice within one minute, with an average volume of 70 decibels and a speaking rate of 150 words per minute, this data will be recorded as part of the audio data.

[0074] The environmental data collection module uses sensors to collect data such as the light intensity, background noise, and network status of the student's environment. For example, a light sensor can record the classroom's brightness, a noise sensor can record the noise level, and a network status sensor can record the student's network connection speed and stability. Assuming the classroom brightness is 500 lux, the noise level is 40 decibels, and the student's network latency is 20 milliseconds, these data will be recorded as part of the environmental data.

[0075] Next, a multimodal feature fusion network is constructed to process the collected data. The network consists of three sub-networks: feature extraction sub-network, time series analysis sub-network, and attention fusion sub-network.

[0076] The feature extraction subnetwork extracts feature vectors from behavioral data, audio data, and video data. For example, it extracts features such as frown frequency and smile duration from behavioral data, volume and speech rate from audio data, and hand-raising times and interaction frequency from video data.

[0077] The time series analysis subnetwork uses a long short-term memory (LSTM) architecture to analyze the temporal variations of each feature vector. For example, it can analyze the changing trends in a student's attention level to determine whether they are gradually losing or regaining focus. For example, if a student's attention level is high for the first five minutes and then gradually decreases, the time series analysis subnetwork can capture this trend.

[0078] The attention fusion subnetwork adaptively fuses multimodal feature vectors with environmental data to generate features representing the student's interaction status. For example, it combines student attention levels and classroom engagement with environmental data such as classroom brightness, noise levels, and network status to generate a feature vector that reflects the student's overall interaction status.

[0079] Then, a remote teaching interaction evaluation model is constructed based on the characteristics of student interaction status. This model calculates the student's attention level and classroom participation in real time. For example, based on the student's feature vector, the student's attention level is calculated to be 80 points and the classroom participation is calculated to be 75 points.

[0080] Based on the evaluation model's calculation results, personalized teaching strategies are generated. For example, if a student's attention level is low, the system will suggest that the teacher adjust the teaching pace, such as slowing down the speech speed and increasing interactive sessions. If the student's class participation is low, the system will suggest that the teacher optimize the interactive method, such as increasing questions and conducting group discussions.

[0081] Finally, personalized teaching strategies are fed back to the teacher's terminal in real time via the IoT communication network, enabling dynamic optimization and adjustment of the remote teaching process. For example, guidance strategies such as "suggesting slower speaking speed" or "suggesting more questions" can be sent to the teacher's computer or mobile phone in real time, allowing the teacher to adjust the teaching plan in a timely manner.

[0082] The beneficial effects of this method can be summarized in the following three aspects:

[0083] 1. Improve teaching effectiveness: By monitoring students' interaction status in real time and providing personalized teaching guidance strategies, teachers can better understand students' learning situation and adjust teaching plans in a timely manner, thereby improving teaching effectiveness.

[0084] 2. Promote personalized learning: Based on the characteristics of students' interactive status, personalized learning suggestions and resources can be provided to students to help them better master the learning content.

[0085] 3. Optimize the allocation of teaching resources: By analyzing students' learning data, it can help schools better understand students' learning needs, optimize the allocation of teaching resources, and improve the utilization efficiency of teaching resources.

[0086] In an optional embodiment, the multimodal feature fusion network includes a feature extraction subnetwork, a timing analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the timing analysis subnetwork analyzes the temporal variation patterns of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain student interaction status features including:

[0087] Behavioral data features, audio data features, and video data features are extracted separately through a parallel three-way feature extraction architecture. The behavioral data features are extracted by using a multi-scale convolution layer and a residual connection to extract local features of the facial area, and the head posture data is processed by a dual-stream attention module. The output result of the dual-stream attention module is adaptively weighted by a feature recalibration module to obtain a behavioral feature vector. The audio data features are extracted by obtaining a time-frequency spectrum through a short-time Fourier transform, and a multi-scale one-dimensional convolution is used to extract the local time-frequency features of the time-frequency spectrum. The local time-frequency features are processed by a channel attention mechanism to obtain an audio feature vector. The video data features are extracted by using a spatiotemporal convolution module to process a sixteen-frame image sequence to obtain a video feature vector.

[0088] Inputting the behavior feature vector, the audio feature vector, and the video feature vector into a bidirectional long short-term memory network for temporal analysis, wherein the bidirectional long short-term memory network includes a modality-specific forget gate mechanism, wherein the modality-specific forget gate mechanism uses a first forget gate threshold for the behavior feature vector and a second forget gate threshold for the audio feature vector and the video feature vector, wherein the first forget gate threshold is greater than the second forget gate threshold; and processing the output features of the bidirectional long short-term memory network through a temporal attention mechanism to obtain temporal features;

[0089] The temporal features are input into the attention fusion subnetwork, which includes an intra-modal attention module and an inter-modal attention module. The intra-modal attention module calculates the importance weights of features at different time steps within the same modality, and the inter-modal attention module learns the complementary relationship between features of different modalities. The environmental data is mapped to the feature vector dimensional space through a multi-layer perceptron to obtain environmental features, and a conditional attention mechanism is constructed based on the environmental features to guide the feature fusion process. The output features of the attention fusion subnetwork are enhanced through residual connections and self-attention mechanisms to obtain student interaction status features.

[0090] Obtain student behavior data, audio data, video data, and environmental data. For example, in a classroom group discussion scenario, behavioral data uses cameras to capture students' facial expressions and head postures, audio data uses microphones to record students' voices, and video data uses cameras to record students' body movements. Environmental data includes the number of group members, classroom lighting intensity, and noise level. Assume that a set of data contains four students and lasts 10 minutes.

[0091] First, multimodal features are extracted. For behavioral data, a ResNet-50 network is used to extract local features of the facial region. Head pose data (e.g., head orientation, nodding frequency) is fed into two separate fully connected layers to generate two feature vectors. These two feature vectors are then passed through a feature recalibration module, which consists of a fully connected layer and a Sigmoid activation function to generate weights for the two feature vectors. The two weighted feature vectors are concatenated with the local features extracted by ResNet-50 to generate a behavioral feature vector. For audio data, a short-time Fourier transform is used to convert the audio data into a spectrogram. Multiple one-dimensional convolutional layers with different kernel sizes are then used to extract local time-frequency features from the spectrogram. These local features are then processed using a channel-wise attention mechanism, which consists of global average pooling, a fully connected layer, and a Sigmoid activation function to generate weights for each channel, resulting in an audio feature vector. For video data, a 3D convolutional network, such as the C3D network, is used to process a sequence of 16 consecutive frames to extract a video feature vector.

[0092] Next, time series analysis is performed. The extracted behavioral feature vectors, audio feature vectors, and video feature vectors are input into a bidirectional long short-term memory network (Bi-LSTM). The Bi-LSTM network contains a modality-specific forget gate mechanism. Specifically, for behavioral feature vectors, the first forget gate threshold is used; for audio and video feature vectors, the second forget gate threshold is used. The first forget gate threshold is set to 0.7, and the second forget gate threshold is set to 0.5, which means that redundant information in behavioral features is more easily forgotten, while more information in audio and video features is retained. The output features of the Bi-LSTM are then processed by a temporal attention mechanism, which generates weights for each time step through a fully connected layer and a softmax activation function to obtain time series features.

[0093] Finally, feature fusion is performed. The temporal features are input into the attention fusion subnetwork. This subnetwork consists of an intra-modal attention module and an inter-modal attention module. The intra-modal attention module uses fully connected layers and a softmax activation function to calculate the importance weights of features at different time steps within the same modality. The inter-modal attention module learns the complementary relationships between features from different modalities by calculating the similarity between feature vectors (for example, cosine similarity). Environmental data is mapped to the same dimensional space as the feature vectors using a multi-layer perceptron to obtain environmental features. A conditional attention mechanism is constructed based on the environmental features. For example, after concatenating the environmental features with features from each modality, weights are generated using a fully connected layer and a softmax activation function to guide the feature fusion process. The output features of the attention fusion subnetwork are then enhanced using residual connections and a self-attention mechanism to obtain the final student interaction status features. The self-attention mechanism uses scaled dot product attention to calculate the relationship between different dimensions of the features and performs weighted fusion. The resulting student interaction status features can be used to assess student engagement and motivation, for example.

[0094] Beneficial effects:

[0095] 1. Improved accuracy in identifying students' interactive states: Through multimodal feature fusion, we can more comprehensively capture students' various behaviors, thereby improving recognition accuracy. For example, we can more accurately identify whether students are actively participating in discussions and whether they understand the class content.

[0096] 2. Adaptive fusion of features from different modalities: The attention mechanism dynamically adjusts the weights of features from different modalities based on their importance, enabling adaptive fusion. For example, in a group discussion, if a student's voice is more important, the audio feature will be given a higher weight; if a student's body language is more important, the video feature will be given a higher weight.

[0097] 3. Enhanced model robustness: The modality-specific forget gate mechanism can filter out redundant information in different modal features, thereby enhancing the robustness of the model. For example, in a noisy classroom environment, the audio features may contain a lot of noise information. The forget gate mechanism can effectively filter out this noise and retain the important speech information.

[0098] In an optional embodiment, a remote teaching interaction evaluation model is constructed based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and classroom participation value in real time, including:

[0099] Receive a student interaction state feature sequence within a ten-second time window and input the student interaction state feature sequence into a temporal convolution module. The temporal convolution module extracts multi-scale temporal dependency features using different convolution kernel sizes. The multi-scale temporal dependency features are passed through a temporal importance evaluation unit to calculate the attention weight of each time step to obtain micro-evaluation features.

[0100] Inputting the micro-evaluation features into a hierarchical long short-term memory network, wherein the bottom network of the hierarchical long short-term memory network processes the micro-evaluation features to generate intermediate attention features, and the top network of the hierarchical long short-term memory network integrates the intermediate attention features to obtain long-term change features;

[0101] Inputting the long-term variation features into a multi-task learning framework, the multi-task learning framework extracts universal interaction features through a multi-head self-attention mechanism, and the universal interaction features are respectively input into an attention level evaluation branch and a class engagement evaluation branch, wherein the attention level evaluation branch includes a spiking neural network module, and the class engagement evaluation branch includes a temporal reasoning module of a graph attention network;

[0102] A learning scenario recognition module is constructed based on the course type and teaching activity form. The output of the learning scenario recognition module is passed through a scenario adaptation layer to obtain feature weights and evaluation thresholds. The feature weights and evaluation thresholds are input into a Bayesian neural network, which uses Monte Carlo dropout to quantify the uncertainty of model predictions and obtain a reliability index.

[0103] The output result of the attention level evaluation branch is passed through a nonlinear mapping function to obtain an attention level value, and the output result of the classroom participation evaluation branch is passed through a cumulative probability distribution function to obtain a classroom participation value. The attention level value and the classroom participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result.

[0104] Obtain a sequence of student interaction state features within a ten-second time window. A time window consists of multiple time steps, each of which contains multiple features, such as the number of mouse clicks, keyboard strokes, camera gaze duration, voice volume, and head posture change frequency. For example, in a time window of 10 time steps, each time step records five student interaction state features. The student interaction state feature sequence for that time window is a 10-row, 5-column matrix.

[0105] The student interaction state feature sequence is input into the temporal convolution module. This module uses convolution kernels of varying sizes to extract multi-scale temporal dependency features. For example, convolution kernels of sizes 3 and 5 are used to extract short-term and long-term temporal dependency features, respectively. Assuming the input feature sequence is a 10x5 matrix, convolution with a 3x5 kernel and a stride of 1 yields an 8x1 feature vector. Convolution with a 5x5 kernel and a stride of 1 yields a 6x1 feature vector. These two feature vectors are concatenated to yield a 14x1 feature vector, representing the multi-scale temporal dependency feature.

[0106] The multi-scale temporal dependency features are input into the temporal importance assessment unit. This unit calculates the attention weights for each time step to obtain micro-evaluation features. For example, an attention mechanism, such as the Softmax function, can be used to calculate the weights for each time step. Assuming the multi-scale temporal dependency feature is a 14x1 vector, after passing it through the Softmax function, a 14x1 weight vector is obtained. The weight vector is multiplied by the corresponding multi-scale temporal dependency feature to obtain the weighted multi-scale temporal dependency feature, i.e., the micro-evaluation feature.

[0107] The micro-evaluation features are input into a hierarchical long short-term memory (LSTM) network. The bottom layer of the network processes the micro-evaluation features to generate intermediate attention features. The top layer integrates the intermediate attention features to generate long-term variation features. For example, the bottom layer can consist of multiple LSTM units, each processing a portion of the micro-evaluation features. The top layer then integrates the outputs of the bottom layers, for example, using another LSTM unit, to generate long-term variation features. Assuming the micro-evaluation features are a 14x1 vector, the bottom layer contains two LSTM units, each processing 7 features and outputting two 7x1 vectors. The top layer concatenates these two 7x1 vectors into a 14x1 vector, which is then input into a single LSTM unit to generate the final long-term variation features, for example, a 1x64 vector.

[0108] The long-term change features are fed into the multi-task learning framework. This framework uses a multi-head self-attention mechanism to extract general interaction features. These general interaction features are fed into the attention level assessment branch and the classroom engagement assessment branch. For example, eight self-attention heads are used, each extracting different features. These features are then concatenated to generate general interaction features. Assuming the long-term change features are a 1x64 vector, after the multi-head self-attention mechanism, a 1x512 general interaction feature is generated.

[0109] The attention level assessment branch includes a spiking neural network module. This module simulates the information processing method of biological neurons, processes the general interaction features, and outputs features related to the attention level. For example, the Leaky Integrate-and-Fire (LIF) model can be used to construct a spiking neural network. Assuming the general interaction feature is a 1x512 vector, after passing it through the spiking neural network, a 1x1 attention level feature is obtained.

[0110] The class engagement assessment branch includes a temporal reasoning module within a graph attention network. This module leverages the graph structure to capture the relationships between students' interaction states, performs temporal reasoning, and outputs features related to class engagement. For example, a graph can be constructed where nodes represent students and edges represent interactions between students. Assuming the general interaction feature is a 1x512 vector, after passing it through the temporal reasoning module within the graph attention network, a 1x1 class engagement feature is obtained.

[0111] A learning scenario recognition module is constructed based on the course type and teaching activity format. The module's output is passed through the scenario adaptation layer to obtain feature weights and evaluation thresholds. For example, for "explaining" courses, the "asking questions" activity has a higher weight; for "discussing" courses, the "speaking" activity has a higher weight. If the learning scenario recognition module identifies the current scenario as a "asking questions" activity in an "explaining" course, the scenario adaptation layer will output the corresponding feature weights and evaluation thresholds.

[0112] The feature weights and evaluation thresholds are input into a Bayesian neural network. The network uses Monte Carlo dropout to quantify the uncertainty of the model's predictions and generate a reliability metric. For example, multiple dropout operations are performed on the Bayesian neural network to generate multiple predictions. The variance of these predictions is then calculated as a reliability metric.

[0113] The output of the attention level evaluation branch is passed through a nonlinear mapping function to obtain the attention level value. For example, the Sigmoid function is used to map the attention level feature to a value between 0 and 1. Assuming the attention level feature is 0.8, after the Sigmoid function, the attention level value obtained is 0.69.

[0114] Apply the output of the class participation evaluation branch to the cumulative probability distribution function to obtain the class participation value. For example, using the normal distribution cumulative probability distribution function. Suppose the class participation feature is 1.5. After applying the normal distribution cumulative probability distribution function, the class participation value is 0.93.

[0115] The attention level and class participation values ​​are linearly interpolated to achieve a smooth transition to the final evaluation result. For example, the attention level and class participation values ​​are weighted averaged according to the reliability index. Assuming the attention level value is 0.69, the class participation value is 0.93, and the reliability index is 0.8, the final evaluation result is 0.69*(1-0.8)+0.93*0.8=0.882.

[0116] Beneficial effects:

[0117] 1. Improved assessment accuracy: Through multi-scale temporal feature extraction, hierarchical attention modeling, and multi-task learning, the model can more comprehensively capture the student interaction state, thereby improving the accuracy of assessment.

[0118] 2. Enhanced evaluation reliability: Through the learning scenario recognition module and Bayesian neural network, the model can adjust the evaluation strategy according to different learning scenarios and quantify the uncertainty of the prediction, thereby enhancing the reliability of the evaluation.

[0119] 3. Personalized assessment: This model takes into account the course type, teaching activity form, and individual differences of students, enabling more personalized assessment and providing support for personalized learning.

[0120] In an optional embodiment, the method further includes determining an attention evaluation value by a spiking neural network module:

[0121] Normalizing the input universal interaction features to obtain normalized features, mapping the normalized features to the time domain via a logarithmic function to generate a time-series pulse pattern, inputting the time-series pulse pattern into a first neuron layer having 256 integrating and releasing neurons, the integrating and releasing neurons of the first neuron layer converting input current into postsynaptic current via synaptic weights, and the synaptic weights are dynamically adjusted via a spike-timing-dependent plasticity learning rule;

[0122] Inputting the output signal of the first neuron layer into a second neuron layer having one hundred and twenty-eight leaky integral-sparking neurons, constructing lateral inhibitory connections between the leaky integral-sparking neurons in the second neuron layer to form a local competition mechanism, wherein the inhibitory strength of the lateral inhibitory connections decreases as the distance between adjacent neurons increases;

[0123] Inputting the output signal of the second neuron layer into a third neuron layer having sixty-four adaptive leaky integrate-and-send neurons, wherein the firing thresholds of the adaptive leaky integrate-and-send neurons in the third neuron layer are adaptively adjusted according to the neuron activity history;

[0124] Establishing a multi-time-scale membrane potential dynamics model for each adaptive leaky integrate-and-spark neuron in the third neuronal layer, wherein the multi-time-scale membrane potential dynamics model includes a fast synaptic current with a time constant of milliseconds, a slow modulation current with a time constant of hundreds of milliseconds, and an adaptation current with a time constant of seconds;

[0125] Calculate the group average firing rate of the third neuron layer within the time window, map the group average firing rate to the interval of zero to one through the sigmoid function to obtain an initial attention value, and perform exponential sliding average on the initial attention value to obtain a final attention evaluation value, wherein the size of the time window dynamically shrinks as the group average firing rate increases.

[0126] Get the input general interaction features. General interaction features can be multimodal, for example, including image visual features, text semantic features, user behavior features, etc. A specific example is that in an e-commerce recommendation system, general interaction features can include the image feature vector of the user's historical browsing history (for example, a 2048-dimensional feature vector extracted using ResNet50), the text semantic feature vector of the product title (for example, a 768-dimensional feature vector extracted using BERT), and the user's click, favorite, purchase, and other behavioral features over the past week (for example, a 3-dimensional vector representing the number of clicks, favorites, and purchases, respectively).

[0127] Normalize the acquired common interaction features. Normalization can be performed using a variety of methods, such as min-max normalization and Z-score normalization. The goal is to scale the numerical ranges of features of different dimensions to a uniform interval, such as between 0 and 1, to prevent certain features from having a significant impact on the results due to their large numerical ranges. For example, in the e-commerce recommendation system described above, min-max normalization can be performed on the image feature vector, text semantic feature vector, and user behavior feature vector, scaling the numerical range of each feature vector to between 0 and 1.

[0128] The normalized features are mapped to the time domain using a logarithmic function to generate a time-series pulse pattern. The logarithmic function is used here to convert the eigenvalue into the time interval of the pulse emission. The larger the eigenvalue, the shorter the corresponding pulse emission time interval, indicating a higher pulse emission frequency. For example, the normalized eigenvalue x can be converted to time t using the function -log(1-x), where the time unit can be milliseconds. Assuming that the normalized value of an image feature is 0.9, the corresponding pulse emission time interval is -log(1-0.9)≈2.3 milliseconds.

[0129] The generated temporal spike pattern is input to the first neuron layer. The first neuron layer contains 256 integrate-and-release neurons. Each neuron receives an input spike signal and converts it into a postsynaptic current. The strength of the postsynaptic current is determined by the synaptic weight. The synaptic weight is dynamically adjusted using a spike-timing-dependent plasticity learning rule. For example, if a neuron's input spike and its own output spike are close in time, the synaptic weight connecting the two neurons is strengthened; conversely, if the input spike and output spike are far apart in time, the synaptic weight is weakened.

[0130] The output signals of the first neuron layer are input to the second neuron layer. This second neuron layer contains 128 leaky integration-and-release neurons. These neurons establish lateral inhibitory connections, forming a local competition mechanism. The inhibitory strength of these lateral inhibitory connections decreases with the distance between adjacent neurons. This means that the activity of one neuron inhibits the activity of its surrounding neurons, and the closer the distance, the greater the inhibitory strength. This local competition mechanism enhances neuronal selectivity, ensuring that only the most responsive neurons remain active.

[0131] The output signals of the second neuron layer are input to the third neuron layer. This third neuron layer contains 64 adaptive leaky integrate-and-spark neurons. The firing thresholds of these neurons are adaptively adjusted based on the neuron's activity history. For example, if a neuron fires frequently within a short period of time, its firing threshold increases, making it more difficult for it to fire again. Conversely, if a neuron does not fire for a long time, its firing threshold decreases, making it more likely to fire. This adaptive mechanism prevents neurons from becoming overly excited or overly inhibited.

[0132] A multi-timescale membrane potential dynamics model was established for each neuron in the third layer. This model incorporates a fast synaptic current with a time constant on the order of milliseconds, a slow modulating current with a time constant on the order of hundreds of milliseconds, and an adaptation current with a time constant on the order of seconds. These three currents work together to determine changes in the neuron's membrane potential, thereby influencing its firing behavior.

[0133] Calculate the population average firing rate of the third neuron layer within the time window. For example, you can count the total number of spikes fired by 64 neurons in a 100-ms time window, and then divide it by the number of neurons and the time window size to get the population average firing rate.

[0134] The initial attention value is obtained by mapping the group average firing rate to the range of 0 to 1 through the sigmoid function. The sigmoid function is an S-shaped function that can map any value between 0 and 1.

[0135] The final attention estimate is obtained by applying an exponential sliding average to the initial attention values. Exponential sliding average is a commonly used smoothing method that can reduce fluctuations in attention values. The size of the time window dynamically decreases as the population average firing rate increases. This means that when neuronal activity increases, the time window becomes smaller, allowing for a faster response to changes.

[0136] Beneficial effects:

[0137] 1. Improve the accuracy of attention assessment: By simulating the information processing mechanism of biological neurons, multimodal features can be extracted and integrated more effectively, thereby improving the accuracy of attention assessment.

[0138] 2. Enhance the robustness of the model: The multi-timescale membrane potential dynamics model and adaptive neuronal mechanism make the model more robust to noise and input changes.

[0139] 3. Improve computing efficiency: The event-driven characteristics of the pulse neural network can reduce the amount of calculation and improve computing efficiency, which is especially suitable for application scenarios with high real-time requirements.

[0140] In an optional embodiment, the method further includes determining an engagement representation vector by a temporal reasoning module of a graph attention network:

[0141] Mapping the input universal interaction feature sequence to a graph structure, where the feature vector of each time step corresponds to a node, and the feature dimension of the node is 512. Eight attention heads are used to calculate the edge weights between nodes. The attention heads perform linear transformations on the node features to obtain query vectors and key vectors. Attention coefficients are calculated based on the query vectors and the key vectors, and the outputs of the eight attention heads are concatenated to obtain an edge weight matrix.

[0142] A three-layer graph attention network is constructed. In the first layer, each node is connected to the nodes within two adjacent time steps and relative position encoding is introduced. In the second layer, the connection range is expanded to the adjacent five time steps and features are extracted through multi-scale convolution operations. In the third layer, global connections are achieved and long-range dependency features are extracted through the transformer structure.

[0143] Performing spectral clustering on the node features output by the three-layer graph attention network to obtain multiple node clusters, performing a maximum pooling operation on each of the node clusters to obtain node cluster features, calculating the importance weight of each of the node cluster features, and weighting the node cluster features based on the corresponding importance weights to obtain an engagement representation vector;

[0144] The method further comprises:

[0145] A gated graph neural network is used to update the node information in the graph structure. The degree of retention of historical information is calculated through the update gate, and the integration ratio of new information is calculated through the reset gate. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node to obtain the updated node state.

[0146] Take a sequence containing common user interaction features, such as a user's viewing history on a video platform, including viewing time, likes, comments, shares, and other behaviors. Assume that the sequence consists of 10 time steps, and the feature dimension of each time step is 512, which can be represented as a 10x512 matrix.

[0147] Map the input universal interaction feature sequence to a graph structure. The feature vector of each time step corresponds to a node in the graph, and the feature dimension of the node is 512. In this example, 10 nodes will be obtained, each node representing the interaction feature of a time step.

[0148] Calculate edge weights between nodes. Use eight attention heads to calculate edge weights between nodes. Each attention head performs a linear transformation on node features, generating a query vector and a key vector. Taking the first attention head as an example, each node's 512-dimensional feature vector is linearly transformed into a query vector and a key vector. Then, based on the query and key vectors, an attention coefficient is calculated, reflecting the degree of association between nodes. The outputs of the eight attention heads are concatenated to form a 10x10 edge weight matrix, which represents the strength of the connection between any two nodes.

[0149] Construct a three-layer graph attention network. In the first layer of the graph attention network, each node is connected to the nodes in the adjacent positive and negative two time steps, and relative position encoding is introduced to distinguish the relative position of the node in the sequence. For example, the third node is connected to the 1st, 2nd, 4th, and 5th nodes. In the second layer of the graph attention network, the connection range is expanded to the adjacent positive and negative five time steps, and features are extracted through multi-scale convolution operations. For example, the third node is connected to the 1st, 2nd, 4th, 5th, 6th, 7th, and 8th nodes, and the convolution kernel sizes are 3, 5, and 7 respectively to capture temporal information of different scales. In the third layer of the graph attention network, global connection is achieved, each node is connected to all nodes, and long-range dependency features are extracted through the Transformer structure to capture global information.

[0150] Perform spectral clustering on the node features output by the three-layer graph attention network. Assume that spectral clustering divides 10 nodes into 3 clusters.

[0151] Perform a max pooling operation on each node cluster to obtain the node cluster features. For example, perform a max pooling operation on the node features in the first cluster to obtain a 512-dimensional cluster feature vector.

[0152] Calculate the importance weight of each node cluster feature. For example, calculate the importance weight of the cluster based on the number of nodes in each cluster and the variance of the node features. Suppose the importance weights of the three clusters are 0.2, 0.3, and 0.5 respectively.

[0153] The node cluster features are weighted based on their corresponding importance weights to obtain the participation representation vector. The feature vectors of the three clusters are multiplied by the corresponding weights and then added together to obtain the final 512-dimensional participation representation vector.

[0154] A gated graph neural network is used to update node information in the graph structure. Taking the first node as an example, the update gate calculates the degree of historical information retention, and the reset gate calculates the proportion of new information incorporated. Assume the output of the update gate is 0.8 and the output of the reset gate is 0.2. Based on the outputs of the update and reset gates, a candidate state is calculated. The candidate state is weightedly fused with the node's current state (512-dimensional feature vector) to obtain the updated node state. Similar update operations are performed on all nodes.

[0155] Beneficial effects:

[0156] 1. Improved the accuracy of the model's user engagement predictions. By combining a graph attention network and a gated graph neural network, the model can more effectively capture the complex temporal dependencies and inter-node associations in user interaction sequences, thereby improving prediction accuracy.

[0157] 2. Enhanced model interpretability. By dividing nodes into different clusters through spectral clustering and calculating the importance weight of each cluster, we can explain the contribution of different types of user behaviors to engagement and improve the interpretability of the model.

[0158] 3. Improved model generalization. The combination of multi-scale convolution and Transformer architecture enables the model to capture temporal information and long-range dependencies at different scales, enhancing its generalization and adapting to different user behavior patterns.

[0159] In an optional embodiment, a personalized teaching guidance strategy is generated based on the calculation results of the remote teaching interaction evaluation model. The personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions, including:

[0160] A sliding time window is constructed to perform time series analysis on the attention level and classroom participation output by the distance learning interactive evaluation model. The sliding time window has a window size of ten minutes and slides every thirty seconds. The mean, standard deviation, rate of change, and periodic pattern of the attention level and classroom participation within the sliding time window are calculated to obtain the time series variation characteristics of the attention level and the dynamic characteristics of the classroom participation.

[0161] performing cognitive load analysis based on the temporal variation characteristics of the attention level, detecting the trend of attention level changes within three consecutive sliding time windows, triggering rhythm adjustment when the attention level drops by more than 20%, estimating the cognitive load cycle based on the cyclical pattern of the attention level, and dividing the cognitive load cycle into a cognitively active period and a cognitively fatigued period;

[0162] Based on the division results of the cognitive active period and the cognitive fatigue period, an interaction strategy is generated in combination with the dynamic characteristics of the class participation. The optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period. A mapping matrix between the interaction form and the effect of improving the class participation is established. The interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages.

[0163] Constructing an environmental feature vector, performing a multivariate regression analysis on the temporal variation characteristics of the attention level and the dynamic characteristics of the classroom participation with the environmental feature vector, quantifying the influence weight of each environmental factor in the environmental feature vector, and making environmental adaptability corrections to the interaction depth adjustment plan based on the influence weights to obtain a personalized interaction strategy that includes environmental parameter thresholds;

[0164] The execution effect of the personalized interactive strategy at different cognitive stages is input into a decision tree model. Based on the real-time feedback of the attention level and the classroom participation, the strategy generation rules of the decision tree model are continuously optimized through reinforcement learning to achieve dynamic adjustment and optimization of the teaching guidance strategy.

[0165] The distance teaching interaction evaluation model outputs data on students' attention levels and classroom participation during distance learning.

[0166] First, the system constructs a sliding time window with a window size of ten minutes, sliding every thirty seconds. The system calculates the mean, standard deviation, rate of change, and periodic pattern of attention levels and classroom participation within each time window. For example, within a time window, the attention levels of 10 students are 80, 85, 90, 75, 82, 88, 92, 78, 85, and 90, respectively. The mean value of the attention level within the time window is 84.5, and the standard deviation is 5.8. The same is true for other indicators. Through such calculations, the system can obtain the temporal variation characteristics of attention levels and the dynamic characteristics of classroom participation.

[0167] Next, the system performs cognitive load analysis based on the temporal variation characteristics of the attention level. The system continuously monitors the changing trend of the attention level in three consecutive sliding time windows. For example, if the average value of the attention level in three consecutive time windows is 85, 70, and 60 respectively, the decline is (85-60) / 85=29.4%, which exceeds the set 20% threshold, and the system triggers a rhythm adjustment. At the same time, the system estimates the student's cognitive load cycle based on the periodic pattern of the attention level, and divides it into a cognitively active period and a cognitively fatigued period. For example, if it is found that the student's attention level shows a cyclical change of one cycle every 30 minutes, 30 minutes can be used as a cognitive load cycle, and it can be divided into a cognitively active period and a cognitively fatigued period according to the level of attention.

[0168] Then, based on the division results of cognitively active periods and cognitively fatigued periods, the system generates an interaction strategy in combination with the dynamic characteristics of classroom participation. The system calculates the inflection point of the attenuation curve of classroom participation during the cognitively active period. For example, through observation, it is found that students' classroom participation is high within 10 minutes after the start of the cognitively active period, and then begins to decline. In this case, 10 minutes can be set as the optimal interaction interval. At the same time, the system will establish a mapping matrix between the form of interaction and the effect of improving classroom participation. For example, by analyzing historical data, it is found that question-based interaction has the most significant effect on improving classroom participation. In this case, question-based interaction can be given priority during the cognitively active period. The system will also determine the interaction depth adjustment plan based on the duration of classroom participation in different cognitive stages. For example, if students' classroom participation continues to be low during the cognitive fatigue period, the depth of interaction can be appropriately reduced to avoid overstimulation.

[0169] In addition, the system constructs environmental feature vectors, such as ambient light, ambient noise, and network connection quality, and performs a multivariate regression analysis on the temporal variation characteristics of attention levels and the dynamic characteristics of classroom participation with the environmental feature vectors to quantify the weight of the impact of environmental factors on student learning. For example, if the analysis finds that the impact of environmental noise on student attention levels is relatively large, the system will make environmental adaptability adjustments to the interaction depth adjustment plan based on the environmental noise level and set environmental parameter thresholds. For example, when the environmental noise exceeds a certain threshold, the system will automatically reduce the frequency and depth of interaction to reduce interference with students. Ultimately, the system will generate a personalized interaction strategy that includes environmental parameter thresholds.

[0170] Finally, the system inputs the effectiveness of personalized interaction strategies at different cognitive stages into a decision tree model. Based on real-time feedback on attention levels and classroom engagement, the system continuously optimizes the decision tree model's strategy generation rules through reinforcement learning, enabling dynamic adjustment and optimization of teaching guidance strategies. For example, if the system finds that a certain interaction strategy is not performing well at a specific cognitive stage, it will adjust the strategy's execution frequency or replace it with another strategy.

[0171] Through the above steps, the system can dynamically adjust the teaching rhythm and interaction strategy according to the students' real-time learning status and environmental factors, thereby improving the efficiency and quality of distance learning.

[0172] The beneficial effects can be summarized into the following three aspects:

[0173] 1. Personalized learning experience: The system can provide personalized teaching guidance based on each student’s cognitive load and learning status, so that each student can learn in the best state.

[0174] 2. Improve learning efficiency: By dynamically adjusting the teaching rhythm and interactive strategies, the system can effectively improve students' attention levels and classroom participation, thereby improving learning efficiency.

[0175] 3. Strong environmental adaptability: The system can automatically adjust the interactive strategy according to changes in environmental factors to ensure that teaching activities can proceed smoothly in various environments.

[0176] In an optional embodiment, based on the division results of the cognitive active period and the cognitive fatigue period, combined with the dynamic characteristics of the class participation, an interaction strategy is generated. The optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, and a mapping matrix between the interaction form and the effect of improving the class participation is established. The interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages, including:

[0177] An exponential decay function was used to model the time series of classroom participation. Classroom participation data was collected within a five-minute sliding time window. The classroom participation decay parameter was obtained through least squares fitting, and a decay curve of classroom participation over time was established.

[0178] Calculate the second-order derivative of the attenuation curve, detect the inflection point where the second-order derivative turns from negative to positive, verify the local minimum constraint within one minute before and after the inflection point, and determine the time difference between the inflection point that meets the constraint and the last interaction time as the optimal interaction interval;

[0179] Construct a mapping matrix between interaction forms and effect indicators, where the effect indicators include the magnitude of improvement in class participation, the duration of improvement in class participation, and the interaction response rate. Assign a time-decay weight to the historical data of each effect indicator, and calculate the weighted average score of different interaction forms on each effect indicator.

[0180] Setting different weight coefficients for the effect indicators based on the current cognitive stage, performing weighted calculation on the weight coefficients and the scores of the mapping matrix to obtain an interaction form score, and selecting the interaction form with the highest score as the interaction strategy for the current cognitive stage;

[0181] Calculating the duration during which the class participation is maintained above the baseline level after each interaction, and inputting the difference ratio between the duration and the expected duration into the depth adjustment function to generate an interaction depth adjustment plan;

[0182] Setting constraints on the interaction depth adjustment scheme to limit the change in interaction depth between two consecutive times to no more than 30%, lowering the upper limit of the depth adjustment to 15% during cognitive fatigue, and resetting to the baseline depth after three consecutive adjustments are ineffective;

[0183] Calculate the weighted combination score of the improvement in classroom participation, classroom participation duration, and interactive response rate for each interaction, and update the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function based on the weighted combination score to achieve dynamic optimization of interactive strategy generation.

[0184] Acquire classroom engagement data. During classroom instruction, student engagement data is collected every second. For example, cameras capture students' facial expressions, body movements, and eye contact, or voice recognition technology analyzes students' responses, questions, and discussions. This information is converted into a numerical value between 0 and 1, representing the student's engagement level, and stored in a database. For example, within a five-minute window, 300 classroom engagement data points were collected, with values ​​ranging from 0.2 to 0.9.

[0185] Perform time-series modeling of class participation. To analyze the dynamic trends of class participation, an exponential decay function is fitted to the class participation data. Specifically, class participation data is collected within a five-minute sliding time window, and then the least squares method is used to fit the parameters of the exponential decay function. For example, suppose the fitted exponential decay function is y = 0.8*exp(-0.1*x)+0.2, where y represents class participation and x represents time.

[0186] Determine the optimal interaction interval. In order to find the best time for interaction, it is necessary to calculate the inflection point of the attenuation curve. First, calculate the second-order derivative of the attenuation curve. Then, detect the inflection point where the second-order derivative changes from negative to positive. Within one minute before and after the inflection point, verify the local minimum constraint, that is, determine whether the function value at the inflection point is less than all function values ​​within one minute before and after it. If the constraint is met, the time difference between the inflection point and the last interaction moment is determined as the optimal interaction interval. For example, assuming that an inflection point is detected at the 8th minute and the local minimum constraint is met, and the last interaction occurs at the 3rd minute, the optimal interaction interval is 5 minutes.

[0187] Construct a mapping matrix between interaction forms and performance indicators. Create a matrix to record the relationship between different interaction forms and the effect of improving classroom engagement. Performance indicators include the magnitude of improvement in classroom engagement, the duration of improvement in classroom engagement, and the interaction response rate. Assign a time-decay weight to the historical data of each performance indicator, giving more recent data a higher weight. For example, use an exponential decay function for weighting. Then, calculate the weighted average score of different interaction forms on each performance indicator. For example, suppose there are three interaction forms: asking questions, group discussions, and games. By analyzing historical data, you can obtain their weighted average scores on different performance indicators.

[0188] Determine the interaction strategy. Based on the current cognitive stage, assign different weights to different performance indicators. For example, during the cognitively active phase, prioritize the magnitude of the increase in class participation; while during the cognitively fatigued phase, prioritize the duration of the increase. Combine these weights with the scores in the mapping matrix to obtain a final score for each interaction type. Select the interaction type with the highest score as the interaction strategy for the current cognitive stage. For example, if asking questions yields the highest final score during the cognitively active phase, select asking questions as the interaction strategy.

[0189] Adjust the depth of interaction. Calculate the duration of class engagement remaining above baseline after each interaction. Input the ratio of this duration to the expected duration into the depth adjustment function to generate an interaction depth adjustment plan. For example, if class engagement lasted 3 minutes after an interaction and the expected duration was 5 minutes, the difference ratio would be -0.4. Inputting -0.4 into the depth adjustment function yields a corresponding depth adjustment plan, such as reducing the difficulty of the interaction or increasing the frequency of interactions.

[0190] Set depth adjustment constraints. To prevent excessive changes in interaction depth, set constraints. Limit the change in depth between two consecutive interactions to no more than 30%. During periods of cognitive fatigue, reduce the upper limit on depth adjustment to 15%. If three consecutive adjustments are ineffective, reset to the baseline depth.

[0191] Dynamically optimize interaction strategies. Calculate a weighted combined score for each interaction, including the increase in class engagement, class engagement duration, and interaction response rate. Based on this score, update the parameters of the mapping matrix and the depth adjustment function to dynamically optimize interaction strategy generation. For example, if the weighted combined score of a particular interaction type is consistently low, lower its score in the mapping matrix or adjust the parameters of the depth adjustment function to make it more likely to trigger depth adjustment.

[0192] Beneficial effects:

[0193] 1. Improve classroom teaching efficiency: By monitoring classroom participation in real time and dynamically adjusting interaction strategies, students' learning interest and attention can be effectively improved, thereby improving classroom teaching efficiency. For example, based on case data, this method can increase the average classroom participation rate from 0.5 to 0.7.

[0194] 2. Personalized learning experience: This approach can provide personalized interactive experiences based on different students' cognitive states and learning characteristics, promoting better understanding and mastery of knowledge. For example, for students with low engagement, the frequency of interaction can be increased or the difficulty of interaction can be reduced.

[0195] 3. Data-driven teaching optimization: This method collects and analyzes classroom teaching data, providing teachers with timely feedback and guidance, helping them continuously improve their teaching methods and strategies. For example, by analyzing a matrix mapping interaction forms to effectiveness indicators, it is possible to identify which interaction forms are most effective and which need improvement.

[0196] Figure 2 This is a structural diagram of an intelligent classroom teaching optimization system that combines behavior recognition and the Internet of Things according to an embodiment of the present invention. Figure 2 As shown, the system includes:

[0197] The first unit is configured to collect remote teaching interaction data through a distributed Internet of Things data collection system. The distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal. The remote teaching interaction data includes student behavior data, audio data, video data, and environmental data. The behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, including lighting intensity data, background noise data, and network status data.

[0198] The second unit is used to construct a multimodal feature fusion network to process the remote teaching interaction data. The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork. The feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively. The time series analysis subnetwork analyzes the time series variation patterns of each feature vector based on the long short-term memory network structure. The attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain student interaction status features.

[0199] The third unit is used to construct a remote teaching interaction evaluation model based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and classroom participation value in real time; generates a personalized teaching guidance strategy based on the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, thereby realizing dynamic optimization and adjustment of the remote teaching process.

[0200] According to a third aspect of the embodiments of the present invention,

[0201] An electronic device is provided, comprising:

[0202] processor;

[0203] a memory for storing processor-executable instructions;

[0204] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0205] According to a fourth aspect of the embodiments of the present invention,

[0206] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0207] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The intelligent classroom teaching optimization method combining behavior recognition and the Internet of Things is characterized by: include: Collecting remote teaching interaction data through a distributed Internet of Things data collection system, wherein the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on the student terminal. The remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; Constructing a multimodal feature fusion network to process the distance learning interaction data, the multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the time series analysis subnetwork analyzes the temporal variation patterns of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vectors with the environmental data to obtain student interaction status features; A distance learning interaction evaluation model is constructed based on the student interaction status characteristics. The distance learning interaction evaluation model calculates the student attention level value and classroom participation value in real time, including: Receive a student interaction state feature sequence within a ten-second time window and input the student interaction state feature sequence into a temporal convolution module. The temporal convolution module extracts multi-scale temporal dependency features using different convolution kernel sizes. The multi-scale temporal dependency features are passed through a temporal importance evaluation unit to calculate the attention weight of each time step to obtain micro-evaluation features. Inputting the micro-evaluation features into a hierarchical long short-term memory network, wherein the bottom network of the hierarchical long short-term memory network processes the micro-evaluation features to generate intermediate attention features, and the top network of the hierarchical long short-term memory network integrates the intermediate attention features to obtain long-term change features; Inputting the long-term variation features into a multi-task learning framework, the multi-task learning framework extracts universal interaction features through a multi-head self-attention mechanism, and the universal interaction features are respectively input into an attention level evaluation branch and a class engagement evaluation branch, wherein the attention level evaluation branch includes a spiking neural network module, and the class engagement evaluation branch includes a temporal reasoning module of a graph attention network; A learning scenario recognition module is constructed based on the course type and teaching activity form. The output of the learning scenario recognition module is passed through a scenario adaptation layer to obtain feature weights and evaluation thresholds. The feature weights and evaluation thresholds are input into a Bayesian neural network, which uses Monte Carlo dropout to quantify the uncertainty of model predictions and obtain a reliability index. The output result of the attention level evaluation branch is used to obtain an attention level value through a nonlinear mapping function, and the output result of the classroom participation evaluation branch is used to obtain a classroom participation value through a cumulative probability distribution function. The attention level value and the classroom participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result; based on the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, thereby realizing dynamic optimization and adjustment of the remote teaching process.

2. The method according to claim 1, characterized in that The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the time series analysis subnetwork analyzes the temporal variation patterns of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vectors with the environmental data to obtain student interaction status features including: Behavioral data features, audio data features, and video data features are extracted separately through a parallel three-way feature extraction architecture. The behavioral data features are extracted by using a multi-scale convolution layer and a residual connection to extract local features of the facial area, and the head posture data is processed by a dual-stream attention module. The output result of the dual-stream attention module is adaptively weighted by a feature recalibration module to obtain a behavioral feature vector. The audio data features are extracted by obtaining a time-frequency spectrum through a short-time Fourier transform, and a multi-scale one-dimensional convolution is used to extract the local time-frequency features of the time-frequency spectrum. The local time-frequency features are processed by a channel attention mechanism to obtain an audio feature vector. The video data features are extracted by using a spatiotemporal convolution module to process a sixteen-frame image sequence to obtain a video feature vector. Inputting the behavior feature vector, the audio feature vector, and the video feature vector into a bidirectional long short-term memory network for temporal analysis, wherein the bidirectional long short-term memory network includes a modality-specific forget gate mechanism, wherein the modality-specific forget gate mechanism uses a first forget gate threshold for the behavior feature vector and a second forget gate threshold for the audio feature vector and the video feature vector, wherein the first forget gate threshold is greater than the second forget gate threshold; and processing the output features of the bidirectional long short-term memory network through a temporal attention mechanism to obtain temporal features; The temporal features are input into the attention fusion subnetwork, which includes an intra-modal attention module and an inter-modal attention module. The intra-modal attention module calculates the importance weights of features at different time steps within the same modality, and the inter-modal attention module learns the complementary relationship between features of different modalities. The environmental data is mapped to the feature vector dimensional space through a multi-layer perceptron to obtain environmental features, and a conditional attention mechanism is constructed based on the environmental features to guide the feature fusion process. The output features of the attention fusion subnetwork are enhanced through residual connections and self-attention mechanisms to obtain student interaction status features.

3. The method according to claim 1, characterized in that The method further includes determining an attention evaluation value by a spiking neural network module: Normalizing the input universal interaction features to obtain normalized features, mapping the normalized features to the time domain via a logarithmic function to generate a time-series pulse pattern, inputting the time-series pulse pattern into a first neuron layer having 256 integrating and releasing neurons, the integrating and releasing neurons of the first neuron layer converting input current into postsynaptic current via synaptic weights, and the synaptic weights are dynamically adjusted via a spike-timing-dependent plasticity learning rule; Inputting the output signal of the first neuron layer into a second neuron layer having one hundred and twenty-eight leaky integral-sparking neurons, constructing lateral inhibitory connections between the leaky integral-sparking neurons in the second neuron layer to form a local competition mechanism, wherein the inhibitory strength of the lateral inhibitory connections decreases as the distance between adjacent neurons increases; Inputting the output signal of the second neuron layer into a third neuron layer having sixty-four adaptive leaky integrate-and-send neurons, wherein the firing thresholds of the adaptive leaky integrate-and-send neurons in the third neuron layer are adaptively adjusted according to the neuron activity history; Establishing a multi-time-scale membrane potential dynamics model for each adaptive leaky integrate-and-spark neuron in the third neuronal layer, wherein the multi-time-scale membrane potential dynamics model includes a fast synaptic current with a time constant of milliseconds, a slow modulation current with a time constant of hundreds of milliseconds, and an adaptation current with a time constant of seconds; Calculate the group average firing rate of the third neuron layer within the time window, map the group average firing rate to the interval of zero to one through the sigmoid function to obtain an initial attention value, and perform an exponential sliding average on the initial attention value to obtain a final attention evaluation value, wherein the size of the time window dynamically shrinks as the group average firing rate increases.

4. The method according to claim 1, wherein The method further includes determining an engagement representation vector by a temporal reasoning module of a graph attention network: Mapping the input universal interaction feature sequence to a graph structure, where the feature vector of each time step corresponds to a node, and the feature dimension of the node is 512. Eight attention heads are used to calculate the edge weights between nodes. The attention heads perform linear transformations on the node features to obtain query vectors and key vectors. Attention coefficients are calculated based on the query vectors and the key vectors, and the outputs of the eight attention heads are concatenated to obtain an edge weight matrix. A three-layer graph attention network is constructed. In the first layer, each node is connected to the nodes within two adjacent time steps and relative position encoding is introduced. In the second layer, the connection range is expanded to the adjacent five time steps and features are extracted through multi-scale convolution operations. In the third layer, global connections are achieved and long-range dependency features are extracted through the transformer structure. Performing spectral clustering on the node features output by the three-layer graph attention network to obtain multiple node clusters, performing a maximum pooling operation on each of the node clusters to obtain node cluster features, calculating the importance weight of each of the node cluster features, and weighting the node cluster features based on the corresponding importance weights to obtain an engagement representation vector; The method further comprises: A gated graph neural network is used to update the node information in the graph structure. The degree of retention of historical information is calculated through the update gate, and the integration ratio of new information is calculated through the reset gate. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node to obtain the updated node state.

5. The method according to claim 1, wherein Generate a personalized teaching guidance strategy based on the calculation results of the remote teaching interaction evaluation model. The personalized teaching guidance strategy includes suggestions for adjusting the teaching rhythm and optimizing the interaction mode, including: A sliding time window is constructed to perform time series analysis on the attention level and classroom participation output by the distance learning interactive evaluation model. The sliding time window has a window size of ten minutes and slides every thirty seconds. The mean, standard deviation, rate of change, and periodic pattern of the attention level and classroom participation within the sliding time window are calculated to obtain the time series variation characteristics of the attention level and the dynamic characteristics of the classroom participation. performing cognitive load analysis based on the temporal variation characteristics of the attention level, detecting the trend of attention level changes within three consecutive sliding time windows, triggering rhythm adjustment when the attention level drops by more than 20%, estimating the cognitive load cycle based on the cyclical pattern of the attention level, and dividing the cognitive load cycle into a cognitively active period and a cognitively fatigued period; Based on the division results of the cognitive active period and the cognitive fatigue period, an interaction strategy is generated in combination with the dynamic characteristics of the class participation. The optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period. A mapping matrix between the interaction form and the effect of improving the class participation is established. The interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages. Constructing an environmental feature vector, performing a multivariate regression analysis on the temporal variation characteristics of the attention level and the dynamic characteristics of the classroom participation with the environmental feature vector, quantifying the influence weight of each environmental factor in the environmental feature vector, and making environmental adaptability corrections to the interaction depth adjustment plan based on the influence weights to obtain a personalized interaction strategy that includes environmental parameter thresholds; The execution effect of the personalized interactive strategy at different cognitive stages is input into a decision tree model. Based on the real-time feedback of the attention level and the classroom participation, the strategy generation rules of the decision tree model are continuously optimized through reinforcement learning to achieve dynamic adjustment and optimization of the teaching guidance strategy.

6. The method according to claim 5, characterized in that Based on the division results of the cognitive active period and the cognitive fatigue period, an interaction strategy is generated in combination with the dynamic characteristics of the class participation. The optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period. A mapping matrix between the interaction form and the effect of improving the class participation is established. The interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages, including: An exponential decay function was used to model the time series of classroom participation. Classroom participation data was collected within a five-minute sliding time window. The classroom participation decay parameter was obtained through least squares fitting, and a decay curve of classroom participation over time was established. Calculate the second-order derivative of the attenuation curve, detect the inflection point where the second-order derivative turns from negative to positive, verify the local minimum constraint within one minute before and after the inflection point, and determine the time difference between the inflection point that meets the constraint and the last interaction time as the optimal interaction interval; Construct a mapping matrix between interaction forms and effect indicators, where the effect indicators include the magnitude of improvement in class participation, the duration of improvement in class participation, and the interaction response rate. Assign a time-decay weight to the historical data of each effect indicator, and calculate the weighted average score of different interaction forms on each effect indicator. Setting different weight coefficients for the effect indicators based on the current cognitive stage, performing weighted calculation on the weight coefficients and the scores of the mapping matrix to obtain an interaction form score, and selecting the interaction form with the highest score as the interaction strategy for the current cognitive stage; Calculating the duration during which the class participation is maintained above the baseline level after each interaction, and inputting the difference ratio between the duration and the expected duration into the depth adjustment function to generate an interaction depth adjustment plan; Setting constraints on the interaction depth adjustment scheme to limit the change in interaction depth between two consecutive times to no more than 30%, lowering the upper limit of the depth adjustment to 15% during cognitive fatigue, and resetting to the baseline depth after three consecutive adjustments are ineffective; Calculate the weighted combination score of the improvement in classroom participation, classroom participation duration, and interactive response rate for each interaction, and update the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function based on the weighted combination score to achieve dynamic optimization of interactive strategy generation.

7. An intelligent classroom teaching optimization system combining behavior recognition and the Internet of Things, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is configured to collect remote teaching interaction data through a distributed Internet of Things data collection system. The distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal. The remote teaching interaction data includes student behavior data, audio data, video data, and environmental data. The behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, including lighting intensity data, background noise data, and network status data. The second unit is used to construct a multimodal feature fusion network to process the remote teaching interaction data. The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork. The feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively. The time series analysis subnetwork analyzes the time series variation patterns of each feature vector based on the long short-term memory network structure. The attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain student interaction status features. The third unit is used to construct a remote teaching interaction evaluation model based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and classroom participation value in real time; generates a personalized teaching guidance strategy based on the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, thereby realizing dynamic optimization and adjustment of the remote teaching process.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images