Intelligent classroom teaching optimization method and system combining behavior recognition and Internet of Things

Abstract

The invention provides an intelligent classroom teaching optimization method and system combining behavior recognition and the Internet of Things, and relates to the technical field of smart classs.The method comprises the steps that student behavior data, audio and video data and environment data are acquired through a distributed Internet of Things data acquisition system deployed at a student terminal; processing the data by adopting a multi-modal feature fusion network to obtain student interaction state features; and constructing a remote teaching interaction evaluation model based on the features, calculating the attention level and the classroom participation degree of the students in real time, and generating a personalized teaching guidance strategy. According to the method, intelligent sensing of student states and dynamic optimization of teaching strategies in the remote teaching process are realized.

Description

Technical Field

[0001] The present invention relates to smart classroom technology, and in particular to a smart classroom teaching optimization method and system combining behavior recognition and the Internet of Things. Background Art

[0002] With the rapid development of Internet technology, distance learning has become an important part of the education field. In the process of distance learning, it is difficult for teachers to intuitively perceive students' learning status and classroom participation, which has a significant impact on the quality and effectiveness of teaching. At present, the distance learning platform mainly collects students' video data through cameras and analyzes students' online learning status based on computer vision technology. At the same time, by collecting students' operation data on the online learning platform, the learning behavior of students is tracked and analyzed to provide teachers with teaching feedback.

[0003] However, the existing distance learning optimization methods have the following shortcomings: First, the existing methods mainly rely on single video data or operation data for analysis, lack comprehensive analysis of student behavior data, audio data and environmental data, and cannot comprehensively and accurately evaluate students' actual learning status. Secondly, the existing methods usually adopt static feature extraction methods, fail to fully consider the temporal change characteristics of student status, resulting in the lack of temporal consistency in the evaluation results of students' attention level and classroom participation. Thirdly, the existing methods often use fixed rule models when generating teaching guidance strategies, fail to make adaptive adjustments based on individual differences and environmental factors of different students, and are difficult to provide teachers with personalized teaching optimization suggestions.

[0004] In view of the above problems, the present invention proposes an intelligent classroom teaching optimization method combining behavior recognition and the Internet of Things. Through multimodal data collection and fusion analysis, intelligent perception and dynamic optimization of the distance teaching process are realized. Summary of the invention

[0005] The embodiments of the present invention provide a smart classroom teaching optimization method and system combining behavior recognition and the Internet of Things, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides a smart classroom teaching optimization method that combines behavior recognition and the Internet of Things, including:

[0008] The remote teaching interaction data is collected through a distributed Internet of Things data collection system, wherein the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal, and the remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, and the environmental data includes lighting intensity data, background noise data, and network status data;

[0009] Construct a multimodal feature fusion network to process the distance teaching interactive data, the multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from the student behavior data, audio data and video data respectively, the time series analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction state feature;

[0010] A remote teaching interaction evaluation model is constructed based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student's attention level value and classroom participation value in real time; according to the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

[0011] The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from student behavior data, audio data and video data respectively, the time series analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction status features including:

[0012] The behavioral data features, audio data features and video data features are extracted respectively through a parallel three-way feature extraction architecture, wherein the behavioral data features are extracted by using a multi-scale convolution layer and a residual connection to extract local features of the facial area, and the head posture data is processed by a dual-stream attention module, and the output result of the dual-stream attention module is adaptively weighted by a feature recalibration module to obtain a behavioral feature vector; the audio data features are extracted by obtaining a time-frequency spectrum through a short-time Fourier transform, and a multi-scale one-dimensional convolution is used to extract the local time-frequency features of the time-frequency spectrum, and the local time-frequency features are processed by a channel attention mechanism to obtain an audio feature vector; the video data features are extracted by using a spatiotemporal convolution module to process a sixteen-frame image sequence to obtain a video feature vector;

[0013] Inputting the behavior feature vector, the audio feature vector and the video feature vector into a bidirectional long short-term memory network for time series analysis, wherein the bidirectional long short-term memory network includes a modality-specific forget gate mechanism, wherein the modality-specific forget gate mechanism uses a first forget gate threshold for the behavior feature vector and a second forget gate threshold for the audio feature vector and the video feature vector, wherein the first forget gate threshold is greater than the second forget gate threshold; the output features of the bidirectional long short-term memory network are processed by a temporal attention mechanism to obtain a time series feature;

[0014] The temporal features are input into the attention fusion subnetwork, which includes an intra-modal attention module and an inter-modal attention module. The intra-modal attention module calculates the importance weights of features at different time steps in the same modality, and the inter-modal attention module learns the complementary relationship between features of different modalities. The environmental data is mapped to the feature vector dimensional space through a multi-layer perceptron to obtain environmental features, and a conditional attention mechanism is constructed based on the environmental features to guide the feature fusion process. The output features of the attention fusion subnetwork are enhanced through residual connections and self-attention mechanisms to obtain student interaction status features.

[0015] A distance teaching interaction evaluation model is constructed based on the student interaction status characteristics. The distance teaching interaction evaluation model calculates the student attention level value and the classroom participation value in real time, including:

[0016] Receive a student interaction state feature sequence within a ten-second time window, input the student interaction state feature sequence into a temporal convolution module, the temporal convolution module extracts multi-scale temporal dependency features through different convolution kernel sizes, and the multi-scale temporal dependency features are passed through a temporal importance evaluation unit to calculate the attention weight of each time step to obtain a micro-evaluation feature;

[0017] The micro-evaluation features are input into a hierarchical long short-term memory network, the bottom network of the hierarchical long short-term memory network processes the micro-evaluation features to generate intermediate attention features, and the top network of the hierarchical long short-term memory network integrates the intermediate attention features to obtain long-term change features;

[0018] Inputting the long-term change features into a multi-task learning framework, the multi-task learning framework extracts universal interaction features through a multi-head self-attention mechanism, and the universal interaction features are respectively input into an attention level evaluation branch and a class participation evaluation branch, wherein the attention level evaluation branch includes a pulse neural network module, and the class participation evaluation branch includes a temporal reasoning module of a graph attention network;

[0019] A learning scenario recognition module is constructed based on the course type and teaching activity form. The output result of the learning scenario recognition module is passed through the scenario adaptation layer to obtain feature weights and evaluation thresholds. The feature weights and evaluation thresholds are input into a Bayesian neural network. The Bayesian neural network uses Monte Carlo dropout to quantify the uncertainty of model prediction and obtain a reliability index.

[0020] The output result of the attention level evaluation branch is used through a nonlinear mapping function to obtain an attention level value, and the output result of the class participation evaluation branch is used through a cumulative probability distribution function to obtain a class participation value. The attention level value and the class participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result.

[0021] The method also includes determining an attention evaluation value by a spiking neural network module:

[0022] The input universal interaction feature is normalized to obtain a normalized feature, the normalized feature is mapped to the time domain through a logarithmic function to generate a timing pulse pattern, the timing pulse pattern is input into a first neuron layer having 256 integral-release neurons, the integral-release neurons of the first neuron layer convert the input current into a postsynaptic current through synaptic weights, and the synaptic weights are dynamically adjusted through a spike-time-dependent plasticity learning rule;

[0023] Inputting the output signal of the first neuron layer into a second neuron layer having one hundred and twenty-eight leaky integral-discharge neurons, constructing lateral inhibitory connections between the leaky integral-discharge neurons in the second neuron layer to form a local competition mechanism, wherein the inhibitory strength of the lateral inhibitory connections decreases as the distance between adjacent neurons increases;

[0024] Inputting the output signal of the second neuron layer into a third neuron layer having sixty-four adaptive leaky integral-release neurons, wherein the release thresholds of the adaptive leaky integral-release neurons of the third neuron layer are adaptively adjusted according to the neuron activity history;

[0025] Establishing a multi-time-scale membrane potential dynamics model for each adaptive leaky integral-sparking neuron in the third neuron layer, wherein the multi-time-scale membrane potential dynamics model includes a fast synaptic current with a time constant of milliseconds, a slow modulation current with a time constant of hundreds of milliseconds, and an adaptation current with a time constant of seconds;

[0026] Calculate the group average firing rate of the third neuron layer within the time window, map the group average firing rate to the interval of zero to one through a sigmoid function to obtain an initial attention value, and perform exponential sliding average on the initial attention value to obtain a final attention evaluation value, wherein the size of the time window dynamically shrinks as the group average firing rate increases.

[0027] The method also includes determining an engagement representation vector by a temporal reasoning module of a graph attention network:

[0028] Mapping the input universal interactive feature sequence to a graph structure, wherein the feature vector of each time step in the graph structure corresponds to a node, and the feature dimension of the node is 512 dimensions, calculating the edge weights between nodes through eight attention heads, the attention heads linearly transform the node features to obtain a query vector and a key vector, calculating the attention coefficient based on the query vector and the key vector, and concatenating the outputs of the eight attention heads to obtain an edge weight matrix;

[0029] Construct a three-layer graph attention network. In the first layer of the graph attention network, each node is connected to the nodes in the adjacent positive and negative two time steps and relative position encoding is introduced. In the second layer of the graph attention network, the connection range is expanded to the adjacent positive and negative five time steps and features are extracted through multi-scale convolution operations. In the third layer of the graph attention network, global connection is achieved and long-range dependency features are extracted through the transformer structure.

[0030] Performing spectral clustering on the node features output by the three-layer graph attention network to obtain multiple node clusters, performing a maximum pooling operation on each of the node clusters to obtain node cluster features, calculating the importance weight of each of the node cluster features, and weighting the node cluster features based on the corresponding importance weights to obtain a participation representation vector;

[0031] The method further comprises:

[0032] A gated graph neural network is used to update the node information in the graph structure. The retention degree of historical information is calculated through the update gate, and the integration ratio of new information is calculated through the reset gate. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node to obtain the updated node state.

[0033] According to the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions, including:

[0034] Construct a sliding time window to perform time series analysis on the attention level and class participation output by the distance teaching interactive evaluation model. The window size of the sliding time window is ten minutes and slides once every thirty seconds. Calculate the mean, standard deviation, change rate and periodic pattern of the attention level and class participation in the sliding time window to obtain the time series change characteristics of the attention level and the dynamic characteristics of the class participation.

[0035] performing cognitive load analysis according to the time series variation characteristics of the attention level, detecting the variation trend of the attention level within three consecutive sliding time windows, triggering rhythm adjustment when the attention level drops by more than 20%, estimating the cognitive load cycle based on the periodic pattern of the attention level, and dividing the cognitive load cycle into a cognitive active period and a cognitive fatigue period;

[0036] Based on the division results of the cognitive active period and the cognitive fatigue period, the interactive strategy is generated in combination with the dynamic characteristics of the class participation, the optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, a mapping matrix between the interactive form and the effect of improving the class participation is established, and the interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages;

[0037] Constructing an environmental feature vector, performing a multivariate regression analysis on the temporal variation characteristics of the attention level and the dynamic characteristics of the classroom participation and the environmental feature vector, quantifying the influence weight of each environmental factor in the environmental feature vector, and making environmental adaptability corrections to the interaction depth adjustment scheme based on the influence weight, so as to obtain a personalized interaction strategy including an environmental parameter threshold;

[0038] The execution effect of the personalized interactive strategy at different cognitive stages is input into a decision tree model. Based on the real-time feedback of the attention level and the classroom participation, the strategy generation rules of the decision tree model are continuously optimized through reinforcement learning to achieve dynamic adjustment and optimization of the teaching guidance strategy.

[0039] Based on the division results of the cognitive active period and the cognitive fatigue period, the interactive strategy is generated in combination with the dynamic characteristics of the class participation, the optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, and a mapping matrix between the interactive form and the class participation improvement effect is established. The interactive depth adjustment plan is determined according to the duration of the class participation in different cognitive stages, including:

[0040] An exponential decay function is used to model the time series of classroom participation. Classroom participation data is collected within a five-minute sliding time window. The classroom participation decay parameter is obtained through least squares fitting, and a decay curve of classroom participation over time is established.

[0041] Calculate the second-order derivative of the attenuation curve, detect the inflection point where the second-order derivative turns from negative to positive, verify the local minimum constraint condition within one minute before and after the inflection point, and determine the time difference between the inflection point moment that meets the constraint condition and the last interaction moment as the optimal interaction interval time;

[0042] Construct a mapping matrix between interaction forms and effect indicators, wherein the effect indicators include the degree of improvement in class participation, the duration of improvement in class participation, and the interactive response rate, assign a time decay weight to the historical data of each of the effect indicators, and calculate the weighted average score of different interaction forms on each of the effect indicators;

[0043] Different weight coefficients are set for the effect indicators based on the current cognitive stage, the weight coefficients are weighted and calculated with the scores of the mapping matrix to obtain the interaction form scores, and the interaction form with the highest score is selected as the interaction strategy for the current cognitive stage;

[0044] Calculate the duration that the class participation is maintained above the baseline level after each interaction, input the difference ratio between the duration and the expected duration into the depth adjustment function, and generate an interaction depth adjustment plan;

[0045] Setting constraints on the interaction depth adjustment scheme to limit the change in interaction depth between two consecutive interactions to no more than 30%, reducing the upper limit of the depth adjustment to 15% during cognitive fatigue, and resetting to the baseline depth when three consecutive adjustments are invalid;

[0046] The weighted combined score of the improvement in class participation, class participation duration, and interaction response rate for each interaction is calculated, and the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function are updated according to the weighted combined score to achieve dynamic optimization of interaction strategy generation.

[0047] According to a second aspect of the embodiments of the present invention,

[0048] Provides an intelligent classroom teaching optimization system that combines behavior recognition and the Internet of Things, including:

[0049] The first unit is used to collect remote teaching interaction data through a distributed Internet of Things data collection system, wherein the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal, and the remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, wherein the environmental data includes lighting intensity data, background noise data, and network status data;

[0050] The second unit is used to construct a multimodal feature fusion network to process the remote teaching interaction data, and the multimodal feature fusion network includes a feature extraction subnetwork, a timing analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from the student behavior data, audio data and video data respectively, the timing analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction state feature;

[0051] The third unit is used to construct a remote teaching interaction evaluation model based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student's attention level value and classroom participation value in real time; generates a personalized teaching guidance strategy according to the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

[0052] According to a third aspect of the embodiments of the present invention,

[0053] An electronic device is provided, comprising:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] A fourth aspect of the embodiments of the present invention is:

[0058] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0059] The beneficial effects of this application are as follows:

[0060] 1. Improve the accuracy and comprehensiveness of teaching evaluation: This technology collects multimodal data (behavior, audio, video, environment) and uses a multimodal feature fusion network for analysis, which can more comprehensively and accurately evaluate students' learning status, including attention level and classroom participation. Compared with traditional single-dimensional evaluation methods (such as relying solely on classroom performance or homework grades), this method can capture more subtle emotional and behavioral changes in students, thereby more accurately reflecting students' actual learning situation.

[0061] 2. Personalized teaching guidance: The distance teaching interaction evaluation model built based on the characteristics of students' interactive status can calculate students' attention level and participation in real time, and generate personalized teaching guidance strategies, such as suggestions for adjusting teaching rhythm and optimizing interactive methods. This personalized guidance can help teachers better improve the pertinence and effectiveness of teaching.

[0062] 3. Optimize the remote teaching experience and improve teaching efficiency: The personalized teaching guidance strategy can be fed back to teachers in real time through the IoT communication network, which can help teachers adjust teaching methods and rhythm in time, enhance classroom interaction, and improve students' learning interest and enthusiasm. At the same time, this technology can also help teachers better control remote classrooms, optimize remote teaching experience, and ultimately improve teaching efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flow chart of a smart classroom teaching optimization method combining behavior recognition and the Internet of Things according to an embodiment of the present invention;

[0064] Figure 2 This is a structural diagram of an intelligent classroom teaching optimization system that combines behavior recognition and the Internet of Things according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0067] Figure 1 FIG. 1 is a flow chart of an intelligent classroom teaching optimization method combining behavior recognition and the Internet of Things according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0068] S101. Collecting remote teaching interaction data through a distributed Internet of Things data collection system, the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module and an environmental data collection module deployed on a student terminal, the remote teaching interaction data includes student behavior data, audio data, video data and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains audio data and video data of students, and the environmental data collection module obtains environmental data of the environment in which the students are located, and the environmental data includes lighting intensity data, background noise data, and network status data;

[0069] S102. Construct a multimodal feature fusion network to process the remote teaching interactive data, wherein the multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the time series analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain student interaction status features;

[0070] S103. A remote teaching interaction evaluation model is constructed based on the characteristics of the student interaction status, and the remote teaching interaction evaluation model calculates the student's attention level and classroom participation value in real time; a personalized teaching guidance strategy is generated according to the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes suggestions for adjusting the teaching rhythm and optimizing the interaction method; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

[0071] First, a distributed IoT data collection system is deployed on the student terminals. The system consists of three modules: behavior data collection module, audio and video data collection module, and environment data collection module.

[0072] The behavior data collection module uses a camera to capture students' facial expressions and head postures. For example, through image recognition technology, it can identify whether students are frowning, smiling, looking up, lowering their heads, and other actions, and convert these data into feature data, such as the frequency of frowning, the duration of smiling, the angle of head deflection, etc. Suppose a student frowns 3 times in one minute, each time for an average of 0.5 seconds, and the average downward tilt of the head is 15 degrees, then these data will be recorded as part of the behavior data.

[0073] The audio and video data collection module uses microphones and cameras to collect students' voice and image information. For example, it can record audio features such as students' volume, speaking speed, and speaking duration, as well as video features such as whether students raise their hands and whether they interact with other students. Assuming that a student speaks twice in one minute, with an average volume of 70 decibels each time and a speaking speed of 150 words per minute, these data will be recorded as part of the audio data.

[0074] The environmental data collection module uses sensors to collect data such as light intensity, background noise, and network status of the student's environment. For example, the light sensor can record the brightness of the classroom, the noise sensor can record the noise level of the classroom, and the network status sensor can record the speed and stability of the student's network connection. Assuming that the brightness of the classroom is 500 lux, the noise level is 40 decibels, and the student's network delay is 20 milliseconds, then this data will be recorded as part of the environmental data.

[0075] Next, a multimodal feature fusion network is constructed to process the collected data. The network consists of three sub-networks: feature extraction sub-network, time series analysis sub-network and attention fusion sub-network.

[0076] The feature extraction subnetwork extracts feature vectors from behavioral data, audio data, and video data. For example, it extracts features such as frown frequency and smile duration from behavioral data, volume and speech speed from audio data, and hand-raising times and interaction frequency from video data.

[0077] The time series analysis subnetwork uses the long short-term memory (LSTM) network structure to analyze the time series change patterns of each feature vector. For example, it analyzes the changing trend of students' attention levels to determine whether they are gradually losing attention or gradually concentrating their attention. Assuming that a student's attention level is high in the first 5 minutes and then gradually decreases, the time series analysis subnetwork can capture this trend.

[0078] The attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the characteristics of the student interaction status. For example, the student's attention level, class participation, and environmental data such as classroom brightness, noise level, and network status are comprehensively considered to finally obtain a feature vector that can reflect the overall interaction status of the student.

[0079] Then, a distance learning interaction evaluation model is constructed based on the characteristics of the student interaction status. The model calculates the student's attention level and classroom participation value in real time. For example, based on the student's feature vector, the student's attention level is calculated to be 80 points and the classroom participation is 75 points.

[0080] Based on the calculation results of the evaluation model, a personalized teaching guidance strategy is generated. For example, if the student's attention level is low, the system will suggest that the teacher adjust the teaching rhythm, such as slowing down the speed of speech, increasing the interactive links, etc.; if the student's class participation is low, the system will suggest that the teacher optimize the interactive method, such as increasing questions and conducting group discussions, etc.

[0081] Finally, the personalized teaching guidance strategy is fed back to the teacher's terminal in real time through the IoT communication network to achieve dynamic optimization and adjustment of the remote teaching process. For example, guidance strategies such as "suggest slowing down the speed of speech" and "suggesting more questions" are sent to the teacher's computer or mobile phone in real time so that the teacher can adjust the teaching plan in time.

[0082] The beneficial effects of this method can be summarized in the following three aspects:

[0083] 1. Improve teaching effectiveness: By real-time monitoring of students' interaction status and providing personalized teaching guidance strategies, teachers can better understand students' learning situation and adjust teaching plans in time, thereby improving teaching effectiveness.

[0084] 2. Promote personalized learning: Based on the characteristics of students' interactive status, personalized learning suggestions and resources can be provided to students to help them better master the learning content.

[0085] 3. Optimize the allocation of teaching resources: By analyzing students’ learning data, it can help schools better understand students’ learning needs, optimize the allocation of teaching resources, and improve the utilization efficiency of teaching resources.

[0086] In an optional implementation, the multimodal feature fusion network includes a feature extraction subnetwork, a timing analysis subnetwork, and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors, and video feature vectors from student behavior data, audio data, and video data, respectively; the timing analysis subnetwork analyzes the temporal variation law of each feature vector based on the long short-term memory network structure; and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain student interaction status features including:

[0087] The behavioral data features, audio data features and video data features are extracted respectively through a parallel three-way feature extraction architecture, wherein the behavioral data features are extracted by using a multi-scale convolution layer and a residual connection to extract local features of the facial area, and the head posture data is processed by a dual-stream attention module, and the output result of the dual-stream attention module is adaptively weighted by a feature recalibration module to obtain a behavioral feature vector; the audio data features are extracted by obtaining a time-frequency spectrum through a short-time Fourier transform, and a multi-scale one-dimensional convolution is used to extract the local time-frequency features of the time-frequency spectrum, and the local time-frequency features are processed by a channel attention mechanism to obtain an audio feature vector; the video data features are extracted by using a spatiotemporal convolution module to process a sixteen-frame image sequence to obtain a video feature vector;

[0088] Inputting the behavior feature vector, the audio feature vector and the video feature vector into a bidirectional long short-term memory network for time series analysis, wherein the bidirectional long short-term memory network includes a modality-specific forget gate mechanism, wherein the modality-specific forget gate mechanism uses a first forget gate threshold for the behavior feature vector and a second forget gate threshold for the audio feature vector and the video feature vector, wherein the first forget gate threshold is greater than the second forget gate threshold; the output features of the bidirectional long short-term memory network are processed by a temporal attention mechanism to obtain a time series feature;

[0089] The temporal features are input into the attention fusion subnetwork, which includes an intra-modal attention module and an inter-modal attention module. The intra-modal attention module calculates the importance weights of features at different time steps in the same modality, and the inter-modal attention module learns the complementary relationship between features of different modalities. The environmental data is mapped to the feature vector dimensional space through a multi-layer perceptron to obtain environmental features, and a conditional attention mechanism is constructed based on the environmental features to guide the feature fusion process. The output features of the attention fusion subnetwork are enhanced through residual connections and self-attention mechanisms to obtain student interaction status features.

[0090] Obtain student behavior data, audio data, video data, and environmental data. Taking the classroom group discussion scene as an example, the behavior data uses the camera to capture the student's facial expressions and head posture, the audio data uses the microphone to record the student's voice, the video data uses the camera to record the student's body movements, and the environmental data includes the number of group members, classroom light intensity, noise level, etc. Assume that a set of data contains 4 students and lasts for 10 minutes.

[0091] First, multimodal features are extracted. For behavioral data, a ResNet-50 network is used to extract local features of the facial region. Head posture data (e.g., head orientation, nodding frequency) are fed into two separate fully connected layers to obtain two feature vectors. The two feature vectors are then passed through a feature recalibration module, which contains a fully connected layer and a Sigmoi d activation function to generate weights for the two feature vectors. The two weighted feature vectors are concatenated with the local features extracted by ResNet-50 to obtain a behavioral feature vector. For audio data, short-time Fourier transform is used to convert the audio data into a time-frequency spectrogram. Then, multiple one-dimensional convolutional layers with different convolution kernel sizes are used to extract local time-frequency features of the time-frequency spectrogram. These local features are then processed through a channel attention mechanism, which contains global average pooling, a fully connected layer, and a Sigmoi d activation function to generate weights for each channel to obtain an audio feature vector. For video data, a 3D convolutional network, such as a C3D network, is used to process a sequence of 16 consecutive frames of images to extract a video feature vector.

[0092] Next, the time series analysis is performed. The extracted behavioral feature vector, audio feature vector, and video feature vector are input into a bidirectional long short-term memory network (Bi-LSTM). The Bi-LSTM network contains a modality-specific forget gate mechanism. Specifically, for the behavioral feature vector, the first forget gate threshold is used; for the audio and video feature vectors, the second forget gate threshold is used. The first forget gate threshold is set to 0.7 and the second forget gate threshold is set to 0.5, which means that the redundant information in the behavioral features is more easily forgotten, while the information in the audio and video features is retained more. The output features of the Bi-LSTM are then processed by the temporal attention mechanism, which generates weights for each time step through a fully connected layer and a Softmax activation function to obtain the time series features.

[0093] Finally, feature fusion is performed. The temporal features are input into the attention fusion subnetwork. This subnetwork contains an intra-modal attention module and an inter-modal attention module. The intra-modal attention module uses a fully connected layer and a Softmax activation function to calculate the importance weights of features at different time steps within the same modality. The inter-modal attention module learns the complementary relationship between features of different modalities by calculating the similarity between feature vectors (for example, cosine similarity). The environmental data is mapped to the same dimensional space as the feature vector through a multi-layer perceptron to obtain environmental features. A conditional attention mechanism is constructed based on environmental features. For example, after concatenating environmental features with features of each modality, weights are generated through a fully connected layer and a Softmax activation function to guide the feature fusion process. The output features of the attention fusion subnetwork are then enhanced through residual connections and a self-attention mechanism to obtain the final student interaction status features. The self-attention mechanism uses scaled dot product attention to calculate the relationship between different dimensions of features and performs weighted fusion. The final student interaction status features can be used to judge student participation, enthusiasm, etc.

[0094] Beneficial effects:

[0095] 1. Improved the accuracy of student interaction status recognition: Through multimodal feature fusion, the various behaviors of students can be captured more comprehensively, thereby improving recognition accuracy. For example, it can more accurately identify whether students actively participate in discussions and understand the classroom content.

[0096] 2. Adaptive fusion of different modal features: The attention mechanism can dynamically adjust the weights of different modal features according to their importance, thereby achieving adaptive fusion. For example, in a group discussion, if the student's voice information is more important, the weight of the audio feature will be higher; if the student's body language is more important, the weight of the video feature will be higher.

[0097] 3. Enhanced model robustness: The modality-specific forget gate mechanism can filter out redundant information in different modality features, thereby enhancing the robustness of the model. For example, in a noisy classroom environment, the audio features may contain a lot of noise information, and the forget gate mechanism can effectively filter out this noise and retain important voice information.

[0098] In an optional implementation, a remote teaching interaction evaluation model is constructed based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student attention level value and the classroom participation value in real time, including:

[0099] Receive a student interaction state feature sequence within a ten-second time window, input the student interaction state feature sequence into a temporal convolution module, the temporal convolution module extracts multi-scale temporal dependency features through different convolution kernel sizes, and the multi-scale temporal dependency features are passed through a temporal importance evaluation unit to calculate the attention weight of each time step to obtain a micro-evaluation feature;

[0100] The micro-evaluation features are input into a hierarchical long short-term memory network, the bottom network of the hierarchical long short-term memory network processes the micro-evaluation features to generate intermediate attention features, and the top network of the hierarchical long short-term memory network integrates the intermediate attention features to obtain long-term change features;

[0101] Inputting the long-term change features into a multi-task learning framework, the multi-task learning framework extracts universal interaction features through a multi-head self-attention mechanism, and the universal interaction features are respectively input into an attention level evaluation branch and a class participation evaluation branch, wherein the attention level evaluation branch includes a pulse neural network module, and the class participation evaluation branch includes a temporal reasoning module of a graph attention network;

[0102] A learning scenario recognition module is constructed based on the course type and teaching activity form. The output result of the learning scenario recognition module is passed through the scenario adaptation layer to obtain feature weights and evaluation thresholds. The feature weights and evaluation thresholds are input into a Bayesian neural network. The Bayesian neural network uses Monte Carlo dropout to quantify the uncertainty of model prediction and obtain a reliability index.

[0103] The output result of the attention level evaluation branch is used through a nonlinear mapping function to obtain an attention level value, and the output result of the class participation evaluation branch is used through a cumulative probability distribution function to obtain a class participation value. The attention level value and the class participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result.

[0104] Get the student interaction state feature sequence in a ten-second time window. A time window contains multiple time steps, and each time step contains multiple features, such as the number of mouse clicks, the number of keyboard taps, the camera gaze duration, the voice volume, the frequency of head posture changes, etc. For example, in a time window containing 10 time steps, each time step records 5 student interaction state features, then the student interaction state feature sequence in this time window is a matrix with 10 rows and 5 columns.

[0105] Input the student interaction state feature sequence into the temporal convolution module. The temporal convolution module uses convolution kernels of different sizes to extract multi-scale temporal dependency features. For example, convolution kernels of sizes 3 and 5 are used to extract short-term and long-term temporal dependency features respectively. Assuming that the input feature sequence is a 10x5 matrix, a 3x5 convolution kernel is used for convolution operation with a step size of 1 to obtain an 8x1 feature vector; a 5x5 convolution kernel is used for convolution operation with a step size of 1 to obtain a 6x1 feature vector. These two feature vectors are concatenated to obtain a 14x1 feature vector, i.e., a multi-scale temporal dependency feature.

[0106] The multi-scale temporal dependency features are input into the temporal importance evaluation unit. The unit calculates the attention weights for each time step to obtain micro-evaluation features. For example, an attention mechanism, such as the Softmax function, can be used to calculate the weights for each time step. Assuming that the multi-scale temporal dependency feature is a 14x1 vector, after the Softmax function, a 14x1 weight vector is obtained. The weight vector is multiplied by the corresponding multi-scale temporal dependency feature to obtain the weighted multi-scale temporal dependency feature, i.e., the micro-evaluation feature.

[0107] The micro-assessment features are input into a hierarchical long short-term memory network. The bottom network of the network processes the micro-assessment features to generate intermediate attention features. The top network integrates the intermediate attention features to obtain long-term change features. For example, the bottom network can be composed of multiple LSTM units, each unit processes a part of the micro-assessment features; the top network then integrates the output of the bottom network, for example, using another LSTM unit, to obtain long-term change features. Assume that the micro-assessment feature is a 14x1 vector, and the bottom network contains two LSTM units, each unit processes 7 features and outputs two 7x1 vectors. The top network concatenates these two 7x1 vectors into a 14x1 vector, and then inputs it into an LSTM unit to obtain the final long-term change feature, for example, a 1x64 vector.

[0108] The long-term change features are input into the multi-task learning framework. The framework extracts universal interaction features through a multi-head self-attention mechanism. The universal interaction features are input into the attention level evaluation branch and the classroom participation evaluation branch respectively. For example, 8 self-attention heads are used, each head extracts different features, and then these features are concatenated to obtain universal interaction features. Assuming that the long-term change feature is a 1x64 vector, after the multi-head self-attention mechanism, a 1x512 universal interaction feature is obtained.

[0109] The attention level assessment branch contains a spiking neural network module. This module simulates the information processing method of biological neurons, processes the general interaction features, and outputs features related to the attention level. For example, the LIF (LeakyIntegrate-and-Fire) model can be used to build a spiking neural network. Assuming that the general interaction feature is a 1x512 vector, after passing through the spiking neural network, a 1x1 attention level feature is obtained.

[0110] The class participation evaluation branch includes the temporal reasoning module of the graph attention network. This module uses the graph structure to capture the relationship between students' interaction states, performs temporal reasoning, and outputs features related to class participation. For example, a graph can be constructed in which nodes represent students and edges represent the interaction relationship between students. Assuming that the general interaction feature is a 1x512 vector, after passing through the temporal reasoning module of the graph attention network, a 1x1 class participation feature is obtained.

[0111] The learning scene recognition module is constructed based on the course type and teaching activity form. The output of this module is passed through the scene adaptation layer to obtain the feature weight and evaluation threshold. For example, for a course of the "explain" type, the weight of the "ask questions" activity is higher; for a course of the "discussion" type, the weight of the "speak" activity is higher. Assuming that the learning scene recognition module recognizes that the current scene is the "ask questions" activity in a course of the "explain" type, the scene adaptation layer will output the corresponding feature weight and evaluation threshold.

[0112] The feature weights and evaluation thresholds are input into the Bayesian neural network. The network uses Monte Carlo dropout to quantify the uncertainty of model predictions and obtain reliability indicators. For example, multiple dropout operations are performed on the Bayesian neural network to obtain multiple prediction results, and then the variance of these results is calculated as a reliability indicator.

[0113] The output of the attention level evaluation branch is passed through a nonlinear mapping function to obtain the attention level value. For example, the Sigmoid function is used to map the attention level feature to between 0 and 1. Assuming that the attention level feature is 0.8, after the Sigmoid function, the attention level value is 0.69.

[0114] The output of the class participation evaluation branch is used through the cumulative probability distribution function to obtain the class participation value. For example, the cumulative probability distribution function of the normal distribution is used. Assuming that the class participation feature is 1.5, after the cumulative probability distribution function of the normal distribution, the class participation value is 0.93.

[0115] The attention level value and the classroom participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result. For example, the attention level value and the classroom participation value are weighted averaged according to the reliability index. Assuming that the attention level value is 0.69, the classroom participation value is 0.93, and the reliability index is 0.8, the final evaluation result is 0.69*(1-0.8)+0.93*0.8=0.882.

[0116] Beneficial effects:

[0117] 1. Improve evaluation accuracy: Through multi-scale temporal feature extraction, hierarchical attention modeling and multi-task learning, the model can more comprehensively capture the interaction status of students, thereby improving the accuracy of evaluation.

[0118] 2. Enhanced evaluation reliability: Through the learning scenario recognition module and Bayesian neural network, the model can adjust the evaluation strategy according to different learning scenarios and quantify the uncertainty of the prediction, thereby enhancing the reliability of the evaluation.

[0119] 3. Achieve personalized assessment: This model takes into account the course type, teaching activity form and individual differences of students, and can achieve more personalized assessment and provide support for personalized learning.

[0120] In an optional embodiment, the method further includes determining an attention evaluation value by a spiking neural network module:

[0121] The input universal interaction feature is normalized to obtain a normalized feature, the normalized feature is mapped to the time domain through a logarithmic function to generate a timing pulse pattern, the timing pulse pattern is input into a first neuron layer having 256 integral-release neurons, the integral-release neurons of the first neuron layer convert the input current into a postsynaptic current through synaptic weights, and the synaptic weights are dynamically adjusted through a spike-time-dependent plasticity learning rule;

[0122] Inputting the output signal of the first neuron layer into a second neuron layer having one hundred and twenty-eight leaky integral-discharge neurons, constructing lateral inhibitory connections between the leaky integral-discharge neurons in the second neuron layer to form a local competition mechanism, wherein the inhibitory strength of the lateral inhibitory connections decreases as the distance between adjacent neurons increases;

[0123] Inputting the output signal of the second neuron layer into a third neuron layer having sixty-four adaptive leaky integral-release neurons, wherein the release thresholds of the adaptive leaky integral-release neurons of the third neuron layer are adaptively adjusted according to the neuron activity history;

[0124] Establishing a multi-time-scale membrane potential dynamics model for each adaptive leaky integral-sparking neuron in the third neuron layer, wherein the multi-time-scale membrane potential dynamics model includes a fast synaptic current with a time constant of milliseconds, a slow modulation current with a time constant of hundreds of milliseconds, and an adaptation current with a time constant of seconds;

[0125] Calculate the group average firing rate of the third neuron layer within the time window, map the group average firing rate to the interval of zero to one through a sigmoid function to obtain an initial attention value, and perform exponential sliding average on the initial attention value to obtain a final attention evaluation value, wherein the size of the time window dynamically shrinks as the group average firing rate increases.

[0126] Get the input universal interactive features. Universal interactive features can be multimodal, for example, including image visual features, text semantic features, user behavior features, etc. A specific example is that in an e-commerce recommendation system, universal interactive features can include image feature vectors of users' historical browsing of products (for example, 2048-dimensional feature vectors extracted using ResNet50), text semantic feature vectors of product titles (for example, 768-dimensional feature vectors extracted using BERT), and user behavior features such as clicks, favorites, and purchases in the past week (for example, a 3-dimensional vector representing the number of clicks, favorites, and purchases, respectively).

[0127] Normalize the obtained common interaction features. There are many ways to perform normalization, such as minimum-maximum normalization, Z-score normalization, etc. The purpose is to scale the numerical range of features of different dimensions to a uniform interval, such as between 0 and 1, to avoid some features having too much impact on the results due to their large numerical range. Taking the above-mentioned e-commerce recommendation system as an example, the image feature vector, text semantic feature vector, and user behavior feature vector can be normalized by minimum-maximum normalization, and the numerical range of each feature vector can be scaled to between 0 and 1.

[0128] The normalized features are mapped to the time domain through a logarithmic function to generate a time series pulse pattern. The logarithmic function is used here to convert the feature value into the time interval of pulse emission. The larger the feature value, the shorter the corresponding pulse emission time interval, which means that the pulse emission frequency is higher. For example, the normalized feature value x can be converted to time t through the function -log(1-x), where the time unit can be milliseconds. Assuming that the normalized value of an image feature is 0.9, the corresponding pulse emission time interval is -log(1-0.9)≈2.3 milliseconds.

[0129] The generated timing pulse pattern is input to the first neuron layer. The first neuron layer contains 256 integration-and-release neurons. Each neuron receives the input pulse signal and converts it into a postsynaptic current. The strength of the postsynaptic current is determined by the synaptic weight. The synaptic weight is dynamically adjusted by the spike-time-dependent plasticity learning rule. For example, if the input pulse of a neuron and its own output pulse are close in time, the synaptic weight connecting the two neurons will be strengthened; conversely, if the input pulse and the output pulse are far apart in time, the synaptic weight will be weakened.

[0130] The output signal of the first neuron layer is input to the second neuron layer. The second neuron layer contains 128 leaky integral-discharge neurons. Lateral inhibitory connections are constructed between these neurons to form a local competition mechanism. The inhibitory strength of lateral inhibitory connections decreases as the distance between adjacent neurons increases. This means that the activity of one neuron will inhibit the activity of its surrounding neurons, and the closer the distance, the greater the inhibitory strength. This local competition mechanism can enhance the selectivity of neurons so that only the neurons with the strongest response can remain active.

[0131] The output signal of the second neuron layer is input to the third neuron layer. The third neuron layer contains 64 adaptive leaky integral firing neurons. The firing thresholds of these neurons are adaptively adjusted according to the history of neuron activity. For example, if a neuron frequently fires pulses in a short period of time, its firing threshold will increase, making it more difficult to fire pulses again; conversely, if a neuron does not fire pulses for a long time, its firing threshold will decrease, making it easier to fire pulses. This adaptive mechanism can prevent neurons from being overly excited or inhibited.

[0132] A multi-time scale membrane potential dynamics model is established for each neuron in the third neuron layer. The model includes fast synaptic currents with a time constant of milliseconds, slow modulation currents with a time constant of hundreds of milliseconds, and adaptation currents with a time constant of seconds. These three currents work together to determine the changes in the membrane potential of the neuron, thereby affecting the firing behavior of the neuron.

[0133] Calculate the group average firing rate of the third neuron layer in the time window. For example, you can count how many pulses the 64 neurons fired in a 100 millisecond time window, and then divide it by the number of neurons and the time window size to get the group average firing rate.

[0134] The initial attention value is obtained by mapping the group average firing rate to the range of 0 to 1 through the sigmoid function. The sigmoid function is an S-shaped function that can map any value between 0 and 1.

[0135] The final attention evaluation value is obtained by performing an exponential sliding average on the initial attention value. Exponential sliding average is a commonly used smoothing method that can reduce the fluctuation of attention value. The size of the time window is dynamically reduced as the average firing rate of the group increases. This means that when the neuron activity increases, the time window will become smaller, so that it can respond to changes more quickly.

[0136] Beneficial effects:

[0137] 1. Improve the accuracy of attention assessment: By simulating the information processing mechanism of biological neurons, multimodal features can be extracted and integrated more effectively, thereby improving the accuracy of attention assessment.

[0138] 2. Enhance the robustness of the model: The multi-time scale membrane potential dynamics model and adaptive neuron mechanism make the model more robust to noise and input changes.

[0139] 3. Improve computing efficiency: The event-driven characteristics of pulse neural networks can reduce the amount of calculation and improve computing efficiency, which is especially suitable for application scenarios with high real-time requirements.

[0140] In an optional embodiment, the method further includes determining the engagement representation vector through a temporal reasoning module of a graph attention network:

[0141] Mapping the input universal interactive feature sequence to a graph structure, wherein the feature vector of each time step in the graph structure corresponds to a node, and the feature dimension of the node is 512 dimensions, calculating the edge weights between nodes through eight attention heads, the attention heads linearly transform the node features to obtain a query vector and a key vector, calculating the attention coefficient based on the query vector and the key vector, and concatenating the outputs of the eight attention heads to obtain an edge weight matrix;

[0142] Construct a three-layer graph attention network. In the first layer of the graph attention network, each node is connected to the nodes in the adjacent positive and negative two time steps and relative position encoding is introduced. In the second layer of the graph attention network, the connection range is expanded to the adjacent positive and negative five time steps and features are extracted through multi-scale convolution operations. In the third layer of the graph attention network, global connection is achieved and long-range dependency features are extracted through the transformer structure.

[0143] Performing spectral clustering on the node features output by the three-layer graph attention network to obtain multiple node clusters, performing a maximum pooling operation on each of the node clusters to obtain node cluster features, calculating the importance weight of each of the node cluster features, and weighting the node cluster features based on the corresponding importance weights to obtain a participation representation vector;

[0144] The method further comprises:

[0145] A gated graph neural network is used to update the node information in the graph structure. The retention degree of historical information is calculated through the update gate, and the integration ratio of new information is calculated through the reset gate. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node to obtain the updated node state.

[0146] Get a sequence containing common user interaction features, such as the user's viewing history on a certain video platform, including viewing time, likes, comments, sharing, etc. Assume that the sequence contains 10 time steps, and the feature dimension of each time step is 512, which can be represented as a 10x512 matrix.

[0147] Map the input common interaction feature sequence to a graph structure. The feature vector of each time step corresponds to a node in the graph, and the feature dimension of the node is 512. In this example, 10 nodes will be obtained, each node represents the interaction feature of a time step.

[0148] Calculate the edge weights between nodes. Use 8 attention heads to calculate the edge weights between nodes. Each attention head performs a linear transformation on the node features to obtain a query vector and a key vector. Taking the first attention head as an example, the 512-dimensional feature vector of each node is linearly transformed into a query vector and a key vector respectively. Then, the attention coefficient is calculated based on the query vector and the key vector to reflect the degree of association between the nodes. The outputs of the 8 attention heads are concatenated to obtain an edge weight matrix with a dimension of 10x10, which represents the connection strength between any two nodes.

[0149] Construct a three-layer graph attention network. In the first layer of the graph attention network, each node is connected to the nodes in the adjacent positive and negative two time steps, and relative position encoding is introduced to distinguish the relative position of the node in the sequence. For example, the third node is connected to the 1st, 2nd, 4th, and 5th nodes. In the second layer of the graph attention network, the connection range is expanded to the adjacent positive and negative five time steps, and features are extracted through multi-scale convolution operations. For example, the third node is connected to the 1st, 2nd, 4th, 5th, 6th, 7th, and 8th nodes, and the convolution kernel sizes are 3, 5, and 7 respectively to capture temporal information of different scales. In the third layer of the graph attention network, global connection is achieved, each node is connected to all nodes, and long-range dependency features are extracted through the Transformer structure to capture global information.

[0150] Perform spectral clustering on the node features output by the three-layer graph attention network. Assume that spectral clustering divides 10 nodes into 3 clusters.

[0151] Perform a maximum pooling operation on each node cluster to obtain the node cluster features. For example, perform a maximum pooling operation on the node features in the first cluster to obtain a 512-dimensional cluster feature vector.

[0152] Calculate the importance weight of each node cluster feature. For example, calculate the importance weight of the cluster based on the number of nodes in each cluster and the variance of the node features. Suppose the importance weights of the three clusters are 0.2, 0.3, and 0.5 respectively.

[0153] The node cluster features are weighted based on the corresponding importance weights to obtain the participation representation vector. The feature vectors of the three clusters are multiplied by the corresponding weights and then added together to obtain the final 512-dimensional participation representation vector.

[0154] A gated graph neural network is used to update the node information in the graph structure. Taking the first node as an example, the degree of retention of historical information is calculated through the update gate, and the proportion of integration of new information is calculated through the reset gate. Assume that the output value of the update gate is 0.8 and the output value of the reset gate is 0.2. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node (512-dimensional feature vector) to obtain the updated node state. Similar update operations are performed on all nodes.

[0155] Beneficial effects:

[0156] 1. Improved the accuracy of the model's prediction of user engagement. By combining the graph attention network and the gated graph neural network, the model can more effectively capture the complex temporal dependencies and association information between nodes in the user interaction sequence, thereby improving the accuracy of the prediction.

[0157] 2. Enhanced model interpretability. By dividing nodes into different clusters through spectral clustering and calculating the importance weight of each cluster, we can explain the contribution of different types of user behaviors to engagement and improve the interpretability of the model.

[0158] 3. Improved the generalization ability of the model. The combination of multi-scale convolution and Transformer structure enables the model to capture temporal information and long-range dependencies of different scales, enhancing the generalization ability of the model and enabling it to adapt to different user behavior patterns.

[0159] In an optional implementation, a personalized teaching guidance strategy is generated based on the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions including:

[0160] Construct a sliding time window to perform time series analysis on the attention level and class participation output by the distance teaching interactive evaluation model. The window size of the sliding time window is ten minutes and slides once every thirty seconds. Calculate the mean, standard deviation, change rate and periodic pattern of the attention level and class participation in the sliding time window to obtain the time series change characteristics of the attention level and the dynamic characteristics of the class participation.

[0161] performing cognitive load analysis according to the time series variation characteristics of the attention level, detecting the variation trend of the attention level within three consecutive sliding time windows, triggering rhythm adjustment when the attention level drops by more than 20%, estimating the cognitive load cycle based on the periodic pattern of the attention level, and dividing the cognitive load cycle into a cognitive active period and a cognitive fatigue period;

[0162] Based on the division results of the cognitive active period and the cognitive fatigue period, the interactive strategy is generated in combination with the dynamic characteristics of the class participation, the optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, a mapping matrix between the interactive form and the effect of improving the class participation is established, and the interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages;

[0163] Constructing an environmental feature vector, performing a multivariate regression analysis on the temporal variation characteristics of the attention level and the dynamic characteristics of the classroom participation and the environmental feature vector, quantifying the influence weight of each environmental factor in the environmental feature vector, and making environmental adaptability corrections to the interaction depth adjustment scheme based on the influence weight, so as to obtain a personalized interaction strategy including an environmental parameter threshold;

[0164] The execution effect of the personalized interactive strategy at different cognitive stages is input into a decision tree model. Based on the real-time feedback of the attention level and the classroom participation, the strategy generation rules of the decision tree model are continuously optimized through reinforcement learning to achieve dynamic adjustment and optimization of the teaching guidance strategy.

[0165] The distance teaching interactive assessment model outputs data on students’ attention levels and classroom participation during distance learning.

[0166] First, the system constructs a sliding time window with a window size of ten minutes, sliding every thirty seconds. The system calculates the mean, standard deviation, rate of change, and periodic pattern of attention levels and classroom participation in each time window. For example, in a time window, the attention levels of 10 students are 80, 85, 90, 75, 82, 88, 92, 78, 85, and 90, respectively. The mean value of the attention level in the time window is 84.5, and the standard deviation is 5.8. Other indicators are calculated in the same way. Through such calculations, the system can obtain the temporal variation characteristics of attention levels and the dynamic characteristics of classroom participation.

[0167] Next, the system performs cognitive load analysis based on the temporal variation characteristics of the attention level. The system will continuously monitor the changing trend of the attention level in three consecutive sliding time windows. For example, if the average values ​​of the attention levels in three consecutive time windows are 85, 70, and 60, respectively, the decline is (85-60) / 85=29.4%, which exceeds the set threshold of 20%, and the system will trigger a rhythm adjustment. At the same time, the system will estimate the students' cognitive load cycle based on the periodic pattern of the attention level, and divide it into cognitively active periods and cognitively fatigued periods. For example, if it is found that the student's attention level shows a periodic change of one cycle every 30 minutes, 30 minutes can be used as a cognitive load cycle, and it can be divided into cognitively active periods and cognitively fatigued periods according to the level of attention.

[0168] Then, the system generates an interactive strategy based on the division results of cognitive active period and cognitive fatigue period, combined with the dynamic characteristics of classroom participation. The system calculates the inflection point of the attenuation curve of classroom participation in the cognitive active period. For example, through observation, it is found that students have high classroom participation within 10 minutes after the start of the cognitive active period, and then begin to decline. In this case, 10 minutes can be set as the optimal interaction interval. At the same time, the system will establish a mapping matrix between the form of interaction and the effect of improving classroom participation. For example, by analyzing historical data, it is found that question-based interaction has the most significant effect on improving classroom participation, so question-based interaction can be used preferentially during the cognitive active period. The system will also determine the interaction depth adjustment plan based on the duration of classroom participation in different cognitive stages. For example, if students' classroom participation continues to be low during the cognitive fatigue period, the depth of interaction can be appropriately reduced to avoid overstimulation.

[0169] In addition, the system constructs environmental feature vectors, such as ambient light, ambient noise, network connection quality, etc., and conducts multivariate regression analysis on the temporal variation characteristics of attention levels and the dynamic characteristics of classroom participation with environmental feature vectors to quantify the weight of environmental factors on student learning. For example, through analysis, it is found that the weight of environmental noise on students' attention levels is large. The system will make environmental adaptability corrections to the interaction depth adjustment plan according to the environmental noise level and set environmental parameter thresholds. For example, when the environmental noise exceeds a certain threshold, the system will automatically reduce the frequency and depth of interaction to reduce interference with students. Finally, the system will generate a personalized interaction strategy that includes environmental parameter thresholds.

[0170] Finally, the system inputs the execution effect of personalized interactive strategies at different cognitive stages into the decision tree model, and based on the real-time feedback of attention level and classroom participation, continuously optimizes the strategy generation rules of the decision tree model through reinforcement learning, so as to achieve dynamic adjustment and optimization of teaching guidance strategies. For example, if the system finds that a certain interactive strategy is not effective at a specific cognitive stage, it will adjust the execution frequency of the strategy or replace it with another strategy.

[0171] Through the above steps, the system can dynamically adjust the teaching rhythm and interaction strategy according to the students' real-time learning status and environmental factors, thereby improving the efficiency and quality of distance learning.

[0172] The beneficial effects can be summarized into the following three aspects:

[0173] 1. Personalized learning experience: The system can provide personalized teaching guidance based on each student’s cognitive load and learning status, so that each student can learn in the best state.

[0174] 2. Improve learning efficiency: By dynamically adjusting the teaching rhythm and interactive strategies, the system can effectively improve students' attention levels and classroom participation, thereby improving learning efficiency.

[0175] 3. Strong environmental adaptability: The system can automatically adjust the interactive strategy according to changes in environmental factors to ensure that teaching activities can proceed smoothly in various environments.

[0176] In an optional implementation, based on the division results of the cognitive active period and the cognitive fatigue period, the interactive strategy is generated in combination with the dynamic characteristics of the class participation, the optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, and a mapping matrix between the interactive form and the class participation improvement effect is established. The interactive depth adjustment plan is determined according to the duration of the class participation in different cognitive stages, including:

[0177] An exponential decay function is used to model the time series of classroom participation. Classroom participation data is collected within a five-minute sliding time window. The classroom participation decay parameter is obtained through least squares fitting, and a decay curve of classroom participation over time is established.

[0178] Calculate the second-order derivative of the attenuation curve, detect the inflection point where the second-order derivative turns from negative to positive, verify the local minimum constraint condition within one minute before and after the inflection point, and determine the time difference between the inflection point moment that meets the constraint condition and the last interaction moment as the optimal interaction interval time;

[0179] Construct a mapping matrix between interaction forms and effect indicators, wherein the effect indicators include the degree of improvement in class participation, the duration of improvement in class participation, and the interactive response rate, assign a time decay weight to the historical data of each of the effect indicators, and calculate the weighted average score of different interaction forms on each of the effect indicators;

[0180] Different weight coefficients are set for the effect indicators based on the current cognitive stage, the weight coefficients are weighted and calculated with the scores of the mapping matrix to obtain the interaction form scores, and the interaction form with the highest score is selected as the interaction strategy for the current cognitive stage;

[0181] Calculate the duration that the class participation is maintained above the baseline level after each interaction, input the difference ratio between the duration and the expected duration into the depth adjustment function, and generate an interaction depth adjustment plan;

[0182] Setting constraints on the interaction depth adjustment scheme to limit the change in interaction depth between two consecutive interactions to no more than 30%, reducing the upper limit of the depth adjustment to 15% during cognitive fatigue, and resetting to the baseline depth when three consecutive adjustments are invalid;

[0183] The weighted combined score of the improvement in class participation, class participation duration, and interaction response rate for each interaction is calculated, and the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function are updated according to the weighted combined score to achieve dynamic optimization of interaction strategy generation.

[0184] Obtain classroom participation data. During classroom teaching, students' classroom participation data is collected every second, such as by capturing students' expressions, body movements, eye contact, etc. through cameras, or by analyzing students' answers, questions, discussions, etc. through voice recognition technology, and converting this information into a value between 0 and 1 to represent the students' participation level, and storing this data in a database. For example, within a five-minute time window, 300 classroom participation data points were collected, with a value range of 0.2 to 0.9.

[0185] Perform time series modeling on class participation. In order to analyze the dynamic trend of class participation, an exponential decay function is used to fit the class participation data. Specifically, class participation data is collected within a five-minute sliding time window, and then the least squares method is used to fit the parameters of the exponential decay function. For example, assume that the fitted exponential decay function is y=0.8*exp(-0.1*x)+0.2, where y represents class participation and x represents time.

[0186] Determine the optimal interaction interval. In order to find the best time for interaction, it is necessary to calculate the inflection point of the attenuation curve. First, calculate the second-order derivative of the attenuation curve. Then, detect the inflection point where the second-order derivative changes from a negative value to a positive value. Within one minute before and after the inflection point, verify the local minimum constraint, that is, determine whether the function value at the inflection point is less than all function values ​​within one minute before and after it. If the constraint is met, the time difference between the inflection point and the last interaction is determined as the optimal interaction interval. For example, assuming that an inflection point is detected at the 8th minute and the local minimum constraint is met, and the last interaction occurred at the 3rd minute, the optimal interaction interval is 5 minutes.

[0187] Construct a mapping matrix between interaction forms and effect indicators. Create a matrix to record the relationship between different interaction forms and the effect of improving class participation. Effect indicators include the degree of improvement in class participation, the duration of improvement in class participation, and the interactive response rate. Assign time decay weights to the historical data of each effect indicator, that is, assign higher weights to recent data, such as using an exponential decay function for weighting. Then, calculate the weighted average score of different interaction forms on each effect indicator. For example, suppose there are three forms of interaction: asking questions, group discussions, and games. By analyzing historical data, you can get their weighted average scores on different effect indicators.

[0188] Determine the interaction strategy. Set different weight coefficients for different effect indicators according to the current cognitive stage. For example, in the cognitive active period, more attention is paid to the degree of improvement in class participation; while in the cognitive fatigue period, more attention is paid to the duration of improvement in class participation. Weighted calculation is performed on these weight coefficients and the scores of the mapping matrix to obtain the final score of each form of interaction. Select the interaction form with the highest score as the interaction strategy for the current cognitive stage. For example, assuming that in the cognitive active period, asking questions has the highest final score, then ask questions as the interaction strategy.

[0189] Adjust the depth of interaction. Calculate the duration of class participation after each interaction that remains above the baseline level. Input the difference ratio between this duration and the expected duration into the depth adjustment function to generate an interaction depth adjustment plan. For example, assuming that the duration of class participation after the interaction is 3 minutes and the expected duration is 5 minutes, the difference ratio is -0.4. Input -0.4 into the depth adjustment function to get the corresponding depth adjustment plan, such as reducing the difficulty of interaction or increasing the frequency of interaction.

[0190] Set depth adjustment constraints. To avoid excessive changes in interaction depth, set constraints. Limit the change in depth between two consecutive interactions to no more than 30%. During cognitive fatigue, reduce the upper limit of depth adjustment to 15%. When three consecutive adjustments are invalid, reset to the baseline depth.

[0191] Dynamically optimize the interaction strategy. Calculate the weighted combined score of the improvement in class participation, class participation duration, and interaction response rate for each interaction. Update the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function based on the score to achieve dynamic optimization of interaction strategy generation. For example, if the weighted combined score of a certain form of interaction is continuously low, reduce its score in the mapping matrix, or adjust the parameters of the depth adjustment function to make it easier to trigger depth adjustment.

[0192] Beneficial effects:

[0193] 1. Improve classroom teaching efficiency: By real-time monitoring of classroom participation and dynamically adjusting interactive strategies, students’ learning interest and attention can be effectively improved, thereby improving classroom teaching efficiency. For example, according to case data, this method can increase the average classroom participation from 0.5 to 0.7.

[0194] 2. Personalized learning experience: This method can provide personalized interactive experience according to the cognitive status and learning characteristics of different students, so as to promote students to better understand and master knowledge. For example, for students with low participation, the frequency of interaction can be increased or the difficulty of interaction can be reduced.

[0195] 3. Data-driven teaching optimization: This method can collect and analyze classroom teaching data, provide timely feedback and guidance to teachers, and help them continuously improve their teaching methods and strategies. For example, by analyzing the mapping matrix between interaction forms and effect indicators, it is possible to find out which interaction forms are more effective and which need to be improved.

[0196] Figure 2 FIG. 1 is a schematic diagram of the structure of an intelligent classroom teaching optimization system combining behavior recognition and the Internet of Things according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0197] The first unit is used to collect remote teaching interaction data through a distributed Internet of Things data collection system, wherein the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal, and the remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, wherein the environmental data includes lighting intensity data, background noise data, and network status data;

[0198] The second unit is used to construct a multimodal feature fusion network to process the remote teaching interaction data, and the multimodal feature fusion network includes a feature extraction subnetwork, a timing analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from the student behavior data, audio data and video data respectively, the timing analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction state feature;

[0199] The third unit is used to construct a remote teaching interaction evaluation model based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student's attention level value and classroom participation value in real time; generates a personalized teaching guidance strategy according to the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

[0200] According to a third aspect of the embodiments of the present invention,

[0201] An electronic device is provided, comprising:

[0202] processor;

[0203] a memory for storing processor-executable instructions;

[0204] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0205] According to a fourth aspect of the embodiments of the present invention,

[0206] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0207] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent classroom teaching optimization method combining behavior recognition and the Internet of Things, characterized in that: include: The remote teaching interaction data is collected through a distributed Internet of Things data collection system, wherein the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal, and the remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, and the environmental data includes lighting intensity data, background noise data, and network status data; Construct a multimodal feature fusion network to process the distance teaching interactive data, the multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from the student behavior data, audio data and video data respectively, the time series analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction state feature; A remote teaching interaction evaluation model is constructed based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student's attention level value and classroom participation value in real time; according to the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

2. The method according to claim 1, characterized in that: The multimodal feature fusion network includes a feature extraction subnetwork, a time series analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from student behavior data, audio data and video data respectively, the time series analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction status features including: The behavioral data features, audio data features and video data features are extracted respectively through a parallel three-way feature extraction architecture, wherein the behavioral data features are extracted by using a multi-scale convolution layer and a residual connection to extract local features of the facial area, and the head posture data is processed by a dual-stream attention module, and the output result of the dual-stream attention module is adaptively weighted by a feature recalibration module to obtain a behavioral feature vector; the audio data features are extracted by obtaining a time-frequency spectrum through a short-time Fourier transform, and a multi-scale one-dimensional convolution is used to extract the local time-frequency features of the time-frequency spectrum, and the local time-frequency features are processed by a channel attention mechanism to obtain an audio feature vector; the video data features are extracted by using a spatiotemporal convolution module to process a sixteen-frame image sequence to obtain a video feature vector; Inputting the behavior feature vector, the audio feature vector and the video feature vector into a bidirectional long short-term memory network for time series analysis, wherein the bidirectional long short-term memory network includes a modality-specific forget gate mechanism, wherein the modality-specific forget gate mechanism uses a first forget gate threshold for the behavior feature vector and a second forget gate threshold for the audio feature vector and the video feature vector, wherein the first forget gate threshold is greater than the second forget gate threshold; the output features of the bidirectional long short-term memory network are processed by a temporal attention mechanism to obtain a time series feature; The temporal features are input into the attention fusion subnetwork, which includes an intra-modal attention module and an inter-modal attention module. The intra-modal attention module calculates the importance weights of features at different time steps in the same modality, and the inter-modal attention module learns the complementary relationship between features of different modalities. The environmental data is mapped to the feature vector dimensional space through a multi-layer perceptron to obtain environmental features, and a conditional attention mechanism is constructed based on the environmental features to guide the feature fusion process. The output features of the attention fusion subnetwork are enhanced through residual connections and self-attention mechanisms to obtain student interaction status features.

3. The method according to claim 1, characterized in that A distance teaching interaction evaluation model is constructed based on the student interaction status characteristics. The distance teaching interaction evaluation model calculates the student attention level value and the classroom participation value in real time, including: Receive a student interaction state feature sequence within a ten-second time window, input the student interaction state feature sequence into a temporal convolution module, the temporal convolution module extracts multi-scale temporal dependency features through different convolution kernel sizes, and the multi-scale temporal dependency features are passed through a temporal importance evaluation unit to calculate the attention weight of each time step to obtain a micro-evaluation feature; The micro-evaluation features are input into a hierarchical long short-term memory network, the bottom network of the hierarchical long short-term memory network processes the micro-evaluation features to generate intermediate attention features, and the top network of the hierarchical long short-term memory network integrates the intermediate attention features to obtain long-term change features; Inputting the long-term change features into a multi-task learning framework, the multi-task learning framework extracts universal interaction features through a multi-head self-attention mechanism, and the universal interaction features are respectively input into an attention level evaluation branch and a class participation evaluation branch, wherein the attention level evaluation branch includes a pulse neural network module, and the class participation evaluation branch includes a temporal reasoning module of a graph attention network; A learning scenario recognition module is constructed based on the course type and teaching activity form. The output result of the learning scenario recognition module is passed through the scenario adaptation layer to obtain feature weights and evaluation thresholds. The feature weights and evaluation thresholds are input into a Bayesian neural network. The Bayesian neural network uses Monte Carlo dropout to quantify the uncertainty of model prediction and obtain a reliability index. The output result of the attention level evaluation branch is used through a nonlinear mapping function to obtain an attention level value, and the output result of the class participation evaluation branch is used through a cumulative probability distribution function to obtain a class participation value. The attention level value and the class participation value are smoothly transitioned through linear interpolation to obtain the final evaluation result.

4. The method according to claim 3, characterized in that The method also includes determining an attention evaluation value by a spiking neural network module: The input universal interaction feature is normalized to obtain a normalized feature, the normalized feature is mapped to the time domain through a logarithmic function to generate a timing pulse pattern, the timing pulse pattern is input into a first neuron layer having 256 integral-release neurons, the integral-release neurons of the first neuron layer convert the input current into a postsynaptic current through synaptic weights, and the synaptic weights are dynamically adjusted through a spike-time-dependent plasticity learning rule; Inputting the output signal of the first neuron layer into a second neuron layer having one hundred and twenty-eight leaky integral-discharge neurons, constructing lateral inhibitory connections between the leaky integral-discharge neurons in the second neuron layer to form a local competition mechanism, wherein the inhibitory strength of the lateral inhibitory connections decreases as the distance between adjacent neurons increases; Inputting the output signal of the second neuron layer into a third neuron layer having sixty-four adaptive leaky integral-release neurons, wherein the release thresholds of the adaptive leaky integral-release neurons of the third neuron layer are adaptively adjusted according to the neuron activity history; Establishing a multi-time-scale membrane potential dynamics model for each adaptive leaky integral-sparking neuron in the third neuron layer, wherein the multi-time-scale membrane potential dynamics model includes a fast synaptic current with a time constant of milliseconds, a slow modulation current with a time constant of hundreds of milliseconds, and an adaptation current with a time constant of seconds; Calculate the group average firing rate of the third neuron layer within the time window, map the group average firing rate to the interval of zero to one through a sigmoid function to obtain an initial attention value, and perform exponential sliding average on the initial attention value to obtain a final attention evaluation value, wherein the size of the time window dynamically shrinks as the group average firing rate increases.

5. The method according to claim 3, characterized in that: The method also includes determining an engagement representation vector by a temporal reasoning module of a graph attention network: Mapping the input universal interactive feature sequence to a graph structure, wherein the feature vector of each time step in the graph structure corresponds to a node, and the feature dimension of the node is 512 dimensions, calculating the edge weights between nodes through eight attention heads, the attention heads linearly transform the node features to obtain a query vector and a key vector, calculating the attention coefficient based on the query vector and the key vector, and concatenating the outputs of the eight attention heads to obtain an edge weight matrix; Construct a three-layer graph attention network. In the first layer of the graph attention network, each node is connected to the nodes in the adjacent positive and negative two time steps and relative position encoding is introduced. In the second layer of the graph attention network, the connection range is expanded to the adjacent positive and negative five time steps and features are extracted through multi-scale convolution operations. In the third layer of the graph attention network, global connection is achieved and long-range dependency features are extracted through the transformer structure. Performing spectral clustering on the node features output by the three-layer graph attention network to obtain multiple node clusters, performing a maximum pooling operation on each of the node clusters to obtain node cluster features, calculating the importance weight of each of the node cluster features, and weighting the node cluster features based on the corresponding importance weights to obtain a participation representation vector; The method further comprises: A gated graph neural network is used to update the node information in the graph structure. The retention degree of historical information is calculated through the update gate, and the integration ratio of new information is calculated through the reset gate. The candidate state is calculated based on the output value of the update gate and the output value of the reset gate. The candidate state is weightedly fused with the current state of the node to obtain the updated node state.

6. The method according to claim 1, characterized in that According to the calculation results of the remote teaching interaction evaluation model, a personalized teaching guidance strategy is generated, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions, including: Construct a sliding time window to perform time series analysis on the attention level and class participation output by the distance teaching interactive evaluation model. The window size of the sliding time window is ten minutes and slides once every thirty seconds. Calculate the mean, standard deviation, change rate and periodic pattern of the attention level and class participation in the sliding time window to obtain the time series change characteristics of the attention level and the dynamic characteristics of the class participation. performing cognitive load analysis according to the time series variation characteristics of the attention level, detecting the variation trend of the attention level within three consecutive sliding time windows, triggering rhythm adjustment when the attention level drops by more than 20%, estimating the cognitive load cycle based on the periodic pattern of the attention level, and dividing the cognitive load cycle into a cognitive active period and a cognitive fatigue period; Based on the division results of the cognitive active period and the cognitive fatigue period, the interactive strategy is generated in combination with the dynamic characteristics of the class participation, the optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, a mapping matrix between the interactive form and the effect of improving the class participation is established, and the interaction depth adjustment plan is determined according to the duration of the class participation in different cognitive stages; Constructing an environmental feature vector, performing a multivariate regression analysis on the temporal variation characteristics of the attention level and the dynamic characteristics of the classroom participation and the environmental feature vector, quantifying the influence weight of each environmental factor in the environmental feature vector, and making environmental adaptability corrections to the interaction depth adjustment scheme based on the influence weight, so as to obtain a personalized interaction strategy including an environmental parameter threshold; The execution effect of the personalized interactive strategy at different cognitive stages is input into a decision tree model. Based on the real-time feedback of the attention level and the classroom participation, the strategy generation rules of the decision tree model are continuously optimized through reinforcement learning to achieve dynamic adjustment and optimization of the teaching guidance strategy.

7. The method according to claim 6, characterized in that Based on the division results of the cognitive active period and the cognitive fatigue period, the interactive strategy is generated in combination with the dynamic characteristics of the class participation, the optimal interaction interval is determined by calculating the inflection point of the attenuation curve of the class participation in the cognitive active period, and a mapping matrix between the interactive form and the class participation improvement effect is established. The interactive depth adjustment plan is determined according to the duration of the class participation in different cognitive stages, including: An exponential decay function is used to model the time series of classroom participation. Classroom participation data is collected within a five-minute sliding time window. The classroom participation decay parameter is obtained through least squares fitting, and a decay curve of classroom participation over time is established. Calculate the second-order derivative of the attenuation curve, detect the inflection point where the second-order derivative turns from negative to positive, verify the local minimum constraint condition within one minute before and after the inflection point, and determine the time difference between the inflection point moment that meets the constraint condition and the last interaction moment as the optimal interaction interval time; Construct a mapping matrix between interaction forms and effect indicators, wherein the effect indicators include the degree of improvement in class participation, the duration of improvement in class participation, and the interactive response rate, assign a time decay weight to the historical data of each of the effect indicators, and calculate the weighted average score of different interaction forms on each of the effect indicators; Different weight coefficients are set for the effect indicators based on the current cognitive stage, the weight coefficients are weighted and calculated with the scores of the mapping matrix to obtain the interaction form scores, and the interaction form with the highest score is selected as the interaction strategy for the current cognitive stage; Calculate the duration that the class participation is maintained above the baseline level after each interaction, input the difference ratio between the duration and the expected duration into the depth adjustment function, and generate an interaction depth adjustment plan; Setting constraints on the interaction depth adjustment scheme to limit the change in interaction depth between two consecutive interactions to no more than 30%, reducing the upper limit of the depth adjustment to 15% during cognitive fatigue, and resetting to the baseline depth when three consecutive adjustments are invalid; The weighted combined score of the improvement in class participation, class participation duration, and interaction response rate for each interaction is calculated, and the parameter values ​​of the mapping matrix and the parameter values ​​of the depth adjustment function are updated according to the weighted combined score to achieve dynamic optimization of interaction strategy generation.

8. An intelligent classroom teaching optimization system combining behavior recognition and the Internet of Things, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect remote teaching interaction data through a distributed Internet of Things data collection system, wherein the distributed Internet of Things data collection system includes a behavior data collection module, an audio and video data collection module, and an environmental data collection module deployed on a student terminal, and the remote teaching interaction data includes student behavior data, audio data, video data, and environmental data; wherein the behavior data collection module obtains student behavior data including facial expression features and head posture data, the audio and video data collection module obtains student audio data and video data, and the environmental data collection module obtains environmental data of the student's environment, wherein the environmental data includes lighting intensity data, background noise data, and network status data; The second unit is used to construct a multimodal feature fusion network to process the remote teaching interaction data, and the multimodal feature fusion network includes a feature extraction subnetwork, a timing analysis subnetwork and an attention fusion subnetwork; wherein the feature extraction subnetwork extracts behavior feature vectors, audio feature vectors and video feature vectors from the student behavior data, audio data and video data respectively, the timing analysis subnetwork analyzes the time series variation law of each feature vector based on the long short-term memory network structure, and the attention fusion subnetwork adaptively fuses the multimodal feature vector with the environmental data to obtain the student interaction state feature; The third unit is used to construct a remote teaching interaction evaluation model based on the student interaction status characteristics, and the remote teaching interaction evaluation model calculates the student's attention level value and classroom participation value in real time; generates a personalized teaching guidance strategy according to the calculation results of the remote teaching interaction evaluation model, and the personalized teaching guidance strategy includes teaching rhythm adjustment suggestions and interaction mode optimization suggestions; the remote teaching interaction evaluation model feeds back the personalized teaching guidance strategy to the teacher terminal in real time through the Internet of Things communication network, so as to realize dynamic optimization and adjustment of the remote teaching process.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images