Multi-modal classroom state evaluation method based on attitude motion gating

Abstract

The invention relates to a multi-mode classroom state evaluation method based on attitude motion gating. The method comprises the following steps: acquiring a high-definition video stream of a classroom scene; extracting a facial emotion feature vector and a human skeleton posture feature vector of the high-definition video stream, and destroying an original image frame of the high-definition video stream; encrypting and transmitting the facial emotion feature vector and the human skeleton posture feature vector to a cloud server; performing time sequence alignment on the facial emotion feature vector and the human skeleton posture feature vector, and generating a dynamic reliability gating factor based on the motion state of the human skeleton posture feature vector; utilizing a dynamic reliability gating factor and a multi-modal cross attention network, fusing the facial emotion feature vector and the human skeleton posture feature vector, and generating a space-time semantic feature; and determining a learning concentration level based on the spatio-temporal semantic features. According to the method, intelligent disambiguation of expression-posture conflict semantics in a complex classroom scene is realized by means of a dynamic reliability gating factor.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence education technology and computer vision, specifically to a multimodal classroom state assessment method based on posture motion gating. Background Technology

[0002] With the deepening of the digital education strategy and the widespread application of "artificial intelligence+" in education, the data-driven and intelligent analysis of classroom teaching processes has become a key technological path to improve teaching quality and achieve precise educational evaluation. Relevant national policies and standards explicitly advocate the use of artificial intelligence technology for full-sample, full-process accompanying data collection to achieve a scientific and objective assessment of students' learning status and classroom teaching quality, thereby promoting systemic reform of the education evaluation system.

[0003] However, in actual large-scale deployment and application, existing classroom status analysis technologies still face significant bottlenecks in terms of architecture, algorithms, and compliance, mainly in the following three aspects: First, at the system architecture level, current mainstream solutions generally adopt a centralized architecture of "front-end acquisition - cloud processing". This type of architecture requires the continuous uploading of high-definition video streams captured by front-end cameras to cloud servers for analysis, leading to two prominent problems: First, in scenarios with multiple classrooms in a school, the transmission of high-definition video streams puts enormous pressure on the bandwidth of the campus network's outbound bandwidth, easily causing network latency, packet loss, and delayed analysis feedback, making it difficult to support the timeliness requirements of real-time teaching intervention; Second, directly transmitting and storing raw videos containing students' facial biometric information faces increasingly stringent requirements for personal information protection and compliance with data regulations for minors, posing significant risks of privacy leaks and social ethical controversies.

[0004] Secondly, at the algorithmic level, early technologies often relied on a single visual modality (such as pure facial expression recognition or human pose estimation), which lacked robustness in the complex and uncontrolled environment of a real classroom. For example, facial expression recognition is susceptible to changes in lighting, facial occlusion, and non-frontal angles; while human pose analysis based solely on skeletal key points struggles to distinguish between semantically different but similarly posed behaviors (such as "resting chin in thought" versus "resting chin in sleep"). The limitations of a single modality lead to a high false positive rate, failing to accurately reflect the true learning state.

[0005] Furthermore, although multimodal analysis schemes have emerged in recent years, most remain at the decision-level fusion stage, which involves simply weighting and summing the recognition results of each modality. Such methods cannot effectively model semantic conflicts and complementary relationships between modalities. For example, when a student's facial expression is negative while their posture indicates positive interaction, simple weighting can mask the true state of "high engagement." Existing technologies lack a deep fusion mechanism that dynamically calibrates the weights of one modality based on contextual information from another, resulting in limited system understanding of complex classroom interaction semantics.

[0006] Therefore, there is an urgent need to develop a classroom status assessment method that can protect privacy from the source, reduce bandwidth dependence, and eliminate semantic ambiguity through a deep integration mechanism, so as to support highly reliable, real-time, compliant and reliable learning status perception and teaching optimization in smart education scenarios. Summary of the Invention

[0007] This invention provides a multimodal classroom state assessment method based on posture motion gating. Its purpose is to solve the problems of privacy leakage caused by uploading the original video to the cloud and the high semantic ambiguity and misjudgment rate caused by insufficient modality fusion in existing classroom state assessment technologies.

[0008] To achieve the above objectives, the first aspect of the present invention provides a multimodal classroom state assessment method based on posture motion gating, comprising the following steps: Capture high-definition video streams of classroom scenes; Extract facial emotion feature vectors and human skeleton pose feature vectors from the high-definition video stream, and destroy the original image frames of the high-definition video stream; The facial emotion feature vector and the human skeletal posture feature vector are encrypted and transmitted to the cloud server. The facial emotion feature vector and the human skeleton posture feature vector are time-aligned, and a dynamic reliability gating factor is generated based on the motion state of the human skeleton posture feature vector. The dynamic reliability gating factor is used to characterize the credibility of the facial emotion feature vector. The dynamic reliability gating factor and the multimodal cross-attention network are used to fuse the facial emotion feature vector and the human skeleton pose feature vector to generate spatiotemporal semantic features. The learning focus level is determined based on the spatiotemporal semantic features.

[0009] Furthermore, the dynamic reliability gating factor generated based on the motion state of the human skeleton posture feature vector includes: Calculate the Euclidean distance between the human skeleton posture feature vectors of adjacent time stamps to obtain the posture motion difference value; The attitude motion difference value is input into a gated network containing fully connected layers and activation functions; The gating network outputs a normalized dynamic reliability gating factor, which is negatively correlated with the attitude motion difference value. Through network parameter learning, the output dynamic reliability gating factor value decreases when the attitude motion difference value increases.

[0010] Furthermore, methods for generating spatiotemporal semantic features include: The human skeleton posture feature vector is projected into a query vector, and the facial emotion feature vector is projected into a key vector and a value vector, respectively. Calculate the relevance score between the query vector and the key vector, and normalize it to obtain the attention weight; The value vector is weighted and aggregated using the attention weights to obtain the initial attention output vector; The dynamic reliability gating factor is weighted with the initial attention output vector to obtain the final attention output vector; The final attention output vector is residually connected to the human skeleton pose feature vector, and the connection result is subjected to layer normalization to generate the spatiotemporal semantic features.

[0011] Furthermore, the acquisition of high-definition video streams of classroom scenes includes: deploying a camera with a wide-angle field of view at a preset position in front of the classroom and installing it at a preset downward tilt angle so that the camera's field of view covers the entire student area; and capturing the video stream output by the camera in real time through an edge computing terminal.

[0012] Furthermore, the method for extracting facial emotion feature vectors and human skeletal pose feature vectors from the high-definition video stream includes: On the edge computing terminal, the video stream is processed using a parallel lightweight neural network model; Facial emotion feature vectors are extracted using a first lightweight neural network model; Human skeletal posture feature vectors are extracted using a second lightweight neural network model.

[0013] Furthermore, destroying the original image frames of the high-definition video stream includes: immediately overwriting and destroying the original image frame data stored in memory after the facial emotion feature vector and the human skeleton posture feature vector are extracted.

[0014] Furthermore, the method for determining the learning focus level based on the aforementioned spatiotemporal semantic features includes: The spatiotemporal semantic features are input into a temporal neural network model to combine historical temporal context information for modeling; Based on the output of the temporal neural network model, the probability distribution of the current time belonging to a predefined multi-class learning attention level is calculated through a classification network; The final learning focus level is determined based on the probability distribution.

[0015] To achieve the above objectives, a second aspect of the present invention provides a multimodal classroom state assessment system based on posture motion gating, comprising: An edge computing module deployed at the front end of the classroom, the edge computing module comprising: The visual acquisition unit is used to acquire video streams covering the entire student area of ​​the classroom. The feature decoupling unit is equipped with a first lightweight neural network model and a second lightweight neural network model in parallel, which are used to extract facial emotion feature vectors and human skeleton posture feature vectors, respectively. The data desensitization unit is used to immediately destroy the original image frames in memory after the facial emotion feature vector and the human skeleton posture feature vector are extracted. An encrypted communication unit is used to encrypt and upload the facial emotion feature vector and the human skeleton posture feature vector. And, a cloud-based fusion analysis module, the fusion analysis module including: The temporal alignment unit is used to perform frame-level alignment of feature vectors from different modalities based on timestamps. The motion gating unit is used to calculate the dynamic reliability gating factor based on the temporal difference value of the human skeleton posture feature vector; An uncertainty perception fusion unit is configured with a multimodal cross-attention network, which modulates the attention output vector based on the facial feature credibility represented by the dynamic reliability gating factor, thereby fusing the facial emotion feature vector with the human skeleton posture feature vector to generate spatiotemporal semantic features. The state discrimination unit is configured with a temporal neural network and a classifier to output a learning focus level based on the spatiotemporal semantic features.

[0016] To achieve the above objectives, a third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal classroom state assessment method based on attitude motion gating as described above.

[0017] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the multimodal classroom state assessment method based on attitude motion gating.

[0018] The beneficial effects of this invention are: Compared with existing technologies, this invention provides a multimodal classroom state assessment method based on posture motion gating. By "extracting and destroying edge features in real time," it avoids uploading original videos to the cloud from the source, only uploading desensitized feature vectors that cannot be restored, thus solving the problem of privacy leakage. Furthermore, through the "asymmetric fusion based on posture motion gating" mechanism, it uses the motion energy of posture features to dynamically generate gating factors and adjusts the weights of facial features in real time in a multimodal cross-attention network. This allows the system to focus on posture semantics when students are in vigorous movement and on micro-expressions when they are still, thereby achieving semantic disambiguation for complex scenarios such as "thinking and daydreaming" and "discussion and casual conversation." Combined with LSTM temporal modeling to filter instantaneous interference, it ultimately significantly improves the accuracy and robustness of state assessment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0020] Figure 1 This is a flowchart of a multimodal classroom state assessment method based on posture motion gating disclosed in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the overall architecture of a multimodal classroom state assessment system based on posture motion gating disclosed in an embodiment of the present invention.

[0022] Figure 3 This is a hardware structure block diagram of an edge computing terminal disclosed in an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] According to embodiments of the present invention, it should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the following methods, in some cases the steps shown or described may be executed in a different order than that shown here.

[0025] like Figure 1 As shown, this invention provides a multimodal classroom state assessment method based on posture motion gating, comprising the following steps: Step S100: Acquire high-definition video stream of the classroom scene; In this embodiment, the system first acquires a high-definition video stream of the classroom scene using hardware devices deployed at the front end of the classroom. For example... Figure 2 As shown, at the classroom end, a high-definition camera continuously captures a video stream covering the entire student area and transmits it to the edge computing box. Specifically, a high-definition camera with a wide-angle field of view is deployed in the center above the blackboard in the classroom, and its tilt angle is set to tilt downwards to ensure that its field of view can completely cover the entire student area below the podium, thereby providing a comprehensive source of visual information for subsequent analysis.

[0026] The camera is configured to output a high-definition video stream that meets the requirements of real-time analysis. Its resolution setting needs to be sufficient to clearly capture the details of students' faces and limbs, and its frame rate setting needs to ensure the continuity of movements, avoiding motion blur or loss of key states due to excessively low frame rates, thereby providing a high-quality raw image sequence for feature extraction.

[0027] The capture and initial processing of the video stream are performed by edge computing terminals deployed locally in the classroom. For example... Figure 3 As shown, this terminal uses an embedded hardware platform with an integrated AI acceleration chip, enabling real-time video stream decoding and preprocessing. The camera connects to the edge computing terminal via a video interface (such as USB or a network interface), and the terminal continuously reads video data in a streaming manner and sends it to the subsequent feature extraction pipeline.

[0028] Step S200: Extract the facial emotion feature vector and human skeleton pose feature vector from the high-definition video stream, and destroy the original image frames of the high-definition video stream. In this embodiment, the acquired high-definition video stream is subjected to real-time feature decoupling and desensitization processing locally on the edge computing terminal. For example... Figure 2 As shown, firstly, a parallelized lightweight neural network model is used to synchronously analyze the video stream: On one hand, a facial emotion analysis model based on a lightweight network (such as ShuffleNetV2) is used to process the detected and aligned face region images, extracting a 128-dimensional facial emotion feature vector containing gaze direction, eye and mouth states, and probability distributions of multiple emotion types. This vector includes gaze deflection angle, mouth aspect ratio, eye aspect ratio, and the Softmax probability distributions of seven basic emotions (happiness, sadness, focus, confusion, surprise, disgust, and neutrality). On the other hand, a skeletal pose analysis model based on a lightweight pose estimation network (such as OpenPose or MobileNet-Pose) is used to process the downsampled global image, outputting a 72-dimensional feature vector. This vector is composed of the (x, y) normalized coordinates and confidence scores of 18 key points on the upper body.

[0029] After the above feature extraction process is completed, an irreversible data anonymization operation is immediately performed: such as Figure 2 As shown, by setting a circular buffer in memory and actively writing overwrite data to the corresponding memory area within a very short time (e.g., within 50 milliseconds) after the feature vector is generated, the cached content storing the original image frame is completely destroyed. This mechanism ensures that the student's facial image cannot be recovered through system interfaces or physical debugging methods at any time, thereby achieving privacy protection of biometric information at the data source.

[0030] Step S300: Encrypt and transmit the facial emotion feature vector and the human skeleton posture feature vector to the cloud server; like Figure 2 As shown, specifically, the edge computing terminal encapsulates two feature vectors and information such as precise timestamps into a structured lightweight data packet, and then uploads the data packet to the cloud server through a secure transport layer protocol based on TLS / SSL, or through an encrypted channel that complies with regulatory requirements, such as an encryption algorithm.

[0031] This process uses encryption to ensure the confidentiality and integrity of feature data during transmission over public or campus networks, effectively preventing data theft or tampering during transmission. Simultaneously, because the transmitted content consists only of floating-point feature vectors of tens to hundreds of dimensions, rather than the original high-definition video stream, the data volume is compressed by several orders of magnitude compared to the original video. This significantly reduces the continuous pressure on network bandwidth while ensuring security, making large-scale concurrent uploading and analysis in classrooms possible. Upon receiving the data, the cloud server first decrypts and verifies the encrypted data packet, then sends the parsed feature vectors to the subsequent time-series alignment and fusion analysis module.

[0032] Step S400: Perform temporal alignment between the facial emotion feature vector and the human skeleton posture feature vector, and generate a dynamic reliability gating factor based on the motion state of the human skeleton posture feature vector. In this step, the cloud server first performs multimodal timing alignment. For example... Figure 1 As shown, the server parses the timestamp information in each data packet and, based on the sliding time window mechanism, precisely aligns the facial emotion feature streams that may be uploaded at different frequencies with the human skeletal pose feature streams at the frame level, ensuring that the two modal features at the same time can be correctly paired, providing a consistent time reference for subsequent fusion analysis.

[0033] After timing alignment is completed, attitude motion uncertainty modeling and gating factor generation are performed. The core of this process is calculating the dynamic reliability gating factor. The calculation steps are as follows: Step S401: Calculate the human skeleton pose feature vector after alignment of adjacent time points. and The Euclidean distance between them yields the attitude motion difference value. This is used to quantify the drasticness of the pose change in the current frame.

[0034] in, and They represent Time and The 72-dimensional human skeleton pose feature vector at time step. This represents the L2 norm of a vector.

[0035] Step S402, to Input a lightweight gated network consisting of two fully connected (FC) layers and a sigmoid activation function; Step S403: Output a normalized dynamic reliability gating factor through the gating network. The dynamic reliability gating factor is negatively correlated with the attitude motion difference value. Through network parameter learning, the output dynamic reliability gating factor value decreases as the attitude motion difference value increases.

[0036] in, , These are the weights and bias parameters for the first fully connected layer; , The weights and bias parameters for the second fully connected layer; To modify the activation function of the linear unit; This indicates the Sigmoid activation function, which compresses the output to... interval; This is the dynamic reliability gating factor, and its value is related to... Negative correlation.

[0037] The significance of this gating factor lies in: the dynamic reliability gating factor Used to characterize the reliability of facial emotion feature vectors. When the student's posture changes smoothly ( When young (e.g., sitting upright and listening to a lecture), A value close to 1 indicates that facial expression features are highly reliable and should be prioritized in the fusion process; when students' postures change drastically ( When large (such as turning around to discuss, raising hands), A value close to 0 indicates that the reliability of the face may be reduced due to motion blur, occlusion, or deviation from the camera. The system should automatically reduce its decision weight and rely more on the semantics of the pose features.

[0038] Step S500: The dynamic reliability gating factor is used to characterize the credibility of the facial emotion feature vector. The dynamic reliability gating factor and the multimodal cross-attention network are used to fuse the facial emotion feature vector and the human skeleton pose feature vector to generate spatiotemporal semantic features. The dynamic reliability gating factor generated using the aforementioned steps The constructed uncertainty-aware cross-attention network is used to deeply fuse aligned facial emotion features and human skeletal pose features to generate highly robust spatiotemporal semantic features.

[0039] The method for generating spatiotemporal semantic features includes the following steps: Step S501: Perform linear projection on the feature vectors of the two modalities to generate the query vector, key vector, and value vector required by the attention mechanism.

[0040]

[0041]

[0042] in, for Human skeleton posture feature vector at time step; for Facial emotion feature vector at any given moment; , , The projective weight matrix is ​​a learnable matrix. The query vector is derived from the pose features; , These are the key and value vectors derived from facial expression features.

[0043] Step S502: Calculate attitude query With emoji button The correlation scores between the two values ​​are calculated and then normalized using Softmax to obtain the attention weights. :

[0044] in, The dimension of the key vector is used to scale the dot product result; This is a transpose operation.

[0045] Step S503: Utilize attention weights Weighted aggregate value vector The initial attention output vector is obtained. :

[0046] Step S504: Introduce dynamic reliability gating factor The initial attention output vector is modulated to obtain the final attention output vector. :

[0047] Step S505: The final attention output is fused with the original pose features through residual connections and layer normalization to generate the final spatiotemporal semantic features. :

[0048] This fusion mechanism enables asymmetric semantic correction: when the student is stationary or making slight movements ( The system focuses on... It absorbs subtle facial micro-expression semantics; during students' vigorous exercise ( Attention output is suppressed, and features are fused. Then from the residual term The system is dominated by pose semantics for state discrimination, which effectively solves the problem of facial feature failure caused by motion blur or occlusion, and improves the accuracy and robustness of state assessment in complex dynamic classroom scenarios.

[0049] Step S600: Determine the learning focus level based on the spatiotemporal semantic features.

[0050] The generated spatiotemporal semantic features The input is sent to the temporal state discrimination module. This module first uses a bidirectional long short-term memory network (Bi-LSTM) to model the continuous fused feature sequence. Specifically, it uses two LSTM units with a hidden layer dimension of 512. This network memorizes the historical state context of a past period (e.g., 3-5 seconds) through its gating mechanism, which can effectively filter out state fluctuations caused by brief, non-learning intentional actions (such as tidying clothes or bending down to pick up a pen), thus outputting a smooth and coherent temporal state representation. For example, if a student only picks up a pen (taking 2 seconds), although the posture changes to "bending over," the LSTM, combined with the "sitting upright" state in the context, will smoothly output and maintain the "focused" judgment.

[0051] Then, the hidden state output by the LSTM is... (i.e., the aforementioned temporal state representation) is fed into a fully connected classification layer, and the probability distribution of its belonging to multiple predefined attention levels is calculated using the Softmax function. :

[0052] in, , The weights and bias parameters for the classification layer; These represent the probabilities of the current state being classified into four categories: Class 1: Deep Focus (upright posture, focused gaze, calm expression), Class 2: Superficial Focus (relatively casual posture, gaze in a general area), Class 3: Active Interaction (posture of raising hands / standing, mouth opening and closing), and Class 4: Passive / Fatigued (posture of lying down, eyes closed, head not moving for a long time).

[0053] Finally, based on the probability distribution The category corresponding to the highest probability value is used to determine the student's level of focus at the current moment, and corresponding status labels and structured analysis results are generated to provide data support for real-time teaching intervention or post-class learning analysis. For example... Figure 2 As shown, the analysis results can be fed back to the teacher's terminal or the Web management backend.

[0054] Preferably, this embodiment also includes an adaptive feedback module: (1) Occlusion handling: When the pose detection network recognizes "hand occluding face", the system automatically reduces the weight of facial features and transfers 80% of the decision-making power to pose features and historical state.

[0055] (2) Lighting adaptation: When the overall brightness of the screen is detected to be lower than the threshold (such as when the projector is turned on or the classroom lights are turned off), the edge end automatically switches to "night vision enhancement mode" or only relies on the contour posture for analysis to avoid misjudgment caused by the face being too dark.

[0056] This embodiment constructs an asymmetric multimodal fusion paradigm that responds to real-world classroom dynamics by introducing and applying a "dynamic reliability gating factor." Traditional multimodal fusion methods often use fixed weighting coefficients, which cannot address the issue of facial feature failure caused by drastic changes in student posture. This method, however, calculates the inter-frame motion energy of posture features (…). And map it to a reliability gating factor that can be adjusted in real time ( This allows the system to adaptively determine the reliability of facial features based on the current range of motion. When the student is in a state of large-amplitude movement ( The system automatically suppresses the influence of potentially blurred or occluded facial features on decision-making, instead relying on more robust gesture semantics; when the student is still or making slight movements ( The system then enhances the weighting of facial micro-expression analysis. This dynamic modulation based on the uncertainty of physical motion fundamentally solves the misjudgment caused by the instantaneous failure of a single mode.

[0057] Secondly, this method does not simply use motion energy as a threshold, but instead uses a learnable gating network to map continuous motion difference values ​​into smooth, differentiable modulation scalars, which are then directly embedded into the core computation of the cross-attention mechanism. This allows "posture" to not only be one of the features being fused, but also to serve as a contextual prior, dynamically adjusting the contribution of "facial expression" features in the attention space, thus achieving deep, asymmetric semantic interaction between modalities. This mechanism mimics the cognitive process by which human teachers interpret facial expressions in conjunction with the range of motion when observing students, thereby endowing the system with a high-order semantic disambiguation ability to distinguish subtle states such as "thinking" from "daydreaming," and "attentive listening" from "distracted thinking" in complex classroom scenarios, significantly improving the accuracy of state assessment and scenario adaptability.

[0058] According to another aspect of the embodiments of this application, a multimodal classroom state assessment system based on posture motion gating is also provided, including: An edge computing module deployed at the front end of the classroom, the edge computing module comprising: The visual acquisition unit is used to acquire video streams covering the entire student area of ​​the classroom. The feature decoupling unit is equipped with a first lightweight neural network model and a second lightweight neural network model in parallel, which are used to extract facial emotion feature vectors and human skeleton posture feature vectors, respectively. The data desensitization unit is used to immediately destroy the original image frames in memory after the facial emotion feature vector and the human skeleton posture feature vector are extracted. An encrypted communication unit is used to encrypt and upload the facial emotion feature vector and the human skeleton posture feature vector. And, a cloud-based fusion analysis module, the fusion analysis module including: The temporal alignment unit is used to perform frame-level alignment of feature vectors from different modalities based on timestamps. The motion gating unit is used to calculate the dynamic reliability gating factor based on the temporal difference value of the human skeleton posture feature vector; The uncertainty perception fusion unit is configured with a multimodal cross-attention network, which is used to modulate the attention mechanism using the dynamic reliability gating factor, and fuse the facial emotion feature vector and the human skeleton posture feature vector to generate spatiotemporal semantic features. The state discrimination unit is configured with a temporal neural network and a classifier to output a learning focus level based on the spatiotemporal semantic features.

[0059] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0060] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0061] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0062] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0063] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0064] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal classroom state assessment method based on posture motion gating, characterized in that, Includes the following steps: Capture high-definition video streams of classroom scenes; Extract facial emotion feature vectors and human skeleton pose feature vectors from the high-definition video stream, and destroy the original image frames of the high-definition video stream; The facial emotion feature vector and the human skeletal posture feature vector are encrypted and transmitted to the cloud server. The facial emotion feature vector and the human skeleton posture feature vector are time-aligned, and a dynamic reliability gating factor is generated based on the motion state of the human skeleton posture feature vector. The dynamic reliability gating factor is used to characterize the credibility of the facial emotion feature vector. The dynamic reliability gating factor and the multimodal cross-attention network are used to fuse the facial emotion feature vector and the human skeleton pose feature vector to generate spatiotemporal semantic features. The learning focus level is determined based on the spatiotemporal semantic features.

2. The multimodal classroom state assessment method based on posture motion gating as described in claim 1, characterized in that, The dynamic reliability gating factor generated based on the motion state of the human skeleton posture feature vector includes: Calculate the Euclidean distance between the human skeleton posture feature vectors of adjacent time stamps to obtain the posture motion difference value; The attitude motion difference value is input into a gated network containing fully connected layers and activation functions; The gating network outputs a normalized dynamic reliability gating factor, which is negatively correlated with the attitude motion difference value. Through network parameter learning, the output dynamic reliability gating factor value decreases when the attitude motion difference value increases.

3. The multimodal classroom state assessment method based on posture motion gating as described in claim 1, characterized in that, Methods for generating spatiotemporal semantic features include: The human skeleton posture feature vector is projected into a query vector, and the facial emotion feature vector is projected into a key vector and a value vector, respectively. Calculate the relevance score between the query vector and the key vector, and normalize it to obtain the attention weight; The value vector is weighted and aggregated using the attention weights to obtain the initial attention output vector; The dynamic reliability gating factor is weighted with the initial attention output vector to obtain the final attention output vector; The final attention output vector is residually connected to the human skeleton pose feature vector, and the connection result is subjected to layer normalization to generate the spatiotemporal semantic features.

4. The multimodal classroom state assessment method based on posture motion gating as described in claim 1, characterized in that, The process of collecting high-definition video streams of classroom scenes includes: deploying a camera with a wide-angle field of view at a preset position in front of the classroom and installing it at a preset downward tilt angle so that the camera's field of view covers the entire student area; and capturing the video stream output by the camera in real time through an edge computing terminal.

5. The multimodal classroom state assessment method based on posture motion gating as described in claim 4, characterized in that, The method for extracting facial emotion feature vectors and human skeletal pose feature vectors from the high-definition video stream includes: On the edge computing terminal, the video stream is processed using a parallel lightweight neural network model; Facial emotion feature vectors are extracted using a first lightweight neural network model; Human skeletal posture feature vectors are extracted using a second lightweight neural network model.

6. The multimodal classroom state assessment method based on posture motion gating as described in claim 1, characterized in that, Destroying the original image frames of the high-definition video stream includes: immediately overwriting and destroying the original image frame data stored in memory after the facial emotion feature vector and the human skeleton posture feature vector are extracted.

7. The multimodal classroom state assessment method based on posture motion gating as described in claim 1, characterized in that, The method for determining the learning focus level based on the aforementioned spatiotemporal semantic features includes: The spatiotemporal semantic features are input into a temporal neural network model to combine historical temporal context information for modeling; Based on the output of the temporal neural network model, the probability distribution of the current time belonging to a predefined multi-class learning attention level is calculated through a classification network; The final learning focus level is determined based on the probability distribution.

8. A multimodal classroom state assessment system based on posture motion gating, characterized in that, include: An edge computing module deployed at the front end of the classroom, the edge computing module comprising: The visual acquisition unit is used to acquire video streams covering the entire student area of ​​the classroom. The feature decoupling unit is equipped with a first lightweight neural network model and a second lightweight neural network model in parallel, which are used to extract facial emotion feature vectors and human skeleton posture feature vectors, respectively. The data desensitization unit is used to immediately destroy the original image frames in memory after the facial emotion feature vector and the human skeleton posture feature vector are extracted. An encrypted communication unit is used to encrypt and upload the facial emotion feature vector and the human skeleton posture feature vector. And, a cloud-based fusion analysis module, the fusion analysis module including: The temporal alignment unit is used to perform frame-level alignment of feature vectors from different modalities based on timestamps. The motion gating unit is used to calculate the dynamic reliability gating factor based on the temporal difference value of the human skeleton posture feature vector; An uncertainty perception fusion unit is configured with a multimodal cross-attention network, which modulates the attention output vector based on the facial feature credibility represented by the dynamic reliability gating factor, thereby fusing the facial emotion feature vector with the human skeleton posture feature vector to generate spatiotemporal semantic features. The state discrimination unit is configured with a temporal neural network and a classifier to output a learning focus level based on the spatiotemporal semantic features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal classroom state assessment method based on attitude motion gating as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it executes the steps of the multimodal classroom state assessment method based on attitude motion gating as described in any of claims 1-7.

Images