Student classroom concentration degree detection method and system based on multi-modal fusion, and medium

Abstract

The invention discloses a multi-modal fusion-based student classroom concentration degree detection method and system and a medium, and the method comprises the steps: inputting a classroom video into a concentration degree detection model, and outputting a concentration degree grade classification result; the training process of the concentration degree detection model comprises the following steps: carrying out frame sampling and feature extraction on a classroom video; encoding the features after feature extraction to obtain a global video representation and physiological time sequence feature sequence; constructing a fusion module of an attention mechanism, and splicing the calculated attention weight with the global video representation to obtain a visual representation; splicing preset text description containing field priori and learnable prompt words, and performing coding extraction to obtain text embedding; calculating cosine similarity of visual representation and text embedding to obtain concentration degree probability distribution, so as to realize concentration degree grade classification; according to the detection method, accurate and explainable identification of the classroom concentration state of the student is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, and medium for detecting student classroom attention based on multimodal fusion. Background Technology

[0002] Student classroom attention detection is a core research area in educational artificial intelligence. Its goal is to automatically identify students' attention levels by analyzing their multimodal behavior in the classroom environment. This technology has significant application value in scenarios such as intelligent teaching systems, online education platforms, classroom quality assessment, and personalized learning intervention.

[0003] Currently, the mainstream methods for detecting attention mainly fall into the following two categories:

[0004] 1. Physiological Behavioral Feature-Based Methods: These methods extract physiological features such as eye gaze direction, facial action units, and head pose from students using specialized toolkits (e.g., OpenFace). They then utilize temporal models (e.g., GRU, LSTM, or Transformer) for feature modeling and state classification. The advantage of these methods is the good interpretability of the features; however, their drawbacks include reliance on low-dimensional physiological signals, a lack of rich semantic information, and a high dependence on the accuracy of the feature extraction tools.

[0005] 2. Methods based on deep learning pre-trained models: These methods utilize visual models (such as ResNet, EfficientNet, ViT, etc.) pre-trained on large-scale image datasets to extract deep features from video frames, and then use temporal modeling networks (such as Transformer, TCN, etc.) to predict attention. These methods can capture richer visual information, but the extracted features contain a large amount of background information unrelated to attention and lack utilization of prior knowledge in the educational domain.

[0006] However, existing methods have the following significant limitations: 1) Although physiological characteristics are interpretable, their dimensions are limited, they lack fine-grained semantic information, and it is difficult to capture complex changes in expression; 2) The general visual features extracted by the pre-trained model contain a large amount of visual information that is irrelevant to attention (such as background, lighting, clothing, etc.), which reduces the effectiveness of the features; 3) The lack of introduction of prior knowledge in the field of education makes it difficult to establish a deep connection between visual features and the semantics of attention; 4) The multimodal feature fusion strategy is simple (such as direct splicing or weighted averaging), and fails to fully consider the complementarity and temporal dependence between different modalities. Summary of the Invention

[0007] Based on the technical problems existing in the background technology, this invention proposes a method, system and medium for detecting student classroom attention based on multimodal fusion. By integrating visual features and physiological behavioral features, and introducing a semantic alignment mechanism guided by domain knowledge, it achieves accurate and interpretable identification of students' classroom attention status.

[0008] The proposed method for detecting student classroom attention based on multimodal fusion in this invention inputs classroom video into an attention detection model and outputs attention level classification results. The training process of the attention detection model is as follows: Frame sampling and feature extraction of classroom videos yield visual feature sequences and physiological feature sequences; The visual feature sequence is temporally encoded and a learnable positional code and classification token are added to output a global video representation; the physiological feature sequence is projected onto the CLIP space through a fully connected layer, and the temporal information is extracted by an encoder to obtain the physiological temporal feature sequence. A fusion module for the attention mechanism is constructed, which uses the global video representation as the query and the physiological temporal feature sequence as the key and value. The calculated attention weights are concatenated with the global video representation to obtain the visual representation. The text embedding is obtained by concatenating and encoding a pre-set text description containing domain priors with learnable cue words; The cosine similarity between computational visual representation and text embedding is used to obtain the focus probability distribution, which is then used to classify focus levels.

[0009] Furthermore, the step of sampling frames and extracting features from the classroom video to obtain visual feature sequences and physiological feature sequences specifically involves: The classroom video is sampled frame by frame, and the face region is cropped out using a face detector. The cropped image is then input into the CLIP visual encoder to obtain frame-level feature vectors, which are used to construct a visual feature sequence. Physiological features, including gaze direction, facial action units, and head posture, are extracted from the cropped image. The extracted physiological features are then standardized and concatenated to obtain a physiological feature vector, which is used to construct a physiological feature sequence.

[0010] Furthermore, the step of extracting temporal information through the encoder to obtain the physiological temporal feature sequence specifically involves: extracting the temporal information of the physiological feature sequence through the Transformer encoder to obtain the physiological temporal feature sequence.

[0011] Furthermore, the pre-set text description containing domain priors specifically includes: High level of concentration: Students lean forward, their eyes are focused on the screen, and their facial expressions are focused; General focus: Students maintain a normal sitting posture, with occasional wandering of eyes; Mild distraction: Students exhibit distracting behaviors such as frequently turning their heads and yawning; Completely distracted: Students are completely detached from learning and engage in activities unrelated to the lesson.

[0012] Furthermore, the step of concatenating and encoding a pre-set text description containing domain priors with learnable cue words to obtain a text embedding specifically involves: Set corresponding learnable prompt words for each text description and concatenate them; The concatenated features are input into the CLIP text encoder to generate text embeddings.

[0013] Furthermore, the cosine similarity between the computational visual representation and the text embedding yields the attention probability distribution, specifically: After calculating the cosine similarity between the visual representation and each text embedding, the corresponding attention probability distribution is obtained through an activation function. The text description corresponding to the highest focus probability distribution is selected as the focus level classification result.

[0014] Furthermore, the probability distribution of focus The calculation formula is as follows: ; in, Features predicted by the attention detection model The multi-frame classroom video was used as input for the attention detection model. For visual representation, For the first A text embedding, For temperature parameters, For the first A text embedding, This represents the total number of text embeddings.

[0015] Furthermore, the step of concatenating the calculated attention weights with the global video representation to obtain the visual representation specifically involves concatenating the attention weights with the global video representation and then passing the concatenation through a multilayer perceptron to obtain the visual representation.

[0016] A computer system includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the method described above.

[0017] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor to execute the method described above.

[0018] The advantages of the student classroom attention detection method, system and medium based on multimodal fusion provided by this invention are: by deeply integrating general visual features and structured physiological features, and introducing a knowledge-guided text encoder, more accurate and interpretable attention recognition is achieved, solving the problems of lack of semantic guidance in feature representation, insufficient multimodal fusion and poor interpretability in the prior art. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a schematic diagram of the module structure of the attention detection model. Detailed Implementation

[0020] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0021] like Figure 1 and Figure 2 As shown, the student classroom attention detection method based on multimodal fusion proposed in this invention inputs classroom video into the attention detection model and outputs attention level classification results; The training process of the attention detection model is as follows: Frame sampling and feature extraction of classroom videos yield visual feature sequences and physiological feature sequences; The visual feature sequence is temporally encoded and a learnable positional code and classification token are added to output a global video representation; the physiological feature sequence is projected onto the CLIP space through a fully connected layer, and the temporal information is extracted by an encoder to obtain the physiological temporal feature sequence. A fusion module for the attention mechanism is constructed, which uses the global video representation as the query and the physiological temporal feature sequence as the key and value. The calculated attention weights are concatenated with the global video representation to obtain the visual representation. The text embedding is obtained by concatenating and encoding a pre-set text description containing domain priors with learnable cue words; The cosine similarity between computational visual representation and text embedding is used to obtain the focus probability distribution, which is then used to classify focus levels.

[0022] This embodiment deeply fuses general visual features and structured physiological features through an innovative globally perceptive frame-level fusion strategy, and introduces a text encoder that combines prior knowledge specific to the engagement domain with learnable context to achieve more accurate and interpretable attention recognition. This solves the problems of lack of semantic guidance in feature representation, insufficient multimodal fusion, and poor interpretability in existing technologies.

[0023] In one embodiment, step one involves sampling frames and extracting features from the classroom video to obtain visual feature sequences and physiological feature sequences; specifically: Multimodal feature extraction: A pre-trained CLIP-ViT model is used as the visual encoder to extract general visual features from each frame of the image. Specifically, the input video is first sampled frame by frame, then a face detector is used to crop out the facial region, and the cropped image is input into the CLIP visual encoder to obtain frame-level feature vectors. , This is the frame index for the sampled data.

[0024] It is understandable that frame-level feature vectors It is a single feature vector extracted from each frame of the video, with a dimension of 512. Visual feature sequence It is a sequence composed of multiple frame-level feature vectors arranged in chronological order, i.e. , The number of frames sampled. Therefore, the visual feature sequence is a temporal collection of frame-level feature vectors used to capture spatiotemporal information in the video.

[0025] The OpenFace-2.2.0 toolkit was used to extract physiological features, namely GAP features, from the cropped image. These physiological features include gaze direction. Facial Action Units Head pose The extracted physiological features are then standardized and concatenated to obtain a physiological feature vector. Specifically, the gaze direction, facial action units, and head posture are standardized separately (e.g., using Z-score standardization), and the standardized feature vectors are then concatenated to form the physiological feature vector. : , This is the frame index for the sampled data.

[0026] Physiological characteristic sequence It is a sequence composed of multiple physiological feature vectors arranged in chronological order, that is... .

[0027] The above multimodal feature extraction methods have the following advantages: Multimodal complementarity: Visual feature sequences (from CLIP-ViT) capture general semantic information of images (such as scene and objects), while physiological feature sequences provide fine-grained behavioral cues (such as gaze, facial expressions, and head movements). The fusion of the two can more comprehensively reflect students' attention and avoid the limitations of a single modality.

[0028] Robustness and generalization ability: The CLIP model is pre-trained on a large-scale dataset and has strong visual representation capabilities; the physiological features are based on the OpenFace toolkit, specifically for facial behavior analysis, and incorporate domain priors. This combination improves the model's robustness to changes in lighting, occlusion, etc., and enhances its generalization ability across scenes.

[0029] Temporal information capture: Both are processed in sequence form. By using a temporal encoder (such as Transformer), the dynamic changes in attention can be modeled, thereby more accurately identifying the level of attention.

[0030] Efficiency and scalability: CLIP-ViT and OpenFace are both mature tools with high feature extraction efficiency; the fusion module design is flexible and easy to extend by introducing other modal features (such as audio).

[0031] In one embodiment, step two involves temporally encoding the visual feature sequence and adding learnable positional codes and classification tokens to output a global video representation; projecting the physiological feature sequence onto the CLIP space through a fully connected layer, and extracting temporal information through an encoder to obtain a physiological temporal feature sequence, specifically: Step two specifically includes a visual path and a physiological path. The visual path is used to generate a global video representation, and the physiological path is used to generate a physiological temporal feature sequence, as detailed below.

[0032] (A) Visual path: Long-range dependency modeling of general visual features is performed using temporal Transformer, that is, the visual feature sequence is... Input a time-series Transformer encoder, and add learnable positional codes and classification tokens ( To capture long-term dependencies and output a global video representation. : ; in, It is a timing Transformer encoder.

[0033] Specifically, in visual feature sequences The beginning of the string is followed by a learnable classification token ( ), forming a new sequence ; For sequence Each element in (including) And each Adding a learnable positional encoding provides the model with information about the order of frames.

[0034] The position-encoded sequence is input into a temporal Transformer encoder, whose output is a sequence of the same length. (The corresponding...) The output vector at the location is used as a global video representation. .this It is believed to contain all the temporal information of the entire video segment.

[0035] It is understandable that, since students' attention is not an instantaneous state but a dynamic process that evolves over time, temporal Transformer is used to encode visual features, which has the following advantages: (1) Strong long-range dependency capture capability: The self-attention mechanism allows any frame in the sequence with added positional encoding to interact directly with any other frame, no matter how far apart they are. This can capture complex patterns such as "students are very focused at first, then lose focus in the middle, and finally return to the classroom"; (2) Understanding of global context: By calculating the relationship between all frames in the sequence with added positional encoding, the model can understand which frames are key frames (such as suddenly opening eyes wide or looking down for a long time), thus making a more accurate judgment on attention, rather than simply averaging the information of each frame.

[0036] In the visual feature encoding, learnable positional encoding and classification tokens are introduced. (1) The role of learnable positional encoding is to explicitly inform the model of the order of frames. Video is temporal data, and the order relationship between frames is crucial; without positional encoding, the Transformer will treat the input sequence as an unordered set. Compared to using fixed sine and cosine encoding, learnable positional encoding can adaptively find the most suitable positional representation method for the current task during training. (2) The role of learnable classification tokens is that during training, this classification token will interact with all other frame features through a self-attention mechanism, and its final output value Designed to carry global semantic information of the entire video clip, this learnable classification token provides a fixed-dimensional, efficient global representation, eliminating the need for pooling or other operations on the entire output global video representation, and can be directly used in subsequent classification or fusion modules.

[0037] (B) Physiological pathway; Physiological features are encoded and enhanced using MLP and Transformer. First, the physiological feature sequence is projected onto the CLIP space through a fully connected layer (MLP), and then input into a Transformer encoder to extract temporal information, resulting in a physiological temporal feature sequence. : .

[0038] Specifically, each physiological feature vector By using a fully connected layer (MLP), the features are projected into the same semantic space (e.g., 512-dimensional) as the CLIP visual features. This step is crucial for modality alignment because: (1) Visual features come from the CLIP model and exist in a high-level semantic space. In contrast, the original physiological features (such as gaze angle and muscle activity intensity) are low-level, physically meaningful signals. Directly fusing these two vastly different spaces (e.g., calculating attention) would be ineffective. (2) Creating a common space: By projecting physiological features into the CLIP space through the MLP, the physiological features are essentially converted into vectors with the same dimension and distribution as the visual features. This allows subsequent fusion modules to be computed in a unified semantic space, greatly improving the effectiveness of fusion.

[0039] The projected physiological feature sequence is input into another Transformer encoder (this Transformer may have the same structure as the Transformer encoder for the visual path, but its parameters are not shared). The entire sequence output by this Transformer encoder is... It is a physiological time sequence characteristic. For the first The physiological temporal features corresponding to each sampled frame. Patterns of student behavior (gaze, facial expressions, head posture) over time were captured.

[0040] Global video representation in this embodiment It provides macroscopic, contextualized semantic information (such as whether the student is facing the podium and their overall posture); physiological temporal feature sequences. It provides microscopic, precise behavioral physical signals (such as the trajectory of eye movements); the fusion of the two makes it more robust to noise and occlusion, achieving information complementarity and robustness.

[0041] In addition, by projecting physiological features into the CLIP space, the semantic gap problem commonly encountered in multimodal fusion is solved, achieving effective modality alignment and laying a solid foundation for subsequent cross-modal attention fusion.

[0042] Global video representation and physiological time sequence characteristics The generation process uses the Transformer architecture, which can fully model the continuity and evolution of attention states, rather than making independent frame-level judgments, which is more in line with the actual situation of classroom attention.

[0043] In one embodiment, step three involves constructing a fusion module for the attention mechanism. Using the global video representation as the query and the physiological temporal feature sequence as the key and value, the calculated attention weights are concatenated with the global video representation to obtain the visual representation. Specifically: Globally perceptive multimodal fusion; designing an attention-based fusion module for global video representation. To query, physiological time-series feature sequences Calculate attention weights for keys and values. : .

[0044] Attention weight With global video representation Visual representation is obtained after splicing. : ; in, For multilayer perceptrons, This is a splicing layer.

[0045] Visual representation is obtained by splicing. Its function is to achieve information complementarity and enhancement, specifically reflected in (a1) to (a2): (a1) Information Injection: It is a purely appearance-based global video representation (what the student "looks" doing). Based on The context, dynamically selected from physiological temporal feature sequences, yields the most relevant fine-grained behavioral evidence (how the student "actually" gazes, facial expressions, and head movements). and The splicing is equivalent to injecting behavioral evidence of physiological temporal feature sequences into the global video representation.

[0046] (a2) Feature enhancement and correction; Enhancement: If the visual information of the global video representation is consistent with the behavioral information of the physiological temporal feature sequence (e.g.) The students are shown facing the podium. It also captures its gaze direction towards the blackboard), so the fused features It will be reinforced, and the confidence level will be higher.

[0047] Correction: If there is ambiguity (e.g.) The student is looking down (this could be because they are reading a book or playing on their phone). Key information can be provided (if) The system captures rapid eye movements (which may indicate reading, while prolonged stillness may indicate distraction) to help the model make more accurate judgments.

[0048] This embodiment uses the attention-based fusion module described above to solve the following problems: (b1) This invention addresses the modal gap problem in multimodal fusion: traditional splicing or weighted averaging methods struggle to handle the differences in statistical distribution and semantic granularity of features across different modalities. This embodiment effectively bridges this gap at a high-level semantic level by projecting physiological features into the CLIP space and utilizing an attention mechanism for interaction.

[0049] (b2) This solves the problem of insufficient utilization of temporal information: Existing methods may only use statistical values ​​of physiological characteristics (such as mean and variance), losing dynamic information. This embodiment processes the entire physiological temporal feature sequence. Furthermore, it utilizes attention mechanisms for dynamic selection, fully exploring the information within the dynamic evolution patterns of behavior.

[0050] (b3) Improved robustness and interpretability of the model: Because the fusion is dynamic, the model is more robust to visual occlusion or quality degradation in some frames—it can instead rely more on clear behavioral evidence at other moments. Simultaneously, the attention weights are analyzed. This allows us to better understand which key moments in the behavioral sequence the model focuses on when making judgments, thus providing some interpretability to the model's decisions.

[0051] Therefore, the embodiment employs the above-described attention-based fusion module, which has the following advantages: (c1) Dynamic and adaptive fusion: Unlike early fusion (directly splicing visual feature sequences and physiological feature sequences) or late fusion (global video representation and physiological temporal feature sequences are predicted and then voted), this attention fusion is content-based and dynamic.

[0052] (c2) Addressing information overload and noise: Physiological temporal feature sequences may contain a large number of minor movements unrelated to focus (such as normal blinking or slight adjustments to posture). The attention mechanism, as a soft selector, can automatically filter out this noise and focus only on key behavioral segments that contribute to judging the current level of focus.

[0053] (c3) Capturing long-range temporal dependencies: due to the physiological temporal feature sequences as keys and values It is already a temporal feature encoded by Transformer. It inherently contains the long-range context of the behavior. Therefore, the attention mechanism is... Based on this, selection can link behavioral events that are far apart but semantically related.

[0054] In one embodiment, step four involves concatenating and encoding a pre-set text description containing domain priors with learnable cue words to extract the text embedding; specifically: Knowledge-guided text encoding; designing text descriptions incorporating domain priors for four levels of focus (high focus, moderate focus, slight distraction, and complete distraction): High concentration "Students lean forward, their eyes focused on the screen, and their facial expressions are attentive." General focus "Students maintained normal sitting posture, with occasional wandering eyes"; slight distraction "Students exhibited distracting behaviors such as frequently turning their heads and yawning." Completely distracted Students completely lose their focus on learning and engage in activities unrelated to the classroom.

[0055] Understandably, the textual descriptions of attention levels can be divided into specific numbers as needed.

[0056] The text description is concatenated with learnable cue words and input into the CLIP text encoder to obtain the text embedding. : That is, set for each level of focus. The number of learnable tokens is used as learnable cue words, for example, for high focus. Set learnable prompt words For general focus Set learnable prompt words It is a slight distraction. Set learnable prompt words For complete distraction Set learnable prompt words Similarly, for each level of focus, a corresponding learnable cue word is assigned, where... They are respectively represented as , , , The setting of the first A learnable token.

[0057] The advantages of setting learnable prompts in this embodiment are as follows (d1) to (d3): (d1) Data-driven cue optimization: Learnable cue words transform cue words from fixed strings into optimizable model parameters. During training, the model automatically adjusts the embeddings of these cue words through gradient descent to find the "optimal cue" that best guides the text encoder to generate text embeddings highly aligned with visual patterns.

[0058] (d2) Unlocking CLIP's potential: Allowing the model to fine-tune the text-side representation to better adapt it to downstream tasks without changing the weights of the CLIP text encoder itself (i.e., parameter freezing).

[0059] (d3) Enhanced representation ability: Through learned cue words, the model can generate more discriminative text embeddings than fixed descriptions. These text embeddings are able to better integrate with visual representations within a semantic space. Perform a similarity comparison.

[0060] By concatenating pre-defined text descriptions containing domain priors with learnable cue words, the following advantages are available (e1) to (e3): (e1) Provides strong priors and a convergence starting point: The textual description of the domain priors provides an initial state for the model. This ensures that the model searches in the right direction from the very beginning.

[0061] (e2) Constraint optimization space: The text description of the domain prior is used as a constraint to prevent learnable prompts from going astray.

[0062] (e3) Efficiency and effectiveness coexist: The model does not need to learn all concepts from scratch. It can be adjusted with learnable cue words on the skeleton provided by the textual description of the domain prior, and learn subtle patterns that are difficult to describe precisely in language but exist in the data.

[0063] In this embodiment, the setting of learnable prompts is highly relevant to the task, and a typical specific setting process is as follows (f1) to (f4): (f1) Determine the prompt template; Define a template string containing a [PROMPT] placeholder and a fixed domain description; for example: "A video of a student, [PROMPT], who is [fixed domain description]".

[0064] In this embodiment, the template can be viewed as: [learnable prompt words] [fixed domain description].

[0065] (f2) Initialize learnable parameters: Determine the length of the learnable prompt words (That is, the number of tokens). This is a key hyperparameter. It is usually a small number, between 1 and 10. It can be set to 8 at the beginning, which is a common initial value.

[0066] initialization There are 10 learnable cue words (each cue word has the same dimension as the word embedding dimension of the CLIP text encoder, for example, 512). These vectors are typically initialized randomly from a Gaussian or uniform distribution.

[0067] (f3) Forward propagation: For each level of focus ,Will The learnable cue words are concatenated with the text description embeddings of the fixed domain description. This concatenated sequence is then fed into a frozen CLIP text encoder. The embedding of the [EOS] (end of sentence) marker or the mean-pooled embedding of the entire sequence output by the encoder is the text embedding for that focus level. .

[0068] (f4) Optimization; During model training, learnable cue words can be optimized together with other parts of the model (such as the MLP in the fusion module, the temporal encoder, etc.) through gradient descent.

[0069] Loss function (typically cross-entropy loss, based on visual representation) With all The gradient of the probability distribution calculated from the similarity is backpropagated to these learnable cue words, guiding them on how to adjust to generate more effective results. It is more similar to the correct visual pattern and less similar to the incorrect one.

[0070] In one embodiment, step five involves calculating the cosine similarity between the visual representation and the text embedding to obtain the attention probability distribution, thereby achieving attention level classification; specifically: Computational visual representation With text embedding The cosine similarity, through The operation yields a probability distribution of focus: ; in, To measure the engagement predicted by the attention detection model, The multi-frame classroom video was used as input for the attention detection model. For visual representation, For the first A text embedding, For temperature parameters, For the first A text embedding, This represents the total number of text embeddings.

[0071] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting student classroom attention based on multimodal fusion, characterized in that, The classroom video is input into the attention detection model, and the attention level classification results are output. The training process of the attention detection model is as follows: Frame sampling and feature extraction of classroom videos yield visual feature sequences and physiological feature sequences; The visual feature sequence is temporally encoded and a learnable positional code and classification token are added to output a global video representation; the physiological feature sequence is projected onto the CLIP space through a fully connected layer, and the temporal information is extracted by an encoder to obtain the physiological temporal feature sequence. A fusion module for the attention mechanism is constructed, which uses the global video representation as the query and the physiological temporal feature sequence as the key and value. The calculated attention weights are concatenated with the global video representation to obtain the visual representation. The text embedding is obtained by concatenating and encoding a pre-set text description containing domain priors with learnable cue words; The cosine similarity between computational visual representation and text embedding is used to obtain the focus probability distribution, which is then used to classify focus levels.

2. The detection method according to claim 1, characterized in that, The process of sampling frames and extracting features from the classroom video to obtain visual feature sequences and physiological feature sequences specifically involves: The classroom video is sampled frame by frame, and the face region is cropped out using a face detector. The cropped image is then input into the CLIP visual encoder to obtain frame-level feature vectors, which are used to construct a visual feature sequence. Physiological features, including gaze direction, facial action units, and head posture, are extracted from the cropped image. The extracted physiological features are then standardized and concatenated to obtain a physiological feature vector, which is used to construct a physiological feature sequence.

3. The detection method according to claim 1, characterized in that, The step of extracting temporal information through an encoder to obtain a physiological temporal feature sequence specifically involves: extracting the temporal information of the physiological feature sequence through a Transformer encoder to obtain a physiological temporal feature sequence.

4. The detection method according to claim 1, characterized in that, The pre-set text description containing domain priors is specifically as follows: High level of concentration: Students lean forward, their eyes are focused on the screen, and their facial expressions are focused; General focus: Students maintain a normal sitting posture, with occasional wandering of eyes; Mild distraction: Students exhibit distracting behaviors such as frequently turning their heads and yawning; Completely distracted: Students are completely detached from learning and engage in activities unrelated to the lesson.

5. The detection method according to claim 4, characterized in that, The process of concatenating and encoding a pre-set text description containing domain priors with learnable cue words to obtain a text embedding is as follows: Set corresponding learnable prompt words for each text description and concatenate them; The concatenated features are input into the CLIP text encoder to generate text embeddings.

6. The detection method according to claim 1, characterized in that, The cosine similarity between the computational visual representation and the text embedding yields the attention probability distribution, specifically: After calculating the cosine similarity between the visual representation and each text embedding, the corresponding attention probability distribution is obtained through an activation function. The text description corresponding to the highest focus probability distribution is selected as the focus level classification result.

7. The detection method according to claim 6, characterized in that, Attention probability distribution The calculation formula is as follows: ； in, Features predicted by the attention detection model The multi-frame classroom video was used as input for the attention detection model. For visual representation, For the first A text embedding, For temperature parameters, For the first A text embedding, This represents the total number of text embeddings.

8. The detection method according to claim 1, characterized in that, The step of concatenating the calculated attention weights with the global video representation to obtain the visual representation specifically involves concatenating the attention weights with the global video representation and then passing the concatenation through a multilayer perceptron to obtain the visual representation.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of classification programs, which are used by a processor to execute the method as described in any one of claims 1-8.

Images