Student classroom concentration detection method and system based on multi-modal fusion and medium

Abstract

The application discloses a student classroom concentration detection method and system based on multi-modal fusion, and a medium. The method comprises the following steps: inputting a classroom video into a concentration detection model, and outputting a concentration level classification result; the training process of the concentration detection model is as follows: frame sampling and feature extraction are performed on the classroom video; encoding is performed on the features after the feature extraction to obtain global video representation and physiological time sequence features; a fusion module of an attention mechanism is constructed, visual representation is obtained by splicing the calculated attention weight and the global video representation; a text description containing domain priori and a learnable prompt word are spliced and encoded to extract text embedding; a cosine similarity of the visual representation and the text embedding is calculated to obtain a concentration probability distribution, and concentration level classification is realized according to the concentration probability distribution; and the detection method realizes accurate and interpretable identification of the concentration state of students in the classroom.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a student classroom concentration detection method and system based on multi-modal fusion and a medium. BACKGROUND

[0002] Student classroom concentration detection is a core research direction in the field of educational artificial intelligence, and its goal is to automatically identify the concentration state level of students in the classroom environment by analyzing their multi-modal behavior performance. This technology has important application value in intelligent teaching systems, online education platforms, classroom quality evaluation, personalized learning intervention, and other scenarios.

[0003] Currently, the mainstream concentration detection methods mainly include the following two categories:

[0004] 1. Method based on physiological behavior characteristics: This method extracts physiological features such as student gaze direction, facial action units, and head pose through a special toolkit (such as OpenFace), and then uses a time series model (such as GRU, LSTM, or Transformer) for feature modeling and state classification. The advantage of this method is that the features have good interpretability, but the disadvantage is that it only relies on low-dimensional physiological signals, lacks rich semantic information, and is highly dependent on the accuracy of the feature extraction tool.

[0005] 2. Method based on deep learning pre-trained model: This method uses a visual model (such as ResNet, EfficientNet, ViT, etc.) pre-trained on a large-scale image dataset to extract deep features of video frames, and then uses a time series modeling network (such as Transformer, TCN, etc.) for concentration prediction. This method can capture more rich visual information, but the extracted features contain a lot of irrelevant background information, and lacks the use of prior knowledge in the education field.

[0006] However, the existing methods have the following significant limitations:

[0007] 1) Although physiological features have interpretability, they have limited feature dimensions and lack fine-grained semantic information, making it difficult to capture complex facial changes;

[0008] 2) The general visual features extracted by the pre-trained model contain a lot of irrelevant visual information (such as background, lighting, clothing, etc.) for concentration, reducing the effectiveness of the features;

[0009] 3) Lack of introduction of prior knowledge in the education field, making it difficult to establish a deep connection between visual features and concentration semantics;

[0010] 4) The multi-modal feature fusion strategy is simple (such as direct splicing or weighted average), and the complementarity and time sequence dependence between different modalities are not fully considered. SUMMARY

[0011] Based on the technical problems existing in the background art, the present application proposes a student classroom concentration detection method, system and medium based on multi-modal fusion, which realizes precise and interpretable recognition of student classroom concentration state by fusing visual features and physiological behavior features and introducing a domain knowledge guided semantic alignment mechanism.

[0012] The student classroom concentration detection method based on multi-modal fusion proposed by the present application inputs the classroom video into the concentration detection model and outputs the concentration level classification result;

[0013] The training process of the concentration detection model is as follows:

[0014] Frame sampling and feature extraction are performed on the classroom video to obtain a visual feature sequence and a physiological feature sequence;

[0015] The visual feature sequence is time sequence encoded, and a learnable position encoding and a classification token are added, and a global video representation is output; the physiological feature sequence is projected to the CLIP space through a fully connected layer, and the time sequence information is extracted through an encoder to obtain a physiological time sequence feature sequence;

[0016] A fusion module with an attention mechanism is constructed, the global video representation is taken as a query, and the physiological time sequence feature sequence is taken as a key and a value, the calculated attention weight is spliced with the global video representation to obtain a visual representation;

[0017] A text description containing domain priori knowledge and a learnable prompt word are spliced and encoded to obtain a text embedding;

[0018] The cosine similarity of the visual representation and the text embedding is calculated to obtain a concentration probability distribution, based on which the concentration level classification is realized.

[0019] Further, the frame sampling and feature extraction of the classroom video to obtain the visual feature sequence and the physiological feature sequence are specifically:

[0020] Frame sampling is performed on the classroom video, a face detector is used to crop the face region, the cropped image is input into a CLIP visual encoder to obtain a frame-level feature vector, and based on this, a visual feature sequence is constructed;

[0021] The physiological features including gaze direction, facial action unit and head posture are extracted from the cropped image, the extracted physiological features are standardized and spliced to obtain a physiological feature vector, and based on this, a physiological feature sequence is constructed.

[0022] Further, the time sequence information is extracted by the encoder to obtain a physiological time sequence feature sequence, specifically, time sequence information of the physiological feature sequence is extracted by a Transformer encoder to obtain the physiological time sequence feature sequence.

[0023] Further, the pre-set text description containing domain priori is specifically:

[0024] High concentration: the student leans forward, eyes focus on the screen, and facial expressions are concentrated;

[0025] General concentration: the student maintains a normal sitting posture, and occasionally has eye wandering;

[0026] Minor distraction: the student has frequent head turning and yawning distraction behaviors;

[0027] Complete distraction: the student completely deviates from the learning state and performs activities unrelated to the classroom.

[0028] Further, the pre-set text description containing domain priori is specifically:

[0029] Set the corresponding learnable prompt word for each text description and splice;

[0030] The spliced features are input into a CLIP text encoder to generate text embeddings.

[0031] Further, the cosine similarity between the visual representation and the text embedding is calculated to obtain the concentration probability distribution, specifically:

[0032] The cosine similarity between the visual representation and each text embedding is calculated respectively, and the corresponding concentration probability distribution is obtained through an activation function;

[0033] Select the text description corresponding to the maximum concentration probability distribution as the concentration level classification result.

[0034] Further, the concentration probability distribution The calculation formula is as follows:

[0035] ;

[0036] Wherein, is the feature predicted by the concentration detection model, is the input to the concentration detection model multiple frames of classroom video, is the visual representation, is the text embedding, is the temperature parameter, is the text embedding, Total number of text embeddings.

[0037] Further, the visual representation is obtained by splicing the calculated attention weight and the global video representation, specifically: the attention weight and the global video representation are spliced and then a multi-layer perceptron is used to obtain the visual representation.

[0038] A computer system comprising a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to implement the method as described above.

[0039] A computer-readable storage medium having a plurality of classification programs stored thereon, the plurality of classification programs being configured to be invoked by a processor and to execute the method as described above.

[0040] The student classroom concentration detection method and system based on multi-modal fusion provided by the present application have the advantages that: by deeply fusing general visual features and structured physiological features, and introducing a knowledge-guided text encoder, more accurate and interpretable concentration recognition is achieved, and problems such as lack of semantic guidance in feature representation, insufficient multi-modal fusion, and poor interpretability in the prior art are solved. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a structural schematic diagram of the present application;

[0042] Figure 2 is a module structure schematic diagram of the concentration detection model. DETAILED DESCRIPTION

[0043] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the present application. The following description is not intended to limit the scope of the present application, but to provide an exemplary embodiment of the present application.

[0044] As shown in Figure 1 and Figure 2 The student classroom concentration detection method based on multi-modal fusion provided by the present application inputs a classroom video into a concentration detection model and outputs a concentration level classification result.

[0045] The training process of the concentration detection model is as follows:

[0046] Frame sampling and feature extraction are performed on the classroom video to obtain a visual feature sequence and a physiological feature sequence.

[0047] The visual feature sequence is time-coded, and a learnable position coding and a classification token are added to output a global video representation; the physiological feature sequence is projected to a CLIP space through a fully connected layer, and time sequence information is extracted through an encoder to obtain a physiological time sequence feature sequence;

[0048] A fusion module of an attention mechanism is constructed, the global video representation is taken as a query, the physiological time sequence feature sequence is taken as a key and a value, the calculated attention weight is spliced with the global video representation to obtain a visual representation;

[0049] A text description containing domain priori is spliced with a learnable prompt word and is encoded to obtain a text embedding;

[0050] A cosine similarity between the visual representation and the text embedding is calculated to obtain a concentration probability distribution, based on which a concentration level classification is realized.

[0051] The embodiment deeply fuses general visual features and structured physiological features through an innovative global perception frame-level fusion strategy, and introduces a text encoder combining participation domain-specific priori knowledge and learnable context, so as to realize more accurate and interpretable concentration recognition, and solve the problems of lack of semantic guidance in feature representation, insufficient multi-modal fusion and poor interpretability in the prior art.

[0052] In one of the embodiments, step one, frame sampling and feature extraction are performed on the classroom video to obtain a visual feature sequence and a physiological feature sequence; specifically:

[0053] Multi-modal feature extraction; a pre-trained CLIP-ViT model is used as a visual encoder to extract general visual features from each frame of image. Specifically, first, the input video is frame-sampled, and then a face detector is used to crop the face region, and the cropped image is input into the CLIP visual encoder to obtain a frame-level feature vector , is the index of the sampled frame.

[0054] It can be understood that the frame-level feature vector is a single feature vector extracted from each frame of image in the video, and has a dimension of 512. The visual feature sequence is a sequence composed of a plurality of frame-level feature vectors arranged in time sequence, that is , is the number of sampled frames. Therefore, the visual feature sequence is a time sequence set of frame-level feature vectors, which is used to capture the space-time information in the video.

[0055] The OpenFace-2.2.0 toolkit is used to extract physiological features, i.e. GAP features, from the cropped image, and the physiological features include gaze direction Facial Action Units (FAUs) and Head Pose After the physiological features are extracted, they are standardized and concatenated to form the physiological feature vector. That is, the gaze direction, facial action units, and head pose are standardized (e.g., Z-score normalization), and the standardized feature vectors are concatenated to form the physiological feature vector

[0056] is the index of the sampled frame.

[0057] Physiological Feature Sequence is a sequence composed of multiple physiological feature vectors arranged in chronological order, i.e. .

[0058] The above multi-modal feature extraction method has the following advantages:

[0059] Multi-modal complementarity: The visual feature sequence (from CLIP-ViT) captures general semantic information of the image (such as scene, object), while the physiological feature sequence provides fine-grained behavioral cues (such as gaze, expression, head movement), the fusion of the two can more comprehensively reflect the students' concentration, avoiding the limitations of a single modality.

[0060] Robustness and generalization ability: CLIP model is pre-trained on a large-scale dataset, with strong visual representation ability; physiological features are based on the OpenFace toolkit, specifically for facial behavior analysis, combining domain prior. This combination improves the robustness of the model to changes such as light and occlusion, and enhances the generalization ability across scenes.

[0061] Temporal information capture: Both are processed in sequence, through the temporal encoder (such as Transformer) to model the dynamic changes of concentration, so as to more accurately identify the concentration level.

[0062] Efficiency and scalability: CLIP-ViT and OpenFace are mature tools with high feature extraction efficiency; the fusion module design is flexible and easy to introduce other modalities (such as audio) for expansion.

[0063] In one embodiment, step two, the visual feature sequence is temporally encoded, and a learnable position encoding and classification token are added, outputting a global video representation; the physiological feature sequence is projected to the CLIP space through a fully connected layer, and the temporal information is extracted through the encoder to obtain the physiological temporal feature sequence, specifically:

[0064] ​​Step two specifically includes a visual path for generating global video representation and a physiological path for generating physiological time series features, as follows.

[0065] (A) Visual path:

[0066] Long-range dependencies of general visual features are modeled by using time series Transformer, i.e., the visual feature sequence is input into a time series Transformer encoder with learnable position encoding and classification token to capture long-range dependencies, and the output global video representation is:

[0067] ;

[0068] where is the time series Transformer encoder.

[0069] Specifically, a learnable classification token is added at the beginning of the visual feature sequence to form a new sequence ;

[0070] Each element in the sequence (including and each ) is added with a learnable position encoding, which provides the model with the sequential information of frames.

[0071] The sequence with position encoding is input into a time series Transformer encoder, and the output of the time series Transformer encoder is a sequence of the same length. The output vector at the corresponding position is taken as the global video representation . This is considered to condense all the time series information of the entire video segment.

[0072] It is understandable that, since students' attention is not an instantaneous state but a dynamic process that evolves over time, temporal Transformer is used to encode visual features, which has the following advantages: (1) Strong long-range dependency capture capability: The self-attention mechanism allows any frame in the sequence with added positional encoding to interact directly with any other frame, no matter how far apart they are. This can capture complex patterns such as "students are very focused at first, then lose focus in the middle, and finally return to the classroom"; (2) Understanding of global context: By calculating the relationship between all frames in the sequence with added positional encoding, the model can understand which frames are key frames (such as suddenly opening eyes wide or looking down for a long time), thus making a more accurate judgment on attention, rather than simply averaging the information of each frame.

[0073] In the visual feature encoding, learnable positional encoding and classification tokens are introduced. (1) The role of learnable positional encoding is to explicitly inform the model of the order of frames. Video is temporal data, and the order relationship between frames is crucial; without positional encoding, the Transformer will treat the input sequence as an unordered set. Compared to using fixed sine and cosine encoding, learnable positional encoding can adaptively find the most suitable positional representation method for the current task during training. (2) The role of learnable classification tokens is that during training, this classification token will interact with all other frame features through a self-attention mechanism, and its final output value Designed to carry global semantic information of the entire video clip, this learnable classification token provides a fixed-dimensional, efficient global representation, eliminating the need for pooling or other operations on the entire output global video representation, and can be directly used in subsequent classification or fusion modules.

[0074] (B) Physiological pathway;

[0075] Physiological features are encoded and enhanced using MLP and Transformer. First, the physiological feature sequence is projected onto the CLIP space through a fully connected layer (MLP), and then input into a Transformer encoder to extract temporal information, resulting in a physiological temporal feature sequence. :

[0076] .

[0077] Specifically, each physiological feature vector Through a fully connected layer (MLP), it is projected into the same semantic space as the CLIP visual features (e.g., 512 dimensions). This step is the key to modal alignment. Because: (1) The visual features come from the CLIP model and exist in a high-level semantic space. The original physiological features (such as gaze angle, muscle activity intensity) are low-level, physically meaningful signals. Direct fusion (such as calculating attention) in these two vastly different spaces will be poor. (2) Create a common space: By projecting the physiological features into the CLIP space through the MLP, the physiological features are essentially converted into vectors with the same dimension and distribution as the visual features. This allows the subsequent fusion module to perform calculations in a unified semantic space, greatly improving the effectiveness of fusion.

[0078] The projected physiological feature sequence is input into another Transformer encoder (this Transformer may have the same structure as the Transformer encoder in the visual path, but the parameters are not shared). The entire sequence output by the Transformer encoder is the physiological time series feature sequence, which is the physiological time series feature corresponding to the frame sampled by the th. This captures the pattern of student behavior (gaze, expression, head posture) over time.

[0079] The global video representation of this embodiment provides macroscopic, scene-based semantic information (such as whether the student is facing the podium, overall posture); the physiological time series feature sequence provides microscopic, precise behavioral physical signals (such as the trajectory of eye movement); the fusion of the two is more robust to noise and occlusion, achieving information complementarity and robustness.

[0080] In addition, by projecting the physiological features into the CLIP space, the semantic gap problem commonly encountered in multi-modal fusion is solved, achieving effective modal alignment and laying a solid foundation for subsequent cross-modal attention fusion.

[0081] The generation process of the global video representation and the physiological time series feature sequence both use the Transformer architecture, which can fully model the continuity and evolution of the concentration state, rather than making independent frame-level judgments, which is more in line with the actual situation of classroom concentration.

[0082] In one of the embodiments, step three, the attention mechanism fusion module is constructed, using the global video representation as the query, the physiological time series feature sequence as the key and value, and the calculated attention weight is concatenated with the global video representation to obtain the visual representation, specifically:

[0083] Global-aware multimodal fusion; design a fusion module based on attention mechanism to global video representation For query, physiological time-series feature sequence For key and value, calculate attention weight

[0084] .

[0085] Attention weight with global video representation After splicing, get visual representation

[0086] ;

[0087] Wherein, is a multilayer perceptron, is a splicing layer.

[0088] Visual representation is obtained by splicing , which realizes the complementation and enhancement of information, which is embodied in (a1) to (a2):

[0089] (a1) Information injection: is a pure, appearance-based global video representation (what the student "seems" to do). is the most relevant fine-grained behavior evidence dynamically selected from the physiological time-series feature sequence based on the context (how the student's "actual" gaze, expression, head movement is). Splice with , which is equivalent to injecting the behavior evidence of the physiological time-series feature sequence into the global video representation.

[0090] (a2) Feature enhancement and correction;

[0091] Enhancement: If the visual information of the global video representation and the behavior information of the physiological time-series feature sequence are consistent (for example shows that the student is facing the podium, and also captures its gaze direction towards the blackboard), then the fused feature will be strengthened, with higher confidence.

[0092] Correction: If there is ambiguity (for example shows that the student is looking down, which may be looking at the book or playing the mobile phone), then can provide key information (if ​​​What is captured is a quick eye saccade that can be reading if it is long stationary can be daydreaming) to help the model make more accurate judgments.

[0093] The embodiment adopts the above-mentioned fusion module based on attention mechanism, and solves the following problems:

[0094] (b1) The modality gap problem in multi-modal fusion is solved: the traditional splicing or weighted average method is difficult to handle the difference in statistical distribution and semantic granularity of different modal features. The embodiment bridges this gap at the high-level semantic level by projecting physiological features into the CLIP space and using the attention mechanism for interaction.

[0095] (b2) The problem of insufficient use of time series information is solved: existing methods may only use statistical values (such as mean, variance) of physiological features, losing dynamic information. The embodiment processes the entire physiological time series feature sequence and uses the attention mechanism for dynamic selection, fully mining the information in the dynamic evolution pattern of behavior.

[0096] (b3) The robustness and explainability of the model are improved: since the fusion is dynamic, the model is more robust to visual occlusion or quality decline of some frames - it can rely more on other clear behavior evidence at other times. At the same time, analyzing the attention weight can better understand which key moments in the behavior sequence the model focuses on when making judgments, providing a certain explainability for model decision-making.

[0097] Therefore, the embodiment adopts the above-mentioned fusion module based on attention mechanism, which has the following advantages:

[0098] (c1) Dynamic and adaptive fusion: unlike early fusion (directly splicing visual feature sequence and physiological feature sequence) or late fusion (global video representation and physiological time series feature sequence each predicts and then votes), this attention fusion is content-based and dynamic.

[0099] (c2) Solve information overload and noise: the physiological time series feature sequence may contain a large number of small actions (such as normal blinking, slight adjustment of sitting posture) that are irrelevant to concentration. The attention mechanism as a soft selector can automatically filter out these noises and only focus on key behavior segments that contribute to the current concentration.

[0100] (c3) Capture long-range temporal dependencies: since the physiological time series feature sequence itself is a time series feature encoded by Transformer, it already contains the long-range context of behavior. Therefore, the attention mechanism is in Based on this, selection can link behavioral events that are far apart but semantically related.

[0101] In one embodiment, step four involves concatenating and encoding a pre-set text description containing domain priors with learnable cue words to extract the text embedding; specifically:

[0102] Knowledge-guided text encoding; designing text descriptions incorporating domain priors for four levels of focus (high focus, moderate focus, slight distraction, and complete distraction):

[0103] High concentration "Students lean forward, their eyes focused on the screen, and their facial expressions are attentive."

[0104] General focus "Students maintained normal sitting posture, with occasional wandering eyes";

[0105] slight distraction "Students exhibited distracting behaviors such as frequently turning their heads and yawning."

[0106] Completely distracted Students completely lose their focus on learning and engage in activities unrelated to the classroom.

[0107] Understandably, the textual descriptions of attention levels can be divided into specific numbers as needed.

[0108] The text description is concatenated with learnable cue words and input into the CLIP text encoder to obtain the text embedding. :

[0109] That is, set for each level of focus. The number of learnable tokens is used as learnable cue words, for example, for high focus. Set learnable prompt words For general focus Set learnable prompt words It is a slight distraction. Set learnable prompt words For complete distraction Set learnable prompt words Similarly, for each level of focus, a corresponding learnable cue word is assigned, where... They are respectively represented as , , , The setting of the first A learnable token.

[0110] The advantages of setting learnable prompts in this embodiment are as follows (d1) to (d3):

[0111] (d1) Data-driven prompt optimization: Learnable prompts transform the prompt from a fixed string of characters into an optimizable model parameter. During training, the model automatically adjusts the embedding of these prompts through gradient descent, finding the "optimal prompt" that best guides the text encoder to generate text embeddings that are highly aligned with the visual patterns.

[0112] (d2) Unleashing the potential of CLIP: Allows the model to fine-tune the representation on the text side, making it better suited for downstream tasks without changing the weights of the CLIP text encoder itself (i.e., parameter freezing).

[0113] (d3) Enhancing representation capacity: With learned prompts, the model can generate text embeddings that are more discriminative than fixed descriptions These text embeddings can better compare similarity with visual representations in a semantic space.

[0114] By concatenating the pre-set text description containing domain prior with the learnable prompt, the following advantages (e1) to (e3) are achieved:

[0115] (e1) Provide strong prior and convergence starting point: The domain prior text description provides an initial state for the model. Ensures that the model starts searching in the right direction from the beginning.

[0116] (e2) Constrain optimization space: The domain prior text description serves as a constraint to prevent the learnable prompt from deviating.

[0117] (e3) Efficiency and effectiveness coexist: The model does not need to learn all concepts from scratch. Can adjust through learnable prompts on the skeleton provided by the domain prior text description, learn subtle patterns that are difficult to accurately describe in language but exist in data.

[0118] In this embodiment, the setting of the learnable prompt is highly related to the task, and a typical specific setting process is (f1) to (f4):

[0119] (f1) Determine the prompt template;

[0120] Define a template string that contains a [PROMPT] placeholder and a fixed domain description; for example: "A video of a student, [PROMPT], who is [fixed domain description]".

[0121] In this embodiment, the template can be seen as: [learnable prompt] [fixed domain description].

[0122] (f2) Initialize learnable prompts:

[0123] Determine the length of learnable prompts (i.e., the number of tokens). This is a key hyperparameter, which is usually a small number, between 1 and 10. It can be set to 8 at the beginning, which is a common initial value.

[0124] Initialize learnable prompts (the dimension of each prompt is the same as the word embedding dimension of the CLIP text encoder, e.g., 512). These vectors are usually randomly initialized from a Gaussian distribution or a uniform distribution.

[0125] (f3) Forward propagation:

[0126] For each level of focus , concatenate learnable prompts with the text description embedding of the fixed domain description. Send this concatenated sequence into the frozen CLIP text encoder. The embedding of the [EOS] (End of Sentence) token or the mean-pooled embedding of the entire sequence output by the encoder is the text embedding for this level of focus .

[0127] (f4) Optimization;

[0128] During model training, learnable prompts are optimized together with other parts of the model (such as the MLP of the fusion module, the temporal encoder, etc.) through gradient descent.

[0129] The gradient of the loss function (usually cross-entropy loss, based on the probability distribution calculated from the similarity between visual representation and all text embeddings) is backpropagated to these learnable prompts, guiding them on how to adjust so that the generated is more similar to the correct visual pattern and less similar to the incorrect one.

[0130] In one embodiment, step five, calculating the cosine similarity between visual representation and text embedding to get the focus probability distribution, is used to implement focus level classification; specifically:

[0131] Calculate the cosine similarity between visual representation and text embedding , get the focus probability distribution by operation:

[0132] ;

[0133] where, To measure the engagement predicted by the attention detection model, The multi-frame classroom video was used as input for the attention detection model. For visual representation, For the first A text embedding, For temperature parameters, For the first A text embedding, This represents the total number of text embeddings.

[0134] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting student concentration in class based on multi-modal fusion, characterized in that, Inputting the classroom video into the attention detection model, and outputting an attention level classification result; The training process of the attention detection model is as follows: Frame sampling and feature extraction are performed on the classroom video to obtain a visual feature sequence and a physiological feature sequence; The visual feature sequence is time-series encoded, and learnable position encoding and classification tokens are added, and a global video representation is output; the physiological feature sequence is projected to the CLIP space through a fully connected layer, and the time-series information is extracted through an encoder to obtain a physiological time-series feature sequence; A fusion module with an attention mechanism is constructed, taking the global video representation as the query, and the physiological time-series feature sequence as the key and value, and the calculated attention weight is spliced with the global video representation to obtain a visual representation; A pre-set text description containing domain priori and a learnable prompt word are spliced and encoded to obtain a text embedding; The cosine similarity between the visual representation and the text embedding is calculated to obtain an attention probability distribution, based on which the attention level classification is realized.

2. The detection method according to claim 1, characterized in that, The frame sampling and feature extraction of the classroom video to obtain the visual feature sequence and the physiological feature sequence are as follows: Frame sampling is performed on the classroom video, and a face detector is used to crop the face region, and the cropped image is input into the CLIP visual encoder to obtain a frame-level feature vector, based on which a visual feature sequence is constructed; The physiological features including gaze direction, facial action unit and head posture are extracted from the cropped image, and the extracted physiological features are standardized and spliced to obtain a physiological feature vector, based on which a physiological feature sequence is constructed.

3. The method of claim 1, wherein The physiological time-series feature sequence is obtained by extracting the time-series information of the physiological feature sequence through the encoder.

4. The method of claim 1, wherein The pre-set text description containing domain priori is as follows: High attention: the student leans forward, focuses his eyes on the screen, and has a focused facial expression; General attention: the student maintains a normal sitting posture, and occasionally has eye wandering; Light distraction: the student has frequent head turning and yawning distraction behaviors; Complete distraction: the student completely deviates from the learning state and performs activities unrelated to the classroom.

5. The detection method according to claim 4, characterized in that, The pre-set text description containing domain priori and the learnable prompt word are spliced and encoded to obtain the text embedding, which is as follows: Set the corresponding learnable prompt word for each text description and splice them; Input the spliced features into the CLIP text encoder to generate the text embedding.

6. The method of claim 1, wherein, The cosine similarity between the visual representation and the text embedding is calculated to obtain the attention probability distribution, which is as follows: The cosine similarity between the visual representation and each text embedding is calculated respectively, and the corresponding attention probability distribution is obtained through an activation function; Select the text description corresponding to the maximum attention probability distribution as the attention level classification result.

7. The detection method according to claim 6, characterized in that, Focus probability distribution The calculation formula is as follows: ； wherein, features predicted by the focus detection model, a plurality of frames of classroom video input to the focus detection model, a visual representation, a first text embedding, a first text embedding, a temperature parameter, a first text embedding, a first text embedding, a total number of text embeddings.

8. The method of claim 1, wherein, The attention weight is spliced with the global video representation to obtain the visual representation.

9. A computer system comprising a memory, a processor and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-8.

10. A computer readable storage medium, characterized in that, The computer readable storage medium has stored thereon a number of classification programs for being called by the processor and performing the method according to any one of claims 1-8.

Images