Classroom student attention assessment method, device and equipment and storage medium

Abstract

The invention provides a classroom student attention assessment method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence education application. The method comprises the following steps: acquiring a classroom teaching video clip; carrying out individual segmentation on the student individual masks to obtain a plurality of student individual masks, and then extracting initial individual part motion features of the part of each student individual in a time window; performing splicing according to the initial individual part motion features to obtain spliced fusion features, and then generating target individual part motion features of the part of each student individual; performing feature extraction on the target individual part motion features to obtain individual part attention features; and according to the attention features of the individual parts, individual attention scores of the student individuals are calculated, and then group aggregation calculation is carried out to obtain group attention scores of the whole classroom. According to the invention, the accuracy and stability of evaluating the attention of the students in the classroom can be improved.

Description

A method, device, equipment, and storage medium for assessing student attention in the classroom. Technical Field

[0001] This application relates to the field of artificial intelligence education application technology, as well as the field of video behavior understanding, and in particular to a method, device, equipment, and storage medium for assessing classroom student attention. Background Technology

[0002] In the field of AI education, classroom teaching videos, as objective records of the teaching process, are widely used for teaching quality assessment, learning outcome analysis, and student attention monitoring. Compared to subjective or offline methods such as questionnaires and random sampling, video-based data-driven methods have the advantages of being contactless, traceable, and having broad coverage. They can provide teachers with fine-grained and quantifiable classroom feedback, supporting evidence-based teaching optimization and timely intervention. In recent years, deep learning-driven computer vision and video understanding technologies have continued to evolve. The maturity of technologies such as object detection, human pose estimation, head / gaze estimation, semantic / instance / panoptic segmentation, and sequence modeling (such as RNN / Transformer) provides a feasible technical path for student attention modeling and teaching intervention based on classroom teaching videos.

[0003] The existing automated assessment paths for student attention levels in classroom teaching scenarios mainly fall into three categories: First, relying on wearable sensors such as eye trackers, head-mounted cameras, or physiological monitoring devices. Although this type of solution can obtain high spatiotemporal resolution and individualized signals, it is highly interfering, has high deployment and maintenance costs, and will significantly affect students' learning status in the classroom and is difficult to use on a large scale in real classroom teaching scenarios. Second, based on pure vision, head orientation, gaze area estimation, micro-expression recognition, and action behavior recognition, this type of solution models and analyzes a key part of the human body, and reflects the person's attention level by quantitatively calculating the visual focus area or qualitatively judging the behavior type. While offering advantages such as being contactless and easy to deploy, it is sensitive to frontal visibility, lighting, and occlusion, and lacks robustness in scenarios with multiple occlusions, side views, or back views. Analyzing only single-frame image features also leads to instability in individual time-series information. Thirdly, end-to-end video segment classification methods directly input cropped video slices into RNN / Transformer models to predict the overall classroom's "focus / lack of focus." While simplifying the assessment process, this approach offers very limited fine-grained explanations at the individual level, making it difficult to pinpoint "why there is a lack of focus," and it also struggles to balance group statistics with individual temporal consistency. Furthermore, existing technological approaches do not adequately utilize temporal continuity and dynamic group interactions, failing to guarantee accuracy and stability.

[0004] Therefore, how to provide a method for assessing student attention in the classroom and improve the accuracy and stability of the assessment has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to provide a method, device, electronic device, and storage medium for assessing classroom student attention, which can improve the accuracy and stability of assessing classroom student attention.

[0006] To achieve the above objectives, a first aspect of this application proposes a method for assessing classroom student attention. The method includes: acquiring classroom teaching video segments from a classroom teaching video; segmenting each video frame of the classroom teaching video segment to obtain multiple student individual masks; extracting initial individual part motion features of each student individual's body part within a time window from the classroom teaching video segment using segmented sliding windows and the student individual masks; splicing together the initial individual part motion features of all student individuals' body parts within all time windows to obtain splicing and fusion features; generating target individual part motion features of each student individual's body part based on the student individual's individual code, the time window code of the time window, the body part code, and the splicing and fusion features; extracting features from the target individual part motion features using a motion-aware Transformer network to obtain individual part attention features of the student individual; calculating the individual attention score of the student individual based on the individual part attention features; and performing group aggregation calculation based on the individual attention scores of each student individual to obtain the overall class attention score.

[0007] Optionally, the step of segmenting the classroom teaching video clip into multiple student individual masks includes: performing human detection on the video frames of the classroom teaching video clip to obtain a first candidate box set; filtering the first candidate boxes in the first candidate box set to obtain a second candidate box set; segmenting the second candidate boxes in the second candidate box set using a general segmentation model to obtain a candidate mask for each second candidate box; wherein, pixels with a value of 1 in the candidate mask belong to student individuals, and pixels with a value of 0 do not belong to student individuals; evaluating the confidence of the candidate masks to obtain mask confidence; and filtering the candidate masks based on the mask confidence to obtain the student individual masks.

[0008] Optionally, the confidence evaluation of the candidate mask to obtain the mask confidence includes: calculating the integrity of the candidate mask by using the number of pixels with a value of 1 in the candidate mask as the numerator and the area of ​​the second candidate box as the denominator; sequentially performing grayscale conversion and Gaussian filtering on the video frame to obtain a denoised video frame; calculating the Laplacian response of each pixel within the range of pixels with a value of 1 in the candidate mask based on the denoised video frame; performing variance statistics based on the Laplacian responses of all pixels to obtain the sharpness of the candidate mask; calculating the overlap between the candidate mask and multiple other candidate masks in the same video frame one by one to obtain multiple overlap degrees, and taking the largest of the multiple overlap degrees as the occlusion degree of the candidate mask; and calculating the mask confidence based on the integrity, the sharpness, and the occlusion degree.

[0009] Optionally, the step of extracting the initial individual part motion features of each student's part within a time window from the classroom teaching video clip using segmented sliding windows and the student individual mask includes: extracting the initial individual part temporal motion features and initial individual part frequency domain motion features of each student's part within a time window from the classroom teaching video clip based on the centroid of each part of the student and the segmented sliding windows; and fusing the initial individual part temporal motion features and the initial individual part frequency domain motion features of each time window to obtain the initial individual part motion features.

[0010] Optionally, generating the target individual part motion feature of each student's part based on the student's individual code, the time window code of the time window, the part code of the part, and the splicing and fusion feature includes: projecting the splicing and fusion feature through a linear layer to obtain a projection vector; adding the projection vector to the bias vector to obtain a feature tensor; and adding the feature tensor, the student's individual code, the time window code of the time window, and the part code of the part to obtain the target individual part motion feature of the student's part.

[0011] Optionally, calculating the individual attention score of the student based on the individual part attention characteristics includes: scoring the individual part attention characteristics using a multilayer perceptron to obtain a part attention score; and performing a weighted summation based on the part weights of the student's parts and the part attention score to obtain the individual attention score.

[0012] Optionally, the step of performing group aggregation calculation based on the individual attention scores of each student to obtain the overall class group attention score includes: calculating individual weights based on the individual student masks and the confidence levels of the individual student masks; and performing weighted fusion based on the individual weights and the individual attention scores to obtain the group attention score.

[0013] To achieve the above objectives, a second aspect of this application proposes a classroom student attention assessment device, comprising: a video segment acquisition module for acquiring classroom teaching video segments from a classroom teaching video; an individual segmentation module for segmenting each video frame of the classroom teaching video segments into multiple student individual masks; a motion feature extraction module for extracting the initial individual part motion features of each student individual's body part within a time window from the classroom teaching video segments using a segmented sliding window and the student individual masks; and a feature fusion module for splicing together the initial individual part motion features of all student individuals' body parts within all time windows to obtain spliced ​​fused features. Based on the individual student's code, the time window code of the time window, the part code of the part, and the splicing and fusion features, a target individual part motion feature is generated for each student's part; an attention feature extraction module is used to extract features from the target individual part motion feature using a motion-aware Transformer network to obtain the individual part attention feature of the student; an individual evaluation module is used to calculate the individual attention score of the student based on the individual part attention feature; and a group evaluation module is used to perform group aggregation calculation based on the individual attention scores of each student to obtain the overall class group attention score.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a classroom student attention assessment method as described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the classroom student attention assessment method described in the first aspect.

[0016] This application proposes a method, device, electronic device, and storage medium for assessing classroom student attention. It first segments individuals, extracts key body parts (such as head, shoulders, arms, and upper torso) for motion trajectory extraction and amplitude modeling, and tracks the positional changes of these parts over time. This quantifies the amplitude and frequency of student behavior, allowing for quantitative analysis of individual attention and ultimately, the overall group attention in the classroom. More specifically, this application introduces the motion characteristics of individual student parts, constructs a {part × band × time} temporal feature tensor containing indicators such as trajectory amplitude, dominant frequency / band energy, and stability, and adaptively weights it according to observation quality. Using these features, high-frequency micro-motions and low-frequency posture drift are modeled on a Transformer architecture. Then, through quality-gated cross-branch attention fusion, a more robust individual attention representation to occlusion and viewpoint changes is obtained, thus more robustly characterizing the overall student attention level. Furthermore, this application achieves classroom student attention assessment based on instance-level person segmentation and body part kinematic features, making it highly suitable for non-contact attention feature quantification and real-time feedback in fixed-camera classroom teaching videos.

[0017] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0018] Figure 1 is a flowchart of a classroom student attention assessment method provided in an embodiment of this application; Figure 2 is a schematic diagram of the character layout provided in an embodiment of this application; Figure 3 is a flowchart of step 102 in Figure 1; Figure 4 is a flowchart of step 304 in Figure 3; Figure 5 is a flowchart of step 103 in Figure 1; Figure 6 is a schematic diagram of the overall architecture of the MAT network provided in an embodiment of this application; Figure 7 is a schematic diagram of a classroom student attention assessment device provided in an embodiment of this application; Figure 8 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] This application aims to overcome the limitations of existing attention assessment paths and, for the task of assessing students' overall attention in classroom teaching scenarios, designs a method for extracting motion features of human body parts based on a general segmentation model. This method enables non-contact attention assessment of individual students in complex classroom settings. Furthermore, this application designs a Motion-Aware Transformer (MAT) network based on body part motion features. Unlike network architectures that use pixel tubelets (RGB / optical flow) as tokens, MAT first extracts motion features from human body part-level sequences and then maps them to fast branch tokens (high frequency, short window) and slow branch tokens (low frequency, long window). A quality factor is introduced into the MAT network structure to perform gated fusion of cross-attention (Fast→Slow) between the two branches. Under complex classroom conditions such as multiple occlusions, side profiles, long-distance low-resolution, and lighting changes, it stably extracts dynamic cues related to attention and outputs temporally continuous individual attention scores and explanations of body part contributions.

[0023] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0024] This application provides a method, device, electronic device, and storage medium for assessing classroom student attention. The specific details are illustrated in the following embodiments. First, a method for assessing classroom student attention in this application is described.

[0025] It should be noted that in each specific implementation of this application, when it is necessary to process data related to user identity or characteristics, such as classroom teaching videos, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of this data will comply with relevant laws, regulations and standards.

[0026] Figure 1 is a flowchart of a classroom student attention assessment method provided in an embodiment of this application, which may include, but is not limited to, steps 101 to 107.

[0027] Step 101: Obtain classroom teaching video clips from the classroom teaching video. Step 102: Perform individual segmentation on each frame of the classroom teaching video clips to obtain multiple student individual masks. Step 103: Extract the initial individual part motion features of each student individual's body part within a time window from the classroom teaching video clips using segmented sliding windows and student individual masks. Step 104: Concatenate the initial individual part motion features of all student individuals' body parts across all time windows to obtain concatenated and fused features. Based on the student individual's individual code, the time window code of the time window, the part's part code, and the concatenated and fused features, generate the target individual part motion features for each student individual's body part. Step 105: Extract features from the target individual part motion features using a motion-aware Transformer network to obtain the student individual's part attention features. Step 106: Calculate the individual attention score for each student individual based on the part attention features. Step 107: Perform group aggregation calculation based on the individual attention scores of each student individual to obtain the overall class attention score.

[0028] Steps 101 to 107, as illustrated in the embodiments of this application, introduce the motion characteristics of individual student body parts to construct a {body part × frequency band × time} temporal feature tensor containing indicators such as trajectory amplitude, dominant frequency / band energy, and stability. This tensor is then adaptively weighted according to observation quality. Using these features, high-frequency micro-motions and low-frequency attitude drift are modeled on a Transformer architecture. Finally, through quality-gated cross-branch attention fusion, a more robust individual attention representation to occlusion and viewpoint changes is obtained, thus more robustly characterizing the overall attention level of students. Furthermore, this application achieves classroom student attention assessment based on instance-level character segmentation and body part kinematic features, making it highly suitable for non-contact attention feature quantification and real-time feedback in fixed-camera classroom teaching videos.

[0029] In step 101 of some embodiments, classroom teaching videos can be acquired using a camera device. The classroom teaching videos are then divided into segments to obtain classroom teaching video clips.

[0030] In one example, a high-definition camera (1920×1080 resolution) with an appropriate angle is deployed in a standardized lecture hall environment to ensure coverage of the entire student area from the teacher's perspective. The specific layout of people in the frame should be as shown in Figure 2. The data collection process strictly follows privacy protection guidelines and uses digital watermarking technology to bind classroom metadata (including course information, class information, teacher information, etc.). A timestamp is generated synchronously during data collection to facilitate accurate recording for subsequent analysis.

[0031] Using 100-minute classroom teaching videos (X) as a unit, a video stream with consistent timing is obtained. ,in, H refers to the height of the video frame, W refers to the width of the video frame, t refers to the timestamp, and T is the maximum value of the timestamp. To improve evaluation efficiency, the video is sliced ​​into segments of fixed duration of 5 seconds to obtain a set of video clips. Each of the classroom teaching video clips Includes 150 frames (the display frame rate of the classroom teaching video). ),satisfy , , u refers to the segment number.

[0032] In step 102 of some embodiments, for each 5-second classroom teaching video segment... A general segmentation model (such as Segment Anything Model 3, SAM3) is used to complete student instance-level segmentation, obtain individual student masks, and achieve identity temporal consistency within the segment.

[0033] In one embodiment, referring to FIG3, step 102 may include: step 301, performing human detection on video frames of a classroom teaching video segment to obtain a first candidate box set; step 302, filtering the first candidate boxes in the first candidate box set to obtain a second candidate box set; step 303, segmenting the second candidate boxes in the second candidate box set using a general segmentation model to obtain a candidate mask for each second candidate box; step 304, evaluating the confidence of the candidate masks to obtain mask confidence; step 305, filtering the candidate masks based on the mask confidence to obtain individual student masks.

[0034] In step 301, in the classroom teaching video clip Each video frame First, use a human detector (such as the YOLO model). Generate the first candidate box set Each first candidate box The specific form is (x, y, w, h), where (x, y) is the center of the box, w is the width of the box, and h is the height of the box, representing the first set of candidate boxes. It can be represented as: .

[0035] In step 302, the first candidate box can be deduplicated using non-maximum suppression (NMS), that is, the first candidate box with IoU below a threshold (e.g., 0.5) is retained; and the first candidate box with confidence below a threshold (e.g., 0.3) is filtered out. After deduplication and confidence filtering are completed, the second candidate box is obtained.

[0036] In step 303, the second candidate boxes in the second candidate box set are segmented using a general segmentation model to obtain a candidate mask for each second candidate box. Pixels with a value of 1 in the candidate mask belong to individual students, while pixels with a value of 0 do not belong to individual students. Following the example above, after obtaining the second candidate boxes through filtering, the second candidate boxes are then input as box-prompts into SAM3, which returns the candidate mask within the second candidate box (this candidate mask is a binary mask). .in, =1 indicates that pixel p in the t-th video frame belongs to the student instance identified by j, and 0 indicates that it does not belong to the student instance. It can be represented as: ,in, For the t-th video frame, The second candidate box corresponding to the student instance identified by j. These are the model parameters for SAM3.

[0037] In step 304, the observation quality vector can be calculated for each candidate mask. In order to obtain . Let j be the observation quality vector of the student instance identified by j in the t-th video frame. Let j represent the completeness of the student instance in the t-th video frame. Let j be the resolution of the student instance at the t-th video frame. Let j be the occlusion degree of the student instance in the t-th video frame.

[0038] Finally, in step 305, candidate masks are screened based on mask confidence to obtain individual student masks. A higher mask confidence indicates more reliable observation of the candidate mask, and thus, the candidate mask can be used as the individual student mask.

[0039] The advantage of the embodiments of steps 301 to 305 described above is that they can improve the quality of the mask, which is beneficial to improving the accuracy of student attention assessment.

[0040] In one embodiment, referring to FIG4, step 304 may include: step 401, calculating the integrity of the candidate mask by using the number of pixels with a value of 1 in the candidate mask as the numerator and the area of ​​the second candidate box as the denominator; step 402, sequentially performing grayscale conversion and Gaussian filtering on the video frame to obtain a denoised video frame, calculating the Laplacian response of each pixel within the range of pixels with a value of 1 in the candidate mask based on the denoised video frame, and performing variance statistics based on the Laplacian responses of all pixels to obtain the sharpness of the candidate mask; step 403, calculating the overlap between the candidate mask and multiple other candidate masks in the same video frame one by one to obtain multiple overlap degrees, and taking the largest of the multiple overlap degrees as the occlusion degree of the candidate mask; step 404, calculating the mask confidence degree based on the integrity, sharpness, and occlusion degree.

[0041] In step 401, for example, completeness The completeness index is calculated by using the number of pixels with a value of 1 in the candidate mask as the numerator and the area of ​​the second candidate box as the denominator. The closer the completeness index is to 1, the fuller the segmentation and the better the completeness.

[0042] In step 402, for example, Given the t-th video frame Candidate masks for student instances identified as j First, perform grayscale conversion on each pixel to obtain... Then, Gaussian filtering is used to denoise the result. This operation effectively suppresses errors caused by image noise. Specifically, + + , , Ultimately Each pixel 3×3 neighboring pixels and Laplacian four-neighbor kernel The corresponding positions are multiplied together and then added to obtain the Laplace response. The above operations are all performed only in the mask area. Calculate and avoid background interference. Calculate the candidate mask for the student instance identified by j. Inside, all pixels The variance is calculated statistically, which represents the sharpness. .in, , Clarity .

[0043] In step 403, for example, For each instance mask in the same frame, calculate the IoU and take the maximum overlap; the greater the overlap, the more severe the occlusion. Use 1 - maximum IoU to represent "occlusion degree", and the larger the value, the better the effect.

[0044] In step 404, the observation quality vector of the candidate mask identified by j Then mask confidence Specifically, first, perform z-score normalization on the integrity (c), sharpness (s), and occlusion (o). Then use weights The composite score is calculated using the bias b, and finally compressed into a confidence score using the Sigmoid function. The larger the value, the more reliable the mask observation. Among them, , , .

[0045] The advantage of the embodiments of steps 401 to 404 above is that by determining the mask confidence level in multiple dimensions, the accuracy of determining the mask confidence level is improved, which is conducive to improving the accuracy of student attention assessment.

[0046] In step 103 of some embodiments, the motion features of each student's body part at the initial individual part of the time window are extracted from the classroom teaching video clip using a segmented sliding window and a student individual mask. (Segmented sliding window) Window length Step length For a 5-second classroom teaching video clip, this segmented sliding window can divide it into 7 time windows.

[0047] In one embodiment, referring to FIG5, step 103 may include: step 501, extracting the initial individual part temporal motion features and initial individual part frequency domain motion features of each student individual's part in the time window from the classroom teaching video clip based on the centroid of each part of the student individual and the segmented sliding window; step 502, fusing the initial individual part motion features and initial individual part frequency domain motion features of each time window to obtain the initial individual part motion features.

[0048] For example, for each student instance (also called a student individual) j=1...N, the student individual mask output by SAM3 in the t-th video frame... At the same time, partial masks for each student will be generated. ,in Based on the center of mass of each part And to further calculate the motion characteristics in the time and frequency domains: .

[0049] The initial individual part's temporal motion characteristics include stability and amplitude. First, the inter-frame displacement is calculated. By taking the L2 norm of the displacement vector, a discrete scalar motion amplitude sequence is obtained. Then, the variance of this motion amplitude sequence is calculated. To reflect its stability: , , Simultaneously, the furthest and closest changes within the segmented sliding window were calculated. The larger the value, the larger the action. .

[0050] The initial individual part's frequency domain motion characteristics include frequency band energy. First, the Hann window function used in the STFT (Short-Time Fourier Transform) is defined. : Then, by performing a mean-removal operation, we obtain... This significantly reduces the contamination of the spectrum by slow attitude drift, making the proportion of low frequencies more interpretable. We will take the FFT points. The value is 128. For real signals, the negative spectrum and the positive spectrum are conjugate symmetric, so only the non-negative frequencies need to be retained. Therefore, the physical frequency corresponding to the k-th frequency point is obtained. : k=0,..., , Next, the STFT coefficients are obtained. : (j is the imaginary unit). Then, the power normalized to the window energy is obtained. : .

[0051] Based on different classroom actions, this application uses the following 5 non-equal bandwidth frequency bands to achieve semanticization: (1) extremely low frequency Corresponding to attitude drift and slow camera shake; (2) low frequency (3) Mid-frequency (4) Higher frequency Corresponding to anxiety / leg shaking / frequent small movements; (5) high frequency This corresponds mostly to noise or rapid jitter. The set of frequencies falling within each frequency band is denoted as... This allows us to calculate the energy of different frequency bands. and the energy proportion of each frequency band : , , Thus, we obtain the 2D time-domain motion characteristics and 5D frequency-domain motion characteristics of the j-th student's part k within a time window, and then fuse them to obtain the initial individual part motion characteristics of the j-th student's part k.

[0052] In step 104 of some embodiments, the motion features of the parts of all students in all time windows are spliced ​​together to obtain spliced ​​and fused features. Then, the target individual part motion features of each student's part are generated based on the individual code of the student, the time window code of the time window, the part code of the part, and the spliced ​​and fused features.

[0053] In one embodiment, based on the individual code of the student, the time window code of the time window, the part code of the part, and the splicing and fusion features, the target individual part motion features of each student's part are generated, including: projecting the splicing and fusion features through a linear layer to obtain a projection vector; adding the projection vector to the bias vector to obtain a feature tensor; and adding the feature tensor, the individual code of the student, the time window code of the time window, and the part code of the part to obtain the target individual part motion features of the student's part.

[0054] Specifically, after obtaining the initial motion features of individual body parts, the network needs to understand the prior knowledge of the current segment in the time dimension. To this end, this application abandons the traditional ViT approach of splitting the frame image into patches of the same size and then linearly mapping them to token vectors for input to the encoder. Instead, it uses a new branch (i.e., a motion-aware Transformer network) to process the aforementioned motion features. The specific network architecture is shown in Figure 6. For ease of explanation, let the temporal and frequency domain features of the j-th student individual, body part k, and time window u be respectively... The specific method is as follows: (1) splicing into feature fusion features First, we need to calculate The dimensions. For each 5-second segment, based on the size and step size of the segmented sliding window, there are only 7 fixed time windows; and for each student, the number of body parts can be 4 (including head, left arm, right arm, and torso); simultaneously, to adapt to different class sizes, [the following is used]... To determine the number of individuals to be retained, if the number is insufficient, PADs are used to supplement it; if the number is excessive, the mask confidence level calculated in step 304 is used. Come get Top- Therefore, the final total number of tokens is 28. ,Right now , .

[0055] (2) Convert to feature tensor Figure 6 shows the fusion motion feature encoding: here, a linear layer W is used to project the 7-dimensional fusion features into a projection vector, and the dimension is... Therefore, the shape of the tensor input to the encoder is [ [,256]: b is the bias vector.

[0056] (3) The input vector of the encoder is obtained by adding three types of information encoding, namely the motion characteristics of the target individual parts. Traditional ViT typically uses learnable absolute positional encoding. Similar to this approach, this application uses three types of encodings to be added together. The first is time window coding. The first part represents the temporal information of this feature, shown as time encoding in Figure 6; the second part is the location encoding. The first is the category information representing the feature; the third is the individual code. The value representing the individual to which the feature belongs is shown as an instance encoding in Figure 6. All three types of encodings are represented using vectors corresponding to the indices of the embedding matrix E. These embedding matrices are randomly initialized at the start of training and continuously updated during backpropagation. , , , .

[0057] In step 105 of some embodiments, referring to FIG6, after feature extraction by the Transformer in the Motion-Aware Transformer (MAT) network, the individual part attention features of the part k of the student identified by j on the time window u are obtained. Specifically, each of the obtained Flatten it into a sequence and add the CLS token at the beginning to get... Inside the Transformer encoder, a multi-head attention mechanism is used to ensure that when processing each stitched and fused feature, the model considers information from other stitched and fused features simultaneously, fully capturing the global dependencies of the image, rather than just local features. Specifically, each... When treated as a query, it simultaneously traverses the entire sequence X from multiple independent "focus perspectives" (multiple attention heads), automatically finding other vectors most relevant to itself (which could be different parts of the same student, previous or subsequent times, or neighboring classmates), and then weights and aggregates their information according to relevance into a more contextual representation; in this way, each token carries information about the global context.

[0058] In step 106 of some embodiments, referring to FIG6, the information can be processed by a multilayer perceptron (MLP) in the MAT to output the individual's attention score. However, the weights for different time windows and different parts are kept consistent by default. Specifically: Part Attention Score Partial weight The individual attention score of the student identified by j. .

[0059] In step 107 of some embodiments, a group aggregation calculation is performed based on the individual attention scores of each student to obtain the overall group attention score of the classroom.

[0060] In one embodiment, step 107 may include: calculating individual weights based on individual student masks and the confidence levels of individual student masks; and performing weighted fusion based on individual weights and individual attention scores to obtain a group attention score.

[0061] Specifically, the obtained individual attention score and the mask confidence of individual j in the t-th video frame are used. Group aggregation is used to calculate the attention score for the entire class. First, obtain the mask confidence score of individual j across the entire sample segment. Then use Individual attention score Weighted processing is then applied. Details are as follows: This only indicates whether the mask exists; .

[0062] In summary, the present application can achieve the following beneficial effects: (1) By quantitatively calculating individual attention characteristics and assigning corresponding weight scores based on the individual's image integrity, clarity, and occlusion, the individual's attention score and the overall classroom attention score can be calculated. Furthermore, it is more convenient for teachers to make personalized suggestions for the learning status of individual students, which greatly improves the operability of teaching intervention.

[0063] (2) Traditionally, pixel tubelets (RGB / optical flow) are used as tokens. This application transforms the kinematic sequence of individual parts (stability, amplitude and frequency band energy of four parts: head, torso and arm) into features and then maps them into tokens to achieve part-aware + motion-centric feature serialization.

[0064] Please refer to Figure 7. This application embodiment also provides a classroom student attention assessment device, which can implement the aforementioned classroom student attention assessment method. Figure 7 is a block diagram of the module structure of a classroom student attention assessment device provided in this application embodiment. The device includes: a video segment acquisition module 701, used to acquire classroom teaching video segments from classroom teaching videos; an individual segmentation module 702, used to segment each video frame of the classroom teaching video segment into multiple student individual masks; a motion feature extraction module 703, used to extract the initial individual part motion features of each student individual's part within a time window from the classroom teaching video segment using segmented sliding windows and student individual masks; and a feature fusion module 704, used to perform feature fusion based on the parts of all student individuals... The initial individual part motion features of all time windows are spliced ​​together to obtain spliced ​​and fused features. Based on the individual student code, the time window code of the time window, the part code of the part, and the spliced ​​and fused features, the target individual part motion features of each student's part are generated. The attention feature extraction module 705 is used to extract features from the target individual part motion features through a motion-aware Transformer network to obtain the individual part attention features of each student. The individual evaluation module 706 is used to calculate the individual attention score of each student based on the individual part attention features. The group evaluation module 707 is used to perform group aggregation calculation based on the individual attention scores of each student to obtain the group attention score of the entire class.

[0065] It should be noted that the specific implementation of this classroom student attention assessment device is basically the same as the specific implementation of the above-mentioned classroom student attention assessment method, and will not be repeated here.

[0066] This application also provides an electronic device, including: at least one memory; at least one processor; at least one program; the program is stored in the memory, and the processor executes the at least one program to implement the classroom student attention assessment method described above. This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0067] Please refer to Figure 8, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 801, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 802, which can be implemented using ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory), etc. The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 to implement a classroom student attention assessment method according to the embodiments of this application. The input / output interface 803 is used to realize information input and output. The communication interface 804 is used to realize communication interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 805 transmits information between various components of the device (such as processor 801, memory 802, input / output interface 803 and communication interface 804). The processor 801, memory 802, input / output interface 803 and communication interface 804 realize communication connection between each other within the device through the bus 805.

[0068] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described method for assessing classroom student attention.

[0069] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0070] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0071] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0074] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0075] It should be understood that in this application, "at least one (item)" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0076] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or the indirect coupling or communication connection between the apparatus or units may be electrical, mechanical, or other forms.

[0077] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0078] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for assessing student attention in the classroom, characterized in that, The method includes: acquiring classroom teaching video clips from classroom teaching videos; segmenting each video frame of the classroom teaching video clips to obtain multiple student individual masks; extracting the initial individual part motion features of each student individual's body part within a time window from the classroom teaching video clips using segmented sliding windows and the student individual masks; splicing the initial individual part motion features of all student individuals' body parts within all time windows to obtain splicing and fusion features, and generating target individual part motion features of each student individual's body part based on the student individual's individual code, the time window code of the time window, the body part code, and the splicing and fusion features; extracting features from the target individual part motion features using a motion-aware Transformer network to obtain the individual part attention features of the student individual; calculating the individual attention score of the student individual based on the individual part attention features; and performing group aggregation calculation based on the individual attention scores of each student individual to obtain the overall group attention score of the classroom.

2. The method according to claim 1, characterized in that, The step of segmenting the classroom teaching video clip to obtain multiple student individual masks includes: performing human detection on the video frames of the classroom teaching video clip to obtain a first candidate box set; filtering the first candidate boxes in the first candidate box set to obtain a second candidate box set; segmenting the second candidate boxes in the second candidate box set using a general segmentation model to obtain a candidate mask for each second candidate box; wherein, pixels with a value of 1 in the candidate mask belong to student individuals, and pixels with a value of 0 do not belong to student individuals; evaluating the confidence of the candidate masks to obtain mask confidence; and filtering the candidate masks based on the mask confidence to obtain the student individual masks.

3. The method according to claim 2, characterized in that, The confidence evaluation of the candidate mask to obtain the mask confidence score includes: calculating the integrity of the candidate mask by using the number of pixels with a value of 1 in the candidate mask as the numerator and the area of ​​the second candidate box as the denominator; sequentially performing grayscale conversion and Gaussian filtering on the video frame to obtain a denoised video frame; calculating the Laplacian response of each pixel within the range of pixels with a value of 1 in the candidate mask based on the denoised video frame; performing variance statistics based on the Laplacian responses of all pixels to obtain the sharpness of the candidate mask; calculating the overlap between the candidate mask and multiple other candidate masks in the same video frame one by one to obtain multiple overlap degrees; taking the largest of the multiple overlap degrees as the occlusion degree of the candidate mask; and calculating the mask confidence score based on the integrity, the sharpness, and the occlusion degree.

4. The method according to any one of claims 1 to 3, characterized in that, The step of extracting the initial individual part motion features of each student's part within a time window from the classroom teaching video clip using segmented sliding windows and the student individual mask includes: extracting the initial individual part temporal motion features and initial individual part frequency domain motion features of each student's part within a time window from the classroom teaching video clip based on the centroid of each part of the student and the segmented sliding windows; and fusing the initial individual part temporal motion features and the initial individual part frequency domain motion features of each time window to obtain the initial individual part motion features.

5. The method according to any one of claims 1 to 3, characterized in that, The step of generating target individual part motion features for each student's part based on the student's individual code, the time window code of the time window, the part code of the part, and the splicing and fusion features includes: projecting the splicing and fusion features through a linear layer to obtain a projection vector; adding the projection vector to a bias vector to obtain a feature tensor; and adding the feature tensor, the student's individual code, the time window code of the time window, and the part code of the part to obtain the target individual part motion features for the student's part.

6. The method according to any one of claims 1 to 3, characterized in that, The step of calculating the individual attention score of the student based on the individual part attention characteristics includes: scoring the individual part attention characteristics using a multilayer perceptron to obtain a part attention score; and performing a weighted summation based on the part weights of the student's parts and the part attention score to obtain the individual attention score.

7. The method according to any one of claims 1 to 3, characterized in that, The step of performing group aggregation calculation based on the individual attention scores of each student to obtain the overall class group attention score includes: calculating individual weights based on the individual student masks and the confidence levels of the individual student masks; and performing weighted fusion based on the individual weights and the individual attention scores to obtain the group attention score.

8. A classroom student attention assessment device, characterized in that, The device includes: a video segment acquisition module for acquiring classroom teaching video segments from classroom teaching videos; an individual segmentation module for segmenting each video frame of the classroom teaching video segments into individual segments to obtain multiple student individual masks; a motion feature extraction module for extracting the initial individual part motion features of each student individual's body part in the time window from the classroom teaching video segments using segmented sliding windows and the student individual masks; a feature fusion module for splicing the initial individual part motion features of all student individuals' body parts in all time windows to obtain spliced ​​fusion features, and generating target individual part motion features of each student individual's body part based on the student individual's individual code, the time window code of the time window, the body part code, and the spliced ​​fusion features; an attention feature extraction module for extracting features from the target individual part motion features using a motion-aware Transformer network to obtain the individual part attention features of the student individual; an individual evaluation module for calculating the individual attention score of the student individual based on the individual part attention features; and a group evaluation module for performing group aggregation calculation based on the individual attention scores of each student individual to obtain the overall group attention score of the classroom.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a non-volatile computer-readable storage medium storing at least one program, each program including instructions that, when executed by a terminal, cause the terminal to perform the method according to any one of claims 1 to 7.

Images