Online learner concentration recognition method, system, device and storage medium
By combining adaptive sampling and optical flow feature extraction with an improved spatiotemporal self-attention mechanism, the problem of feature redundancy in online learning scenarios is solved, improving the efficiency and accuracy of learner attention recognition and achieving efficient attention recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2024-03-07
- Publication Date
- 2026-06-26
AI Technical Summary
In online learning scenarios, learner attention recognition suffers from feature redundancy due to slow video changes, resulting in low recognition efficiency and accuracy.
An adaptive sampling method is adopted, which combines optical flow feature extraction and an improved spatiotemporal self-attention mechanism. Through spatiotemporal pipeline embedding and generalized spatiotemporal pipeline self-attention computation, the efficiency and accuracy of feature extraction are improved.
It improved the efficiency and accuracy of online learner attention recognition, with binary and quadruple classification accuracy increasing by 2.5% and 13.7% respectively, reaching 94.2% and 74.6% on public datasets.
Smart Images
Figure CN118230031B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing and relates to a method, system, device, and storage medium for identifying the attention of online learners. Background Technology
[0002] Attention level is a comprehensive reflection of a learner's learning status and an important indicator for evaluating learning quality. Identifying online learner attention is a key task in the integration of online education and computer vision. Video-based learner attention identification has attracted attention from academia and industry due to its non-invasiveness and accessibility. Identifying and analyzing online learner attention can help teachers understand learners' levels of focus and adjust teaching methods and strategies accordingly to ensure the quality of online teaching.
[0003] Currently, learner attention recognition in online learning scenarios relies on video data analysis, and extracting spatiotemporal information from the video has achieved good results. However, due to the inherent characteristics of online learning scenarios, the slow changes in video lead to feature redundancy, resulting in low recognition efficiency and accuracy. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, device and storage medium for identifying the attention of online learners.
[0005] To achieve the above objectives, the present invention employs the following technical solution:
[0006] In a first aspect, the present invention provides a method for identifying the attention span of online learners, comprising:
[0007] Acquire online learner videos and, based on the cumulative number of pixel changes in the online learner videos, obtain the video sampling frame sequence of the online learner videos;
[0008] Based on the spatiotemporal pipeline embedding method combined with optical flow, motion information is captured and embedded in the video sampling frame sequence to obtain the lexical unit sequence;
[0009] Based on spatiotemporal pipeline embedding and an improved separation spatiotemporal self-attention mechanism, a generalized spatiotemporal pipeline self-attention calculation is performed on the lexical unit sequence to obtain the category classification head;
[0010] Input the category header into the preset focus classification model to obtain the focus category of online learners.
[0011] Optionally, obtaining the video sampling frame sequence of the online learner video based on the cumulative number of pixel changes in the online learner video includes:
[0012] Obtain the cumulative change binary image of the online learner's video, and perform statistics based on the cumulative change binary image, recording the pixels with non-zero pixel values as changed pixels, and obtain the number of changed pixels N;
[0013] The adaptive sampling frame count for online learner videos is obtained using the following formula:
[0014]
[0015] Where F is the number of sampled frames in the online learner's video; T is the number of video frames in the online learner's video; H is the video frame height of the online learner's video; and W is the video frame width of the online learner's video. To round up;
[0016] The online learner video is randomly sampled based on the number of sampled frames, resulting in a sequence of sampled video frames.
[0017] Optionally, obtaining the cumulative change binary map of the online learner's video includes:
[0018] Based on the Gaussian mixture difference algorithm, the cumulative change binary calculation is performed on the online learner video to obtain the cumulative change binary map of the online learner video.
[0019] Optionally, the spatiotemporal pipeline embedding method based on optical flow performs motion information capture and embedding operations on the video sampled frame sequence to obtain a lexical unit sequence including:
[0020] Optical flow features are extracted from the video sampled frame sequence. Then, the spatiotemporal pipeline embedding method is used to divide the video sampled frame sequence into several spatiotemporal pipeline blocks using a 3D t×h×w patch. These spatiotemporal pipeline blocks are then converted into the following lexical unit sequence:
[0021]
[0022] Where z is the lexical unit sequence, z cls For the category header, t is the frame number of the spatiotemporal pipeline block, h is the frame height of the spatiotemporal pipeline block, w is the frame width of the spatiotemporal pipeline block, and Ex... (t,h,w) Lexical units of the patch extracted from the spatiotemporal pipeline at position (t,h,w); This involves embedding location information.
[0023] Optionally, the optical flow feature extraction of the video sample frame sequence includes:
[0024] The TV-L1 optical flow algorithm is used to extract optical flow features from the video sample frame sequence.
[0025] Optionally, the generalized spatiotemporal pipeline self-attention calculation on the lexical unit sequence includes:
[0026] The generalized spatiotemporal pipeline self-attention is calculated for the lexical unit sequence using the following formula:
[0027]
[0028]
[0029] Where t is the number of frames in the spatiotemporal pipeline block, h is the frame height of the spatiotemporal pipeline block, w is the frame width of the spatiotemporal pipeline block, and V (t,h,w) Let q be a value matrix. (t,h,w) For querying the matrix, Let d be the key matrix. k Let n be the dimension of the key matrix. t n represents the number of blocks obtained after partitioning in the time dimension. h n represents the number of blocks obtained after partitioning along the height dimension. w This indicates the number of blocks obtained after segmentation along the width dimension. For temporal self-attention, the calculation method is as follows: perform temporal self-attention calculation in a fixed space; For spatial self-attention, the calculation method is as follows: spatial self-attention is calculated at fixed intervals.
[0030] By fusing temporal self-attention and spatial self-attention, a category classification head is obtained.
[0031] Optionally, the preset attention classification model is obtained based on multilayer perceptron pre-training.
[0032] In a second aspect, the present invention provides an online learner attention recognition system, comprising:
[0033] The sampling module is used to acquire online learner videos and obtain the video sampling frame sequence of the online learner videos based on the cumulative number of pixel changes in the online learner videos;
[0034] The feature extraction module is used to capture and embed motion information from video sample frame sequences based on a spatiotemporal pipeline embedding method combined with optical flow, thereby obtaining a lexical unit sequence.
[0035] The attention module is used to perform generalized spatiotemporal pipeline self-attention calculation on lexical unit sequences based on spatiotemporal pipeline embedding and an improved separation spatiotemporal self-attention mechanism to obtain the category classification head.
[0036] The focus classification module is used to input the category classification header into the preset focus classification model to obtain the focus category of online learners.
[0037] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described online learner attention recognition method.
[0038] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described online learner attention recognition method.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] This invention provides an online learner attention recognition method. It obtains a video sampling frame sequence of the online learner video based on the cumulative pixel changes, achieving adaptive sampling. Then, based on a spatiotemporal pipeline embedding method incorporating optical flow, motion information is captured and embedded into the video sampling frame sequence to obtain a lexical unit sequence. Next, based on spatiotemporal pipeline embedding and an improved separable spatiotemporal self-attention mechanism, generalized spatiotemporal pipeline self-attention calculation is performed on the lexical unit sequence to obtain a category classification head. Finally, the category classification head is input into a preset attention classification model to obtain the online learner's attention category. This invention, through adaptive sampling and optical flow extraction techniques, combined with spatiotemporal pipeline embedding and an improved separable spatiotemporal self-attention mechanism, achieves generalized spatiotemporal pipeline self-attention calculation, thereby effectively improving the efficiency and accuracy of online learner attention category classification. After verification, the method of this invention improved the accuracy of binary classification and quad classification by an average of 2.5% and 13.7% on the public dataset DAISEE compared with the baseline method; compared with ResNet-TCN and DenseAttNet, it improved the accuracy of binary classification by 1.2% and 0.7%, and quad classification by 0.8% and 1.2%; and achieved binary classification and quad classification accuracy of 94.2% and 74.6% on its own dataset. Attached Figure Description
[0041] Figure 1 This is a flowchart of the online learner attention recognition method according to an embodiment of the present invention.
[0042] Figure 2 This is a schematic diagram of adaptive sampling of online learner videos according to an embodiment of the present invention.
[0043] Figure 3 This is a schematic diagram illustrating the principle of the spatiotemporal pipeline embedding method combining optical flow according to an embodiment of the present invention.
[0044] Figure 4 This is a schematic diagram illustrating the computational principle of the improved spatiotemporal self-attention mechanism in an embodiment of the present invention.
[0045] Figure 5 This is a block diagram of the online learner attention recognition system according to an embodiment of the present invention. Detailed Implementation
[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0048] The present invention will now be described in further detail with reference to the accompanying drawings:
[0049] See Figure 1 In one embodiment of the present invention, an online learner attention recognition method is provided, which can effectively overcome the problem of feature redundancy caused by the slow changes in the online learner's video in existing online learner attention recognition methods, thereby effectively improving recognition efficiency and accuracy.
[0050] Specifically, the online learner attention recognition method includes the following steps:
[0051] S1: Obtain the online learner's video and, based on the cumulative number of pixel changes in the online learner's video, obtain the video sampling frame sequence of the online learner's video;
[0052] S2: Based on the spatiotemporal pipeline embedding method combined with optical flow, motion information is captured and embedded in the video sampling frame sequence to obtain the lexical unit sequence;
[0053] S3: Based on spatiotemporal pipeline embedding and an improved separation spatiotemporal self-attention mechanism, generalized spatiotemporal pipeline self-attention calculation is performed on lexical unit sequences to obtain the category classification head;
[0054] S4: Input the category classification header into the preset focus classification model to obtain the focus category of online learners.
[0055] This invention provides an online learner attention recognition method. It obtains a video sampling frame sequence of the online learner video based on the cumulative pixel changes, achieving adaptive sampling. Then, based on a spatiotemporal pipeline embedding method incorporating optical flow, motion information is captured and embedded into the video sampling frame sequence to obtain a lexical unit sequence. Next, based on spatiotemporal pipeline embedding and an improved separable spatiotemporal self-attention mechanism, generalized spatiotemporal pipeline self-attention calculation is performed on the lexical unit sequence to obtain a category classification head. Finally, the category classification head is input into a preset attention classification model to obtain the online learner's attention category. This invention, through adaptive sampling and optical flow extraction techniques, combined with spatiotemporal pipeline embedding and an improved separable spatiotemporal self-attention mechanism, achieves generalized spatiotemporal pipeline self-attention calculation, thereby effectively improving the efficiency and accuracy of online learner attention category classification. After verification, the method of this invention improved the accuracy of binary classification and quad classification by an average of 2.5% and 13.7% on the public dataset DAISEE compared with the baseline method; compared with ResNet-TCN and DenseAttNet, it improved the accuracy of binary classification by 1.2% and 0.7%, and quad classification by 0.8% and 1.2%; and achieved binary classification and quad classification accuracy of 94.2% and 74.6% on its own dataset.
[0056] Embedding is a technique that transforms high-dimensional sparse feature vectors into low-dimensional dense feature vectors. This operation is particularly important in deep learning recommendation systems. The original feature vectors are typically ultra-high-dimensional sparse one-hot vectors. Embedding allows these high-dimensional sparse features to be represented with a lower dimension (i.e., embedding size), facilitating subsequent model training. Embedding can be viewed as a table lookup process, using the sparse ID (original vector) to find a specific row in the embedding weight matrix. Therefore, embedding can be considered a fully connected layer, with the weights of this layer being the embedding weight matrix.
[0057] A lexical unit (token) sequence refers to the process of scanning and identifying each word in a source program, determining the type of each word, and converting them into a unified internal machine representation. The token sequence contains a category code and attribute values. The category code identifies the type of the word, while the attribute values store specific information about the word. For example, when a keyword is scanned, a one-word-one-code approach is adopted, meaning each keyword corresponds to a category code in the token. In the case of multiple words sharing one code, the attribute values of the token are assigned multiple times. Token sequences play a crucial role in compiler theory; they are one of the key steps in translating high-level languages into machine language. Through lexical analysis, the source program can be decomposed into a series of token sequences, providing the foundational data for subsequent syntax analysis, semantic analysis, and other processes.
[0058] The purpose of using the cumulative pixel change count in online learner videos is to achieve adaptive sampling of these videos and effectively avoid feature redundancy. The adaptive sampling approach is as follows: Different levels of activity variation in online learner videos lead to varying degrees of feature redundancy. Adaptive sampling represents motion information in the video by using the cumulative pixel change count. When the video has rich motion information, a higher cumulative pixel change count is needed, requiring more sampling frames to better learn the spatiotemporal information of the online learner video. Conversely, when the video has insufficient motion information, the cumulative pixel change count and the number of sampling frames are reduced accordingly, in which case a smaller number of video frames are sufficient to represent the video.
[0059] In one possible implementation, a cumulative change binary map of the online learner's video is obtained, and statistics are performed based on the cumulative change binary map to record pixels with non-zero values as changed pixels, thus obtaining the number of changed pixels N; then, the adaptive sampling frame number of the online learner's video is obtained using the following formula:
[0060]
[0061] Where F is the number of sampled frames in the online learner's video; T is the number of video frames in the online learner's video; H is the video frame height of the online learner's video; and W is the video frame width of the online learner's video. This is for rounding up.
[0062] Finally, the online learner videos are randomly sampled based on the number of sampled frames in the online learner videos to obtain the video sampled frame sequence of the online learner videos.
[0063] Optionally, obtaining the cumulative change binary map of the online learner video includes: calculating the cumulative change binary map of the online learner video based on the Gaussian mixture interpolation algorithm to obtain the cumulative change binary map of the online learner video. For details, see... Figure 2First, the parameters of the online learner video are obtained, including the video frame height, video frame width, and number of video frames. Then, the pixel change difference between adjacent video frames is calculated using a Gaussian mixture interpolation algorithm. If the change difference is higher than a threshold, the pixel value at that point is considered to have changed and is recorded in a binary image. The binary images are then summed to obtain a cumulative change binary image. In the cumulative change binary image, if a pixel has changed, its value is not 0. This allows us to count the number of changed pixels in the entire online learner video. Combining this with the video frame height and the number of video frames completes the calculation of the sampling frame number F corresponding to the online learner video.
[0064] In one possible implementation, the spatiotemporal pipeline embedding method based on optical flow performs motion information capture and embedding operations on a video sample frame sequence to obtain a lexical unit sequence including:
[0065] Optical flow features are extracted from the video sampled frame sequence. Then, the spatiotemporal pipeline embedding method is used to divide the video sampled frame sequence into several spatiotemporal pipeline blocks using a 3D t×h×w patch. These spatiotemporal pipeline blocks are then converted into the following lexical unit sequence:
[0066]
[0067] Where z is the lexical unit sequence, z cls For the category header, t is the frame number of the spatiotemporal pipeline block, h is the frame height of the spatiotemporal pipeline block, w is the frame width of the spatiotemporal pipeline block, and Ex... (t,h,w) Lexical units of the patch extracted from the spatiotemporal pipeline at position (t,h,w); This involves embedding location information.
[0068] Optionally, the optical flow feature extraction of the video sample frame sequence includes: extracting optical flow features of the video sample frame sequence using the TV-L1 optical flow algorithm.
[0069] For details, see Figure 3 The spatiotemporal pipeline embedding method combines and partitions the temporal and spatial dimensions of online learner videos, resulting in a patch that is a three-dimensional 3D spatiotemporal tensor. Given an online learner video... The spacetime pipeline with dimensions t×h×w can be divided to obtain N=n. t ×n h ×n w There are 1 patch, of which... Then, each patch captures motion information within the pipeline using optical flow, and transforms it into... through 3D convolution and reshape... The tensor, combined with positional encoding embedding and classification head embedding, yields the final token sequence input into the Transformer structure.
[0070] Where V represents the input video frequency, T is the number of video frames, H is the height of the video frame, W is the width of the video frame, C is the number of channels, t is the number of frames in the spatiotemporal pipeline block, h is the height of the spatiotemporal pipeline block, w is the width of the spatiotemporal pipeline block, N is the total number of spatiotemporal pipeline blocks obtained after segmentation, and n t n represents the number of spatiotemporal pipeline blocks obtained after partitioning along the time dimension. h n represents the number of spatiotemporal pipeline blocks obtained after segmentation along the height dimension. w This represents the number of spatiotemporal pipeline blocks obtained after segmentation along the width dimension, and d represents the number of features in each spatiotemporal pipeline block.
[0071] In one possible implementation, the generalized spatiotemporal pipeline self-attention computation on the lexical unit sequence includes:
[0072] The generalized spatiotemporal pipeline self-attention is calculated for the lexical unit sequence using the following formula:
[0073]
[0074]
[0075] in, For temporal self-attention, the calculation method is as follows: perform temporal self-attention calculation in a fixed space; For spatial self-attention, the calculation method is as follows: spatial self-attention is calculated at fixed intervals; t is the number of frames in the spatiotemporal pipeline block, h is the frame height of the spatiotemporal pipeline block, w is the frame width of the spatiotemporal pipeline block, and V... (t,h,w) Let q be a value matrix. (t,h,w) For querying the matrix, Let d be the key matrix. k Let n be the dimension of the key matrix. t n represents the number of blocks obtained after partitioning in the time dimension. h n represents the number of blocks obtained after partitioning along the height dimension. w This indicates the number of blocks obtained after dividing the data along the width dimension.
[0076] Specifically, based on the spatiotemporal pipeline embedding method, the video frame sequence is converted into several spatiotemporal pipeline blocks. According to the division method of the spatiotemporal pipeline, the video frame is extended to a generalized frame, and the video spatiotemporal is extended to a generalized spatiotemporal. In a generalized sense, two pipelines adjacent in the temporal dimension can be considered as adjacent "frames," and pipelines adjacent in the spatial dimension can be considered as adjacent pixel blocks within the same "frame"—this is the concept of a generalized frame. Based on the concept of generalized frames, spatiotemporal is extended to a generalized spatiotemporal, where the concept of time changes from the original single frame to the time dimension of the pipeline, and the concept of space changes from the original single-frame image to the pipeline plane.
[0077] See Figure 4 Spatial attention is calculated for the top-left patch and other patches within the same generalized frame at the same time, and temporal attention is calculated for patches at the same spatial position in different generalized frames. The combination of these two approaches constitutes self-attention computation based on a generalized spatiotemporal pipeline. The calculation method is as follows:
[0078]
[0079]
[0080] After completing the self-attention calculation of the generalized spatiotemporal pipeline for spatiotemporal separation, the temporal self-attention and spatial self-attention are added and fused to form the category classification head z. cls Input into the preset focus classification model for focus classification.
[0081] Optionally, the preset attention classification model can be obtained based on multilayer perceptron pre-training.
[0082] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.
[0083] See Figure 5 In another embodiment of the present invention, an online learner attention recognition system is provided, which can be used to implement the above-mentioned online learner attention recognition method. Specifically, the online learner attention recognition system includes a sampling module, a feature extraction module, an attention module, and an attention classification module.
[0084] The sampling module is used to acquire online learner videos and obtain a video sampling frame sequence based on the cumulative number of pixel changes in the online learner videos. The feature extraction module is used to capture and embed motion information of the video sampling frame sequence based on the spatiotemporal pipeline embedding method combined with optical flow to obtain a lexical unit sequence. The attention module is used to perform generalized spatiotemporal pipeline self-attention calculation on the lexical unit sequence based on spatiotemporal pipeline embedding and an improved separate spatiotemporal self-attention mechanism to obtain a category classification head. The focus classification module is used to input the category classification head into a preset focus classification model to obtain the focus category of the online learner.
[0085] All relevant content of each step involved in the aforementioned embodiments of the online learner attention recognition method can be referenced to the functional description of the corresponding functional module of the online learner attention recognition system in the embodiments of the present invention, and will not be repeated here.
[0086] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0087] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of an online learner attention recognition method.
[0088] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the online learner attention recognition method in the above embodiments.
[0089] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0090] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for recognizing the attention level of online learners, characterized in that, include: Acquire online learner videos and, based on the cumulative number of pixel changes in the online learner videos, obtain the video sampling frame sequence of the online learner videos; Based on the spatiotemporal pipeline embedding method combined with optical flow, motion information is captured and embedded in the video sampling frame sequence to obtain the lexical unit sequence; Based on spatiotemporal pipeline embedding and an improved separation spatiotemporal self-attention mechanism, a generalized spatiotemporal pipeline self-attention calculation is performed on the lexical unit sequence to obtain the category classification head; Input the category header into the preset focus classification model to obtain the focus category of online learners; The video sampling frame sequence of the online learner video, obtained based on the cumulative pixel changes in the online learner video, includes: Obtain the cumulative change binary image of the online learner's video, and perform statistics based on the cumulative change binary image, recording pixels with non-zero values as changed pixels, and obtain the number of changed pixels. ; The adaptive sampling frame count for online learner videos is obtained using the following formula: in, The number of sampled frames for the online learner's video; The number of video frames for online learners; For online learners, the video frame rate is high; The video frame width for online learners' videos; To round up; The online learner video is randomly sampled based on the number of sampled frames, resulting in a sequence of sampled video frames.
2. The online learner attention recognition method according to claim 1, characterized in that, The acquisition of the cumulative change binary map of online learner videos includes: Based on the Gaussian mixture difference algorithm, the cumulative change binary calculation is performed on the online learner video to obtain the cumulative change binary map of the online learner video.
3. The online learner attention recognition method according to claim 1, characterized in that, The spatiotemporal pipeline embedding method based on optical flow performs motion information capture and embedding operations on the video sample frame sequence to obtain a lexical unit sequence including: Optical flow features are extracted from the video sample frame sequence, and then a spatiotemporal pipeline embedding method is used to achieve optical flow through three dimensions. The patch divides the video sample frame sequence into several spatiotemporal pipeline blocks, and converts these spatiotemporal pipeline blocks into the following lexical unit sequence: in, A sequence of lexical units, For category headers, t The number of frames in the spacetime pipeline block. h The frame height of the spacetime pipeline block. w The frame width of the spatiotemporal pipeline block. For the location at Lexical units of the patch extracted from the spatiotemporal pipeline; This involves embedding location information.
4. The online learner attention recognition method according to claim 3, characterized in that, The optical flow feature extraction of the video sample frame sequence includes: The TV-L1 optical flow algorithm is used to extract optical flow features from the video sample frame sequence.
5. The online learner attention recognition method according to claim 1, characterized in that, The generalized spatiotemporal pipeline self-attention calculation for the lexical unit sequence includes: The generalized spatiotemporal pipeline self-attention is calculated for the lexical unit sequence using the following formula: in, t The number of frames in the spacetime pipeline block. h The frame height of the spacetime pipeline block. w The frame width of the spatiotemporal pipeline block. For value matrices, For querying the matrix, The key matrix, Let be the dimension of the key matrix. This represents the number of blocks obtained after partitioning along the time dimension. This indicates the number of blocks obtained after segmentation along the height dimension. This indicates the number of blocks obtained after segmentation along the width dimension. For temporal self-attention, the calculation method is as follows: perform temporal self-attention calculation in a fixed space; For spatial self-attention, the calculation method is as follows: spatial self-attention is calculated at fixed intervals. By fusing temporal self-attention and spatial self-attention, a category classification head is obtained.
6. The online learner attention recognition method according to claim 1, characterized in that, The preset attention classification model is obtained based on multilayer perceptron pre-training.
7. An online learner attention recognition system, characterized in that, include: The sampling module is used to acquire online learner videos and obtain the video sampling frame sequence of the online learner videos based on the cumulative number of pixel changes in the online learner videos; The feature extraction module is used to capture and embed motion information from video sample frame sequences based on a spatiotemporal pipeline embedding method combined with optical flow, thereby obtaining a lexical unit sequence. The attention module is used to perform generalized spatiotemporal pipeline self-attention calculation on lexical unit sequences based on spatiotemporal pipeline embedding and an improved separation spatiotemporal self-attention mechanism to obtain the category classification head. The focus classification module is used to input the category classification header into the preset focus classification model to obtain the focus category of online learners; The video sampling frame sequence of the online learner video, obtained based on the cumulative pixel changes in the online learner video, includes: Obtain the cumulative change binary image of the online learner's video, and perform statistics based on the cumulative change binary image, recording pixels with non-zero values as changed pixels, and obtain the number of changed pixels. ; The adaptive sampling frame count for online learner videos is obtained using the following formula: in, The number of sampled frames for the online learner's video; The number of video frames for online learners; For online learners, the video frame rate is high; The video frame width for online learners' videos; To round up; The online learner video is randomly sampled based on the number of sampled frames, resulting in a sequence of sampled video frames.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the online learner attention recognition method as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the online learner attention recognition method as described in any one of claims 1 to 6.