A student concentration recognition method and system

Abstract

The application provides a student concentration recognition method and system, and the method comprises the following steps: obtaining a video frame sequence of a learning video of a target student; performing multi-scale feature extraction on the video frame sequence based on a feature extraction network in a concentration recognition model, and obtaining a multi-scale feature sequence of the video frame sequence; performing concentration classification on the multi-scale feature sequence based on a classification network in the concentration recognition model, and obtaining the concentration of the target student; wherein the concentration recognition model is trained based on a sample video frame sequence and a sample concentration of a student in the sample video frame sequence. The application realizes automatic recognition of student concentration, saves labor costs, and improves the accuracy and efficiency of student concentration recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more particularly relates to a student concentration recognition method and system. BACKGROUND

[0002] The concentration of students in the classroom can be regarded as a comprehensive embodiment of the concentration of students' attention, learning participation, etc. in the classroom, so the concentration of students in the classroom can be used as one of the indicators for teaching process evaluation reference.

[0003] At present, the student concentration recognition method usually adopts a manual recognition method, which requires learners to directly participate in the detection process. This method often requires a lot of time and effort, and there are many factors that researchers cannot control that affect the effectiveness of self-reporting and observer evaluation results, have certain randomness and subjectivity, resulting in low recognition accuracy. SUMMARY

[0004] In view of the defects of the prior art, the purpose of the present application is to provide a student concentration recognition method and system, which aims to solve the problems of high labor cost and low recognition accuracy of the existing student concentration recognition method.

[0005] To achieve the above-mentioned purpose, in a first aspect, the present application provides a student concentration recognition method, comprising:

[0006] S101 obtaining a video frame sequence of a target student's learning video;

[0007] S102 performing multi-scale feature extraction on the video frame sequence based on a feature extraction network in a concentration recognition model and obtaining a multi-scale feature sequence of the video frame sequence;

[0008] S103 performing concentration classification on the multi-scale feature sequence based on a classification network in the concentration recognition model and obtaining the concentration of the target student;

[0009] The concentration recognition model is trained based on a sample video frame sequence and a sample concentration of a student in the sample video frame sequence.

[0010] In an optional example, the sample video frame sequence is obtained based on the following steps:

[0011] Randomly covering each image block of the video frames in the initial sample learning video to obtain a sample learning video;

[0012] Randomly frame sampling the sample learning video to obtain the sample video frame sequence.

[0013] In an optional example, S103 specifically includes:

[0014] The classification network generates a position coding sequence corresponding to the multi-scale feature sequence based on the time information of the multi-scale feature sequence, and performs concentration classification based on the multi-scale feature sequence and its corresponding position coding sequence to obtain the concentration of the target student.

[0015] In an optional example, performing concentration classification based on the multi-scale feature sequence and its corresponding position coding sequence to obtain the concentration of the target student includes:

[0016] Fusing the multi-scale feature sequence and its corresponding position coding sequence to obtain a fused feature sequence;

[0017] The correlation between any two features in the fused feature sequence is analyzed based on the self-attention mechanism, and the concentration level is classified based on the correlation to obtain the concentration level of the target student.

[0018] In an optional example, the feature extraction network includes multiple bottleneck layers; the bottleneck layer includes a first convolutional layer, multiple residual connection branches and a second convolutional layer connected in sequence.

[0019] In a second aspect, the present invention provides a student concentration recognition system, comprising:

[0020] A video frame sequence acquisition module is used to acquire the video frame sequence of the target student's learning video;

[0021] A multi-scale feature extraction module, configured to perform multi-scale feature extraction on the video frame sequence based on a feature extraction network in a concentration recognition model and obtain a multi-scale feature sequence of the video frame sequence;

[0022] A concentration classification module is used to classify the multi-scale feature sequence based on the classification network in the concentration recognition model and obtain the concentration of the target student;

[0023] The concentration recognition model is obtained based on a sample video frame sequence and sample concentration training of students in the sample video frame sequence.

[0024] In an optional example, the system further includes a training sample acquisition module;

[0025] The training sample acquisition module is used to randomly mask each image block of the video frame in the initial sample learning video to obtain the sample learning video, and randomly extract frames from the sample learning video to obtain the sample video frame sequence.

[0026] In an optional example, the concentration classification module is specifically used to generate a position coding sequence corresponding to the multi-scale feature sequence based on the time information of the multi-scale feature sequence by a classification network, and perform concentration classification based on the multi-scale feature sequence and its corresponding position coding sequence to obtain the concentration of the target student.

[0027] In an optional example, the concentration classification module is specifically used to fuse the multi-scale feature sequence and its corresponding position coding sequence to obtain a fused feature sequence, analyze the correlation between any two features in the fused feature sequence based on the self-attention mechanism, and perform concentration classification based on the correlation to obtain the concentration of the target student.

[0028] In an optional example, the feature extraction network includes multiple bottleneck layers; the bottleneck layer includes a first convolutional layer, multiple residual connection branches and a second convolutional layer connected in sequence.

[0029] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0030] The present invention provides a method and system for identifying student concentration. By obtaining a video frame sequence of a target student during the learning process and applying a deep learning model to first perform multi-scale feature extraction on the video frame sequence, the model is able to capture the temporal and spatial feature information in the video frame sequence, and simultaneously include detailed information and overall information. On this basis, the student concentration is analyzed, thereby realizing automatic identification of student concentration, saving labor costs while improving the accuracy and efficiency of student concentration identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 1 is a flow chart of a method for identifying student concentration provided by an embodiment of the present invention;

[0032] Figure 2 This is a structural comparison diagram of the bottleneck provided by an embodiment of the present invention and the bottleneck of a standard ResNet;

[0033] Figure 3 This is an overall framework diagram of a student concentration recognition model based on a self-attention mechanism provided by an embodiment of the present invention;

[0034] Figure 4 is a flow chart of a data enhancement method provided by an embodiment of the present invention;

[0035] Figure 5 4 is an architectural diagram of a student concentration recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not used to limit the present application.

[0037] The progress of computer technology and the rapid development of artificial intelligence promote the integration of information technology and various industries in society, drive the transformation of the industry, and also have a significant impact on the development of the education industry. The traditional teaching evaluation method often takes the teaching results as the main basis for evaluation, and has problems such as relatively simple evaluation means, relatively mechanical evaluation mode, and subjective evaluation subject. Therefore, it is fully reasonable and urgent to include teaching process evaluation into the evaluation system. However, teaching process evaluation needs to observe, record and analyze the performance of each student in the teaching process, which requires a large amount of manpower and is very time-consuming. With the development of science and technology, big data and artificial intelligence technology can be used to provide technical and data support for education evaluation. The concentration of students in the classroom can be regarded as a comprehensive reflection of the concentration of students' attention in the classroom, learning participation, etc., so the concentration of students in the classroom can be used as one of the indicators for teaching process evaluation.

[0038] The student concentration recognition method can be divided into: manual recognition method, automatic recognition method. Among them, the manual recognition method refers to a method in which the learner directly participates in the detection process. This method often requires a lot of time and effort, and has certain randomness and subjectivity, resulting in low recognition accuracy. The automatic recognition method is a method of collecting image, physiological index (heart rate, electroencephalogram, blood pressure, skin electricity, etc.), learning record (click stream, total learning time, forum post quantity, average problem solving time, problem solving accuracy, etc.) and other data of the learner in the learning process, then extracting features from the computer, and finally recognizing through a certain matching rule or deep learning model.

[0039] With the development of science and technology, video acquisition devices have become more affordable and their performance has been continuously improved, which makes it very easy to collect a large amount of high-quality student learning process videos. At the same time, with the continuous improvement of computer computing power, these have created favorable conditions for the rapid development of deep learning technology. Deep learning technology is being used in more and more fields for related research, and is revolutionizing the development of various fields. The concentration recognition field is no exception. The concentration recognition method based on deep learning technology has become the main development direction of the concentration recognition field.

[0040] In view of this, the present application provides a student concentration recognition method based on deep learning technology. Figure 1 is a flowchart of the student concentration recognition method provided by the embodiment of the present application, likeFigure 1 As shown, the method specifically comprises:

[0041] In step S101, a video frame sequence of a learning video of a target student is acquired.

[0042] Here, the target student is a student who needs to be recognized for concentration. The learning process of the target student in the classroom can be video collected, thereby obtaining the learning video of the target student. A plurality of continuous video frames are extracted from the learning video and arranged in corresponding time sequence, thereby obtaining the video frame sequence. Here, the video frame sequence can include facial expressions, eye gaze, posture and other features of the target student.

[0043] In step S102, a multi-scale feature extraction network in the concentration recognition model is used to perform multi-scale feature extraction on the video frame sequence and obtain a multi-scale feature sequence of the video frame sequence.

[0044] In step S103, a classification network in the concentration recognition model is used to perform concentration classification on the multi-scale feature sequence and obtain the concentration of the target student.

[0045] The concentration recognition model is trained based on sample video frame sequences and sample concentrations of students in the sample video frame sequences.

[0046] Specifically, the concentration recognition model is used to analyze each video frame in the input video frame sequence, thereby obtaining the concentration recognition result corresponding to the video frame sequence, i.e., the concentration of the target student. The concentration recognition model can be divided into two stages during concentration recognition. In the first stage, a feature extraction network in the concentration recognition model is used to perform multi-scale feature extraction on the video frame sequence and obtain a multi-scale feature sequence of the video frame sequence. In the second stage, a classification network in the concentration recognition model is used to perform concentration classification according to the multi-scale feature sequence obtained in the first stage, thereby obtaining the concentration of the target student in the classroom.

[0047] It should be noted that the concentration recognition result of a student is closely related to local features and overall features. However, existing concentration recognition models often only select a part of features for reasoning. To solve this problem, the embodiment of the present application designs a multi-scale feature extraction network to extract features containing local information and overall information, and then performs concentration classification, thereby improving the concentration recognition accuracy of the model.

[0048] In addition, before executing step S102, a concentration recognition model can be pre-trained. Specifically, the training can be performed in the following manner: first, a large number of videos of students learning in class are collected as sample learning videos, and sample video frame sequences are obtained, and the sample concentration of students in the sample video frame sequences are obtained by annotation; then, the sample video frame sequence and the sample concentration of students in the sample video frame sequence are input into the initial model for training, thereby obtaining a concentration recognition model.

[0049] The method provided by the embodiment of the present invention obtains a video frame sequence of the target student during the learning process, and applies a deep learning model to first perform multi-scale feature extraction on the video frame sequence, so that the model can capture the temporal and spatial feature information in the video frame sequence, and simultaneously include detailed information and overall information. On this basis, the student's concentration is analyzed, thereby realizing automatic recognition of the student's concentration, saving labor costs while improving the accuracy and efficiency of student concentration recognition.

[0050] Based on the above embodiment, the sample video frame sequence is obtained based on the following steps:

[0051] Randomly mask each image block of the video frame in the initial sample learning video to obtain a sample learning video;

[0052] Randomly extract frames from the sample learning video to obtain a sample video frame sequence.

[0053] Specifically, by analyzing and summarizing the characteristics of the teaching process video, it is found that occlusion is one of the most important factors affecting the accuracy of student concentration recognition. In response to this problem, the embodiment of the present invention proposes a coping method from the data level, proposes a sample data enhancement method that simulates real scenes, draws on the masking idea in the field of natural language processing, and designs a random masking method (V-Mask) for video data. The general idea of ​​the V-Mask method is to first divide the video frames in the acquired initial sample learning video into a number of non-overlapping image blocks, and then calculate whether each image block is covered according to a preset probability, obtain the covered video frame and replace the original video frame, and finally obtain the video after the cover processing, that is, the sample learning video. In this way, the situation where the target object is occluded can be simulated well.

[0054] Subsequently, considering that the total number of frames of each video in the training set is much larger than the input length of the model, and when students learn in class, they often do not have a large amplitude of movement, and the body movement is less, so only part of the frames need to be used as input. Therefore, when the video is input into the model as training data, a random frame extraction based data enhancement method (R-Extract) can be used to extract frames in a random manner, rather than in a fixed rule. Due to the randomness, the same video can generate many different video frame sequences.

[0055] Suppose the total number of frames of a sample learning video is t frames, the sampling frequency is f, and the input length of the model (i.e. the length of the sample video frame sequence) is l, then the random starting frame position s can be calculated, and the calculation formula is as follows:

[0056] s = random_int(0, t-f*l-1)

[0057] Where the random_int function is used to generate a random integer. Then, taking s as the starting position and f as the sampling frequency, the sample video frame sequence is extracted.

[0058] Further, before using the V-Mask method to randomly cover the image area of the video frame, the video frame can be first enhanced by cropping, translation, brightness adjustment, flipping and other conventional enhancement means.

[0059] The data enhancement method proposed in the embodiment of the application can enhance the original data, solve the problem of small amount of data in the field of student concentration recognition, and increase the amount of data while enriching the distribution of data in the data set. The data trained by the enhanced method can well improve the generalization ability and robustness of the concentration recognition model, and improve the accuracy of concentration recognition of students in the classroom under the occlusion scene.

[0060] Based on any of the above embodiments, step S103 specifically comprises:

[0061] The classification network generates a position encoding sequence corresponding to the multi-scale feature sequence based on the time information of the multi-scale feature sequence, and classifies the concentration degree based on the multi-scale feature sequence and the position encoding sequence corresponding thereto, to obtain the concentration degree of the target student.

[0062] Specifically, considering that the features corresponding to the video frames at different time points may have strong correlation, such as a certain state at time a and a certain state at time b have strong correlation and have a large combination relationship, the concentration of the target student can be inferred with a large probability, and for this, the classification network encodes the positions of each multi-scale feature in the multi-scale feature sequence according to the time information of the input multi-scale feature sequence, thereby generating a position encoding sequence corresponding to the multi-scale feature sequence, and then, the concentration is classified in combination with the multi-scale feature sequence and the position encoding sequence corresponding thereto, and the concentration of the target student is obtained, thereby further improving the concentration recognition accuracy of the model.

[0063] Here, the time information can include the relative position relationship between the time points before and after the multi-scale features at different time points, the time interval size and the like.

[0064] Based on any of the above embodiments, the concentration is classified based on the multi-scale feature sequence and the position encoding sequence corresponding thereto, and the concentration of the target student is obtained, including:

[0065] The multi-scale feature sequence and the position encoding sequence corresponding thereto are fused to obtain a fused feature sequence.

[0066] Based on the self-attention mechanism, the correlation between any two features in the fused feature sequence is analyzed, and the concentration is classified based on the correlation to obtain the concentration of the target student.

[0067] Specifically, in the classification network, in order to solve the problems of poor long-distance feature capture ability of the sequence model and inability to perform parallel calculation, the self-attention mechanism is used, and the position encoding method in the self-attention mechanism is modified to include time information. First, the multi-scale feature sequence and the position encoding sequence corresponding thereto can be fused to obtain a fused feature sequence, and then the fused feature sequence is input into the classification module in parallel, and the fused features at any two positions in the sequence can be directly associated through the self-attention mechanism to obtain correlation features, without relying on the transmission of the previous time point, greatly improving the calculation efficiency, and without the information loss caused by the long sequence.

[0068] After extracting the correlation features in the time dimension of the fused feature sequence, it is input into the full connection layer for concentration classification, and the concentration of the target student in the classroom is judged to output the concentration of the target student.

[0069] The embodiments of the present invention target classroom scenarios, student concentration characteristics, and shortcomings of existing models, and propose a student concentration recognition model based on the self-attention mechanism. By adding a multi-scale feature fusion module to the feature extraction network and introducing the self-attention mechanism to the classification network, the problems of the existing concentration recognition model's lack of multi-scale feature extraction capability, the image sequence model's poor long-distance feature capture capability, the inability to perform parallel calculations, and the poor model effect caused by insufficient data volume are solved.

[0070] Based on any of the above embodiments, the feature extraction network includes multiple bottleneck layers; the bottleneck layer includes a first convolutional layer, multiple residual connection branches and a second convolutional layer connected in sequence.

[0071] Specifically, by analyzing the features related to the impact of student concentration, it was found that local and overall features are equally important to the results of student concentration. To this end, the embodiment of the present invention modifies the standard ResNet model, adds support for multi-scale feature extraction to its structure, and designs a multi-scale feature extraction network MS-ResNet-50 so that the extracted features contain both detailed information and overall information. The MS-ResNet-50 model extracts the features of the video frame sequence in the spatial dimension, that is, the multi-scale feature sequence, and then inputs the multi-scale feature sequence into the classification network for concentration classification.

[0072] Modifying the ResNet model structure specifically involves modifying the bottleneck layer structure. The modified bottleneck layer may include a first convolutional layer, multiple residual connection branches, and a second convolutional layer connected in sequence. The structural comparison between the standard ResNet bottleneck and the modified bottleneck is shown in the figure below. Figure 2 The specific modification ideas are as follows:

[0073] (1) Set a scale parameter s, which indicates how many groups the features of the input layer need to be divided into, for example Figure 2 The scale parameter in is set to 5. The feature matrix X output by the first 1*1 convolutional layer is divided into s groups according to the channel, and the features of each group can be expressed as x i ,i∈1,2,…,. Feature x i The number of channels is c i , can be calculated by the number of channels c of the feature matrix X. The specific formula is as follows:

[0074] c=c i *

[0075] (2) Then for each feature x after grouping iThe output value of this layer can be calculated by each residual connection branch respectively. The output value of the first group is directly equal to the input, and from the second group, the input x i First, the output y i-1 of the last group is taken as the input of this group, and then the output y i of this group is obtained after the convolution operation.

[0076] This method can make the extracted features contain multi-scale information, thereby improving the feature expression ability. The formula is as follows:

[0077]

[0078] Where, conv i represents the convolution operation on the i-th group of features.

[0079] (3) The outputs y i of all groups obtained in the last step are spliced by channel to obtain a matrix Y. Then Y is input into a second 1*1 size convolution layer (i.e. a second convolution layer) to calculate the output value of the bottleneck.

[0080] Based on any of the above embodiments, the automatic recognition method of student concentration can be generally divided into three categories: based on log files, based on sensor data, and based on computer vision technology. The method based on computer vision technology is the most mainstream recognition method at present, which can be further divided into a recognition model based on three-dimensional convolution and a recognition model based on two-dimensional convolution plus recurrent neural network. The recognition model based on three-dimensional convolution is to input the collected video into the recognition model at one time, and the model is recognized through a three-dimensional convolution model containing time dimension. The recognition model based on two-dimensional convolution plus recurrent neural network is to first extract features of each frame of the video, and then input the extracted feature sequence into a recurrent neural network for recognition and classification.

[0081] The existing problems in the field of student concentration recognition based on computer vision technology are as follows: the number of public data sets in the field of student concentration recognition is small, and the amount of data in the data set is small; face occlusion is easy to occur in the classroom, resulting in low accuracy of concentration recognition; the recognition model based on three-dimensional convolution has many parameters and low inference efficiency; the recognition model based on two-dimensional convolution plus recurrent neural network has poor long-distance feature capture ability, cannot be calculated in parallel, and has poor efficiency; the concentration of students is greatly related to local features and overall features, but the existing concentration recognition model often only selects a part of features for inference.

[0082] In view of the above problems and the characteristics of the classroom scene, the present application proposes a student concentration recognition model based on self-attention mechanism (DSA-SER). Figure 3The overall framework diagram of the student concentration recognition model based on the self-attention mechanism provided by the embodiment of the present application is divided into three parts: data enhancement, feature extraction, and classification.

[0083] Step 1: input data enhancement

[0084] Figure 4 The flowchart of the data enhancement method provided by the embodiment of the present application is shown in the figure. The input sample data is enhanced according to the enhancement process. Figure 4 The conventional enhancement method in the image field, the V-Mask method and the R-Extract method are used in the data enhancement module. First, the initial sample video frame sequence of the teaching process is input into the data enhancement module. In the module, the initial sample video frame is first enhanced by random translation, cropping, brightness adjustment and other conventional enhancement methods, then the V-Mask method is used to randomly cover the image area, and finally the R-Extract method is used to extract the sample video frame sequence. The extracted sample video frame sequence is input into the feature extraction network.

[0085] The implementation idea of the random covering algorithm (V-Mask) is as follows: for a video frame picture, it can be uniformly divided into 4*4 cells, i.e. image blocks. A covering probability p is set in advance, for example, p is set to 0.25. Each cell has a probability p of being replaced by a blank pixel, i.e. covering.

[0086] The implementation idea of the random frame extraction algorithm (R-Extract) is as follows: assuming that the total number of frames of a sample video is t frames, the sampling frequency is f, and the input length of the model is l, the random starting frame position s can be calculated, then the sample video frame sequence is extracted with s as the starting position and f as the sampling frequency. For the same video, a total of (-f*l-1) input sequences can be generated.

[0087] Step 2: feature extraction

[0088] The standard ResNet model is modified, and support for multi-scale feature extraction is added to its structure. A multi-scale feature extraction network MS-ResNet-50 is designed. The MS-ResNet-50 is divided into 5 Conv Stages, and the last 4 Conv Stages can be composed of modified bottleneck, as shown in Figure 2 .

[0089] In the feature extraction network, the MS-ResNet-50 model is used to extract the spatial dimension features of the video frame sequence, i.e. the multi-scale feature sequence, and then the multi-scale feature sequence is input into the classification network for concentration classification.

[0090] Step 3: classification

[0091] The core structure of the classification network is as follows Figure 3 As shown in the figure, it includes a multi-head self-attention layer, an addition and normalization layer, a feedforward neural network layer, and a fully connected layer (MLP head). The classification network can adopt the Transformer model, thus introducing the Transformer model to the field of attention recognition. The sequence feature extraction method in the classification network uses the self-attention mechanism, which enables the model to calculate the input sequence in parallel. Moreover, the correlation between any two vectors in the input sequence can be directly calculated. This is different from the situation in recurrent neural networks where the number of calculations required to calculate the correlation between two positions increases with the distance between them.

[0092] At the same time, the position encoding method in the self-attention mechanism was modified to include time information. After extracting the correlation features in the time dimension from the feature sequence, it was input into the fully connected layer for classification, thereby obtaining the student's concentration.

[0093] In summary, the present invention proposes a real-time student concentration recognition method for occlusion scenarios, which has the following advantages: a data enhancement method for videos is proposed, which provides a solution for scenarios where students are occluded in the classroom; the student concentration recognition model is optimized, and the self-attention mechanism is introduced into the field of concentration recognition, thereby improving the model's concentration recognition accuracy in long-sequence videos; the feature extraction network is modified, and a multi-scale feature fusion module is added to the model, so that the features extracted by the model contain both detailed information and overall information.

[0094] Based on any of the above embodiments, an embodiment of the present invention provides a student concentration recognition system. Figure 5 This is an architecture diagram of the student concentration recognition system provided by an embodiment of the present invention. Figure 5 As shown, the system specifically includes:

[0095] A video frame sequence acquisition module 510 is used to acquire a video frame sequence of a target student's learning video;

[0096] A multi-scale feature extraction module 520 is configured to perform multi-scale feature extraction on a video frame sequence based on a feature extraction network in a concentration recognition model and obtain a multi-scale feature sequence of the video frame sequence;

[0097] A concentration classification module 530 is used to classify the multi-scale feature sequence based on the classification network in the concentration recognition model and obtain the concentration of the target student;

[0098] Among them, the concentration recognition model is obtained based on the sample video frame sequence and the sample concentration training of students in the sample video frame sequence.

[0099] The system provided by the embodiment of the present invention obtains a video frame sequence of the target student during the learning process, and applies a deep learning model to first perform multi-scale feature extraction on the video frame sequence, so that the model can capture the temporal and spatial feature information in the video frame sequence, and simultaneously include detailed information and overall information. On this basis, the student's concentration is analyzed, thereby realizing automatic recognition of student concentration, saving labor costs while improving the accuracy and efficiency of student concentration recognition.

[0100] It is understandable that the detailed functional implementation of each of the above modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0101] In addition, an embodiment of the present invention provides another device for identifying student concentration, which includes: a memory and a processor;

[0102] The memory is used to store computer programs;

[0103] The processor is configured to implement the method in the above embodiment when executing the computer program.

[0104] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0105] Based on the method in the above embodiment, an embodiment of the present invention provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0106] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for identifying student concentration, characterized in that: include: S101 obtains a video frame sequence of a target student's learning video; S102 performs multi-scale feature extraction on the video frame sequence based on a feature extraction network in the concentration recognition model to obtain a multi-scale feature sequence of the video frame sequence; S103, based on the classification network in the concentration recognition model, classifies the multi-scale feature sequence into concentration levels and obtains the concentration level of the target student; The concentration recognition model is obtained based on a sample video frame sequence and a sample concentration training of students in the sample video frame sequence; The sample video frame sequence is obtained based on the following steps: Randomly mask each image block of the video frame in the initial sample learning video to obtain a sample learning video; Randomly extracting frames from the sample learning video to obtain the sample video frame sequence; S103 specifically includes: The classification network generates a position coding sequence corresponding to the multi-scale feature sequence based on the time information of the multi-scale feature sequence, and performs concentration classification based on the multi-scale feature sequence and its corresponding position coding sequence to obtain the concentration of the target student; The performing concentration classification based on the multi-scale feature sequence and its corresponding position coding sequence to obtain the concentration of the target student includes: Fusing the multi-scale feature sequence and its corresponding position coding sequence to obtain a fused feature sequence; Analyzing the correlation between any two features in the fused feature sequence based on a self-attention mechanism, and performing concentration classification based on the correlation to obtain the concentration of the target student; The feature extraction network includes multiple bottleneck layers; the bottleneck layer includes a first convolutional layer, multiple residual connection branches and a second convolutional layer connected in sequence.

2. A student concentration recognition system, characterized by: include: A video frame sequence acquisition module is used to acquire the video frame sequence of the target student's learning video; a multi-scale feature extraction module, configured to perform multi-scale feature extraction on the video frame sequence based on a feature extraction network in a concentration recognition model and obtain a multi-scale feature sequence of the video frame sequence; A concentration classification module is used to classify the multi-scale feature sequence based on the classification network in the concentration recognition model and obtain the concentration of the target student; The concentration recognition model is obtained based on a sample video frame sequence and a sample concentration training of students in the sample video frame sequence; It also includes a training sample acquisition module; The training sample acquisition module is used to randomly mask each image block of the video frame in the initial sample learning video to obtain a sample learning video, and randomly extract frames from the sample learning video to obtain the sample video frame sequence; The concentration classification module is specifically configured to generate, by a classification network, a position coding sequence corresponding to the multi-scale feature sequence based on time information of the multi-scale feature sequence, and perform concentration classification based on the multi-scale feature sequence and its corresponding position coding sequence to obtain the concentration of the target student; The concentration classification module is specifically used to fuse the multi-scale feature sequence and its corresponding position code sequence to obtain a fused feature sequence, analyze the correlation between any two features in the fused feature sequence based on the self-attention mechanism, and perform concentration classification based on the correlation to obtain the concentration of the target student; The feature extraction network includes multiple bottleneck layers; the bottleneck layer includes a first convolutional layer, multiple residual connection branches and a second convolutional layer connected in sequence.

Images