Student attention evaluation method based on double-flow sliding attention network

Abstract

The embodiment of the application discloses a student attention evaluation method based on a double-flow sliding attention network, relates to the technical field of intelligent teaching, and comprises the following steps: acquiring image information and physiological signals of a target student in a teaching period, inputting preprocessed data into a trained model to identify head posture and physiological indexes; judging the head posture type, and calculating a first attention evaluation index; if the first attention evaluation index is lower than a threshold value, further calculating a second attention evaluation index based on the physiological indexes in the time period corresponding to the head posture type; and further calculating an attention evaluation result. The embodiment of the application can automatically, timely and accurately give the evaluation index of attention based on the head posture and the physiological indexes by simultaneously collecting the physiological signals and the image information of the target student, can effectively avoid misjudgment, is beneficial to giving a more accurate attention evaluation result, and can help teachers better understand the learning state of students in the network teaching scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent teaching, and particularly relates to a student attention evaluation method based on a double-flow sliding attention network. BACKGROUND

[0002] In a traditional teaching mode, the interaction between teachers and students mainly depends on language and written expression, but this mode has problems such as low communication efficiency and limited understanding. In this case, the teacher often cannot timely feel the class mood and specific concentration of the students and adjust the teaching state and content.

[0003] At present, with the development of Internet technology, online education occupies an increasingly large share in the education industry. The network teaching in the form of video shows explosive growth. Student attention concentration is one of the prerequisites for ensuring the quality of the classroom. Unlike the traditional classroom, in the online class scene, the teacher can only communicate with the students through video. At the same time, the teacher can only pay attention to a small number of student videos, and it is difficult to grasp the attention of the students during the class. Especially it is difficult to be specific to the learning state of a single student. Therefore, real-time recognition of the learning state of the students and avoidance of a negative class mood are important links for improving the quality of teaching, which has important significance for improving the teaching effect, optimizing the teaching content, and supporting personalized learning. SUMMARY

[0004] The embodiments of the present application provide a student attention evaluation method based on a double-flow sliding attention network, to solve the defect that the attention of a target student in a teaching process cannot be accurately judged in the related art. The technical solution is as follows:

[0005] In a first aspect, the embodiments of the present application provide a student attention evaluation method based on a double-flow sliding attention network, comprising:

[0006] Obtaining image information of a target student in a teaching period and collecting physiological signals of the target student in the teaching period, the physiological signals including electroencephalogram signals and electrooculogram signals; and pre-processing the image information and the physiological signals;

[0007] Inputting the pre-processed image information into a trained head posture recognition model to recognize the head posture of each image frame; and inputting the pre-processed physiological signals into a trained physiological signal recognition model to recognize the physiological indicators of the student at each time point;

[0008] Judging the head posture type of the target student in different time periods according to the head posture of each image frame, and outputting a first attention evaluation index according to the head posture type and the corresponding duration;

[0009] if the first attention evaluation index is lower than a first attention evaluation index threshold, further outputting a second attention evaluation index based on a physiological index of the head posture type in a corresponding time period;

[0010] outputting an attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period.

[0011] In an optional implementation of the first aspect, the method further includes:

[0012] The head posture includes a yaw angle, a pitch angle and a roll angle of the head, and the head posture type of the target student in each time period is determined based on the yaw angle, the pitch angle and the roll angle.

[0013] The head posture type includes a screen-facing posture and other head posture types.

[0014] In an optional implementation of the first aspect, the outputting the first attention evaluation index according to the head posture type and the corresponding duration includes:

[0015] The duration of the corresponding head posture type in each time period is obtained, and the first attention evaluation index is calculated based on the head posture type and the corresponding duration.

[0016] If the head posture type is the screen-facing posture, it is determined that the first attention evaluation index in the corresponding time period is not less than the first attention evaluation index threshold.

[0017] If the head posture type is the other head posture type, it is determined whether the duration of the other head posture type is greater than a posture duration threshold, and if the duration is greater than the posture duration threshold, it is determined that the first attention evaluation index in the corresponding time period is less than the first attention evaluation index threshold.

[0018] In an optional implementation of the first aspect, after the first attention evaluation index in the corresponding time period is determined to be less than the first attention evaluation index threshold, the method further includes:

[0019] The baseline physiological index of the target student is obtained based on historical data, a difference between the physiological index in the time period in which the target student is in the other head posture type and the baseline physiological index is calculated, and an average change rate of the physiological index in the time period in which the target student is in the other head posture type is calculated, the average change rate including an electroencephalogram change rate and an eyeball displacement change rate.

[0020] The second attention evaluation index is calculated based on the average change rate and the difference.

[0021] If the brain electrical signal change rate, the eyeball displacement change rate, and the difference value are all greater than the corresponding physiological indicator threshold, it is determined that the second attention evaluation indicator is less than a second attention evaluation indicator threshold.

[0022] In an optional implementation of the first aspect, the outputting the attention evaluation result of the target student in the teaching period based on the first attention evaluation indicator and / or the second attention evaluation indicator of the target student in the teaching period includes:

[0023] obtaining a number of times and a corresponding duration that the first attention evaluation indicator of the target student is lower than the first attention evaluation indicator and / or a number of times and a corresponding duration that the second attention evaluation indicator is lower than the second attention evaluation indicator;

[0024] obtaining a number of times and a corresponding duration that the first attention evaluation indicator of the target student is lower than the first attention evaluation indicator and / or a number of times and a corresponding duration that the second attention evaluation indicator is lower than the second attention evaluation indicator;

[0025] In an optional implementation of the first aspect, the inputting the preprocessed image information into the trained head posture recognition model to recognize the head posture of each image frame includes:

[0026] obtaining each image frame of the preprocessed RGB video of the target student, inputting each video frame into two data streams of the trained head posture recognition model for feature fusion, and then inputting the full connection layer of the head posture recognition model to output a rotation matrix;

[0027] obtaining the Euler angle of the image frame based on the rotation matrix.

[0028] In a second aspect, the embodiments of the present application further provide a student attention evaluation device based on a double-stream sliding attention network, including:

[0029] an information collection module, configured to obtain image information of a target student in a teaching period, and collect physiological signals of the target student in the teaching period, the physiological signals including brain electrical signals and electro-oculogram signals; and pre-process the image information and the physiological signals;

[0030] an identification module, configured to input the pre-processed image information into a trained head posture recognition model to recognize the head posture of each image frame; and input the pre-processed physiological signals into a trained physiological signal recognition model to recognize the physiological indicators of the student at each time point;

[0031] The first evaluation module is configured to determine a head posture type of the target student in different time periods according to the head postures of the target student in the image frames, and output a first attention evaluation index according to the head posture type and a corresponding duration.

[0032] The second evaluation module is configured to, when the first attention evaluation index is lower than a first attention evaluation index threshold, further output a second attention evaluation index based on a physiological index in a time period corresponding to the head posture type.

[0033] The attention evaluation module is configured to output an attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period.

[0034] In an optional implementation of the first aspect, the attention evaluation module is configured to output the attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period, including:

[0035] The first evaluation module is configured to obtain a number of times and a corresponding duration that the first attention evaluation index of the target student is lower than the first attention evaluation index, and / or the second evaluation module is configured to obtain a number of times and a corresponding duration that the second attention evaluation index is lower than the second attention evaluation index.

[0036] The attention evaluation module is configured to calculate the attention evaluation result of the target student in the teaching period based on the number of times and the duration.

[0037] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the methods provided in the first aspect of the embodiments of the present application when executing the program.

[0038] In a fourth aspect, a non-transitory computer-readable storage medium is provided, which stores a computer program, and the computer program is executable on a processor to implement the steps of the method provided in the first aspect of the embodiments of the present application.

[0039] The technical solutions provided by some embodiments of the present application have at least the following beneficial effects:

[0040] The student attention evaluation method based on the double-flow sliding attention network provided in the embodiment of the application can comprehensively reflect the actual state of the student in class by simultaneously collecting physiological signals and image information of the target student, identifying the head posture and physiological indicators of the target student, and can automatically, in real time and accurately give the identification result of the head posture type and the physiological indicators by identifying through the trained neural network; the first evaluation index is judged based on the head posture type and the corresponding duration, and then it is judged whether the second evaluation index needs to be calculated according to the evaluation result of the first evaluation index, so that the computing power can be saved to the maximum; when the first evaluation index is less than the first attention evaluation index threshold, the second attention evaluation index based on the physiological signals is used to make a judgment again, which can effectively avoid misjudgment and is beneficial to giving a more accurate attention evaluation result, so that the teacher can better understand the learning state of the student in real time and accurately in the network teaching scene. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 is a flowchart of a student attention evaluation method based on a double-flow sliding attention network according to an embodiment of the application;

[0043] Figure 2 is a schematic diagram of an application scenario according to an embodiment of the application;

[0044] Figure 3 is a structural diagram of a trained head posture recognition model according to an embodiment of the application;

[0045] Figure 4 is a structural diagram of a trained physiological signal recognition model according to an embodiment of the application;

[0046] Figure 5 is a structural diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be clearly and completely described below in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0048] The terms "comprise", "comprising", "include", "including", "have" and "having" and any variations thereof in the specification and in the claims are intended to cover both the exclusive and the non-exclusive inclusion of the stated steps or modules. For example, a process, method, system, product, or apparatus that comprises a list of steps or modules is not necessarily limited to those listed steps or modules, but can include additional steps or modules not expressly listed or can include steps or modules inherent to the process, method, system, product, or apparatus.

[0049] It should be noted that the terms "first" and "second" in the present application are only to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first" and "second" can be interchanged in a specific order or sequence as allowed. It should be understood that the objects distinguished by "first" and "second" can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0050] Next, in combination with Figure 1 Taking an example of executing the student attention evaluation method based on the dual-stream sliding attention network at the terminal of the target student, a student attention evaluation method based on a dual-stream sliding attention network is introduced, which comprises the following steps:

[0051] S1, acquiring image information of a target student in a teaching period, and collecting physiological signals of the target student in the teaching period, the physiological signals including electroencephalogram signals and electrooculogram signals; and pre-processing the image information and the physiological signals.

[0052] Optionally, the image information of the student can be acquired through a camera mounted on a net class terminal used by the student, and the image data of the student can also be acquired through an external camera, a pan-tilt camera, etc. The physiological signals of the student can be acquired through a wearable device, for example, the electroencephalogram (EEG) signals of the student can be acquired through a head-mounted device, and the electrooculogram (EOG) signals can be acquired through electrodes around the eyes.

[0053] Optionally, the pre-processing of the image includes collecting a video in the teaching period, and the video can be pre-processed through feature extraction, cutting, portrait recognition, etc. to finally acquire an RGB image of a single frame of the student. After the electroencephalogram signals and the electrooculogram signals are acquired, the signals can be filtered to filter out noise, and information can be extracted through binarization. The present application does not limit this.

[0054] S2, inputting the pre-processed image information into a trained head posture recognition model to recognize the head posture of each image frame; and inputting the pre-processed physiological signals into a trained physiological signal recognition model to recognize the physiological indicators of the student at each time.

[0055] Specifically, after the head pose corresponding to each frame of image is extracted, the data is packaged and saved, and the timestamp corresponding to each head pose is retained, and the head pose log is recorded based on time sequence;

[0056] The physiological index is the recognition result of the electroencephalogram signal and the electrooculogram signal. Specifically, the inventors find that the cognitive behavior and psychological activity of a person have strong correlation with the electroencephalogram signal, and the analysis based on the electroencephalogram signal can reflect the emotional fluctuation of the person. The electrooculogram signal is caused by the potential difference between the cornea and the retina, and can be used to reflect the movement of the eyeball. The eye movement in reading can be divided into several steps such as back vision, saccade and fixation. Different eye movements will produce different characteristic eye movement waveforms. By detecting the electrooculogram signal and analyzing the eye movement waveform, whether the person is focused on the reading state can be reflected.

[0057] S3 determines the head pose type of the target student in different time periods according to the head pose of each image frame, and outputs a first attention evaluation index according to the head pose type and the corresponding duration.

[0058] Specifically, the head pose includes the yaw angle, the pitch angle and the roll angle of the head, and the head pose type of the target student in each time period is determined based on the yaw angle, the pitch angle and the roll angle.

[0059] The head pose type includes a screen-facing pose and other head pose types.

[0060] As an example, the head pose type is divided based on the yaw angle, the pitch angle and the roll angle of the head pose, as shown in Table 1. Other head poses can be divided into looking up, looking down and looking around. Here, this is only used to further illustrate the embodiments of the present application, and the present application is not limited thereto:

[0061] Table 1: Classification rule of student head pose type

[0062]

[0063]

[0064] As an example, if the head pose is screen-facing, and the first attention evaluation index is greater than the first attention evaluation index threshold, then the first attention evaluation index of the target student is higher. If the head pose is looking down, and the looking down duration is 15 minutes, and the duration threshold corresponding to the first attention evaluation index threshold is 20 minutes, then the first attention evaluation index of the target student is higher. If the looking down duration is 30 minutes, then the first attention evaluation index of the target student is lower.

[0065] Based on this, by judging the head posture type, the gaze direction of the target student can be judged, and it can be judged whether it falls on the screen. If it falls on the screen, it can be judged whether the student is in a state of concentrating on listening; otherwise, it is judged that the student may not be concentrating on the content displayed on the screen.

[0066] S4If the first attention evaluation index is lower than the first attention evaluation index threshold, further based on the physiological indicators in the time period corresponding to the head posture type, a second attention evaluation index is output.

[0067] It can be understood that it is not accurate enough to judge whether the student is concentrating only from the fact that the student is not concentrating on the screen. There are some cases where the student needs to think, check the notes, etc. Therefore, the time when the student is in other head posture types needs to be considered, and the emotion recognition of the electroencephalogram signal and the eye waveform recognition of the electrooculogram signal are further combined to judge whether the student is in a state of concentration.

[0068] Further, if the first attention evaluation index of the target student is lower than the threshold, it indicates that the student spends a long time in other head posture types, and the physiological indicators in the time period when the student is in other head posture types need to be combined to further evaluate; therefore, the physiological indicators in the time period when the student is in other head posture types are obtained, and based on this, a second attention evaluation index is obtained.

[0069] As an example, when the physiological indicators obtained by judging the electroencephalogram information and the electrooculogram information are that the brain waves are stable and the electrooculogram signal is smooth, that is, the change rates of the two signals in a given time period are low, the corresponding second attention evaluation index is high, otherwise the second attention evaluation index is low.

[0070] S5Based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period, an attention evaluation result of the target student in the teaching period is output.

[0071] Specifically, based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period, the attention during the teaching period is calculated comprehensively. If the student keeps staring at the screen or keeps other head postures for a short time during the teaching period, only the first attention evaluation index is generated. If the student only exists for a long time in the case of keeping other head postures during the teaching period, the second attention evaluation index is further given.

[0072] Specifically, the application scenario of the embodiment of the present application is as shown in Figure 2 The image information and physiological signals of the student are obtained by the image acquisition device and the physiological signal acquisition device connected by the online course terminal and uploaded to the server. The image information and physiological signals are preprocessed by the server, and then the evaluation index is calculated.

[0073] In a specific embodiment, the outputting the first attention evaluation index according to the head posture type and the corresponding duration includes:

[0074] obtaining the duration of the corresponding head posture type in each time period, and calculating the first attention evaluation index based on the head posture type and the corresponding duration;

[0075] if the head posture type is the screen-facing posture, determining that the first attention evaluation index in the corresponding time period is not less than the first attention evaluation index threshold;

[0076] if the head posture type is the other head posture type, determining whether the duration of the other head posture type is greater than the posture time threshold, and if greater than the posture time threshold, determining that the first attention evaluation index in the corresponding time period is less than the first attention evaluation index threshold.

[0077] wherein, when the head posture type is screen-facing, it can be determined that the first attention evaluation index is not less than the first attention evaluation index threshold, and no further determination is required.

[0078] In a specific embodiment, after determining that the first attention evaluation index in the corresponding time period is less than the first attention evaluation index threshold, the method further includes:

[0079] obtaining a baseline physiological index of the target student based on historical data, calculating the difference between the physiological index of the target student in the time period of the other head posture type and the baseline physiological index, and calculating the average change rate of the physiological index in the time period of the other head posture type, the average change rate including the electroencephalogram signal change rate and the eyeball displacement change rate;

[0080] calculating the second attention evaluation index based on the average change rate and the difference;

[0081] if the electroencephalogram signal change rate, the eyeball displacement change rate, and the difference are all greater than the corresponding physiological index threshold, determining that the second attention evaluation index is less than the second attention evaluation index threshold;

[0082] Specifically, when the electroencephalogram signal change rate, the eyeball displacement change rate, and the difference are all greater than the corresponding physiological index threshold, it is determined that the target student is in a state of emotional fluctuation and eyeball movement at this time, indicating that the target student is not in a state of concentration, and thus the second attention evaluation index is less than the second attention evaluation index threshold.

[0083] Specifically, the EEG and EOG signals of the target student during focused reading can be collected to obtain baseline physiological indicators of the EEG and EOG, so that the second attention evaluation index can be calculated more accurately and for a single student.

[0084] In a specific embodiment, the first attention evaluation index and / or the second attention evaluation index of the target student in the time period are output to obtain an attention evaluation result of the target student in the teaching time period.

[0085] The number of times and the corresponding duration when the first attention evaluation index of the target student is lower than the first attention evaluation index and / or the number of times and the corresponding duration when the second attention evaluation index is lower than the second attention evaluation index are obtained.

[0086] The attention evaluation result of the target student in the teaching time period is calculated based on the number of times and the duration.

[0087] As an example, by counting the number of times and the duration when the two attention evaluation indexes are lower than the corresponding threshold, the total evaluation result can be calculated, for example, the number of times when the first attention evaluation index is lower than the threshold is 0, and the first attention evaluation index is not lower than the threshold, so there is no need to calculate the second attention evaluation index, and the corresponding attention evaluation result can be given a high rating. If, for example, the number of times when the first attention evaluation index is lower than the threshold is 2, and the duration is 1 hour and 5 minutes respectively, and one of the times is longer, the second attention evaluation index is further calculated. If the second attention evaluation index is lower than the threshold, the duration is 5 minutes, and the corresponding attention evaluation result can be given a general rating. This is only an example of the embodiments of the present application and should not be considered as a limitation of the present application.

[0088] The specific evaluation result can be in the form of a score, a rating, etc. The attention evaluation result can be obtained based on a preset attention degree mapping table according to the result of the weighted calculation, and the score of the weighted calculation can directly reflect the level of attention.

[0089] Further, after the target student has a lower attention, a reminder can be sent to the teacher and / or the student to remind the teacher and the student to interact and improve the student's attention, or a prompt message can be displayed on the display screen of the online course terminal to remind the user to focus on listening, which is not limited by the present application.

[0090] Specifically, the preprocessed image information is input into the trained head posture recognition model to recognize the head posture of each image frame, including:

[0091] Each image frame of the preprocessed RGB video of the target student is obtained. Each video frame is input into the two data streams of the trained head pose recognition model for feature fusion and then input into the fully connected layer of the head pose recognition model to output a rotation matrix.

[0092] The Euler angles of the image frame are calculated based on the rotation matrix.

[0093] like Figure 3 As shown, the architecture details of the trained head pose recognition model are as follows: This network is a two-stream sliding attention network, where the input image frames are divided into two data streams for processing separately; specifically, two basic DSC blocks are used for feature fusion: DSCr(c)≡[DSC(3×3,c)-BN-ReLU] and DSCt(c)≡[DSC(3×3,c)-BN-Tanh], where c is a parameter and BN represents batch normalization. Trans represents a custom Transformer encoder with 8 heads, 3 encoder layers, 32 expected features (denoted as k) in the encoder, and 64 feedforward dimensions. The first data stream has [DSCr(16)-AvgPool(2×2)-DSCr(32)-DSCr(32)-AvgPool(2×2)]-[DSCr(32)-DSCr(32)-Trans-AvgPoo1(2×2)]-[DSCr(32)-DSCr(32)-Trans], with each square bracket forming a stage, and AvgPoo1 being average pooling. The second data stream has the same structure as the first, except that it uses DSCt instead of DSCr and max pooling MaxPoo1 instead of average pooling AvgPoo1. The feature fusion module consists of element-wise multiplications of features from both data streams. The prediction head consists of 1×1 convolutional layers, reducing the number of channels from 32 to 16 (the first stage has AvgPoo1 after the 1×1 convolution). The flat array is then fed into a linear layer with six output units, and the final head pose is calculated by taking a weighted average of the three predicted head poses (one for each stage), as shown in the formula. As shown.

[0094] Furthermore, the rotation matrix and the true rotation matrix are calculated using 6d degrees of freedom prediction and geodesic distance loss, including:

[0095] By simply discarding the last column vector of the rotation matrix and performing a Gram-Schmidt mapping within the representation itself, this 3×3 matrix is ​​simplified to a six-parameter rotation representation.

[0096]

[0097] The predicted 6D representation matrix can then be mapped back to SO(3), the group of rotation operations in three dimensions.

[0098]

[0099] The predicted 6D representation matrix can be mapped back to SO(3). From this, the remaining column vectors are simply determined by the cross product, which ensures that the orthogonality constraint is satisfied for the resulting 3x3 matrix.

[0100]

[0101] The predicted 6D representation matrix can be mapped back to SO(3). From this, the remaining column vectors are simply determined by the cross product, which ensures that the orthogonality constraint is satisfied for the resulting 3x3 matrix. Thus, the network only has to predict 6 parameters, which are mapped to a 3x3 rotation matrix in the subsequent transformation, while also satisfying the orthogonality constraint. A commonly used loss function for head pose related tasks is the l2 norm. However, using the Frobenius norm to measure the distance between two matrices would break the SO(3) manifold geometry. Instead, the shortest path between two 3D rotations is geometrically interpreted as the geodesic distance. Let R p and R gt be the estimated and the true rotation matrices respectively, then the geodesic distance between two rotation matrices is defined as:

[0102]

[0103] Further, the specific steps of converting the rotation matrix into Euler angle representation are as follows:

[0104] Let the Euler angles of the three axes x, y, z be θ x , θ y , θ z , respectively, and rotate in the order of z-y-x in the way of intrinsic rotation, and the sine values, cosine values are s x , c x , s y , c y , s z , c z Then the rotation matrix is:

[0105] R(θ z , θ y , θ z ) = R(z, θ z ) · R(y, θ y ) · R(x, θ z );

[0106]

[0107] Euler angles from rotation matrix:

[0108]

[0109] Euler angles from solving equations:

[0110]

[0111]

[0112]

[0113] Optionally, the head pose recognition model is trained by historical measured student head pose data and / or commonly used 300W_LP dataset as a training set.

[0114] Further, please refer to Figure 3 The training process of the physiological signal recognition model is as follows:

[0115] The preprocessed electroencephalogram (EEG) and electrooculogram (EOG) signals are input into a feature extraction and fusion module, where the EEG and EOG signals can be 30s continuous waveform graphs, and the start time can be the start time of the long-time head posture anomaly record.

[0116] Subsequently, they are input into a ResNet50 deep convolutional neural network for learning and inference. In order to learn global feature information, an attention mechanism is improved and adopted without increasing much computational cost, i.e., a channel and spatial joint attention (CSJA) block and a squeeze and excitation (SE) block to recalibrate features. Correspondingly, a channel complexity adjustment factor (f) is introduced to uniformly expand the number of channels, which then cooperates with the attention block.

[0117] Further, the features are input into a spatial joint attention (CSJA) block, in which the original features are globally averaged and maximally pooled in the "squeeze" part. In the excitation part, convolution is used to form a new feature map containing the importance of position information. In the last feature fusion part, attention feature maps trained at the spatial and channel levels are added respectively. Finally, the adjusted feature map is obtained and fused with double attention.

[0118] A channel attention mechanism called SE block is introduced at the bottom of the network, which aims to adaptively recalibrate the abstract features extracted by the encoder. First, the spatial information corresponding to each channel is compressed, such as the global response of the signal within 30s. Second, two convolution operations are used to learn nonlinear features and perform channel-level feature selection. Finally, tensor multiplication enhances useful features and suppresses ineffective features.

[0119] Finally, it is input into the classification prediction head, which consists of multiple linear projection layers, and the final prediction is divided into 2 groups, emotional stability and emotional fluctuation, smooth eye movement and saccade, and the final categories include: emotional stability smooth eye movement, emotional fluctuation smooth eye movement, emotional stability saccade, emotional fluctuation saccade.

[0120] According to the above scheme, the SE module is specifically designed as follows:

[0121] The SE module mainly includes two parts, namely, compression Squeeze and excitation Excitation. Since the output is generated by the summation of all channels, the channel correlation is implicitly embedded in the network, but it is intertwined with the local spatial correlation captured by the filter. Therefore, it is expected to enhance the learning of convolutional features by explicitly modeling the interdependence of channels, so that the network can improve its sensitivity to information features that can be utilized by subsequent transformations. Therefore, the module hopes to provide a way to obtain global information and recalibrate the filter response in two steps (Squeeze and Excitation) before inputting the next transformation. The filter is the convolution kernel of the neural network.

[0122] First, consider the signal of each channel in the output feature map.

[0123] Further, since each learned filter works with a local receptive field, each unit of the transformation output U cannot utilize the contextual information outside the region. To solve this problem, global spatial information can be compressed into channel descriptors by generating channel statistics using a global average pool. To be precise, z∈R C is the result of performing global average pooling on the feature U in the spatial dimension HxW, so each element of Z is represented as follows:

[0124]

[0125] In order to utilize the information aggregated in the compression Squeeze operation, the second operation Excitation needs to be performed, which aims to fully capture the channel dependency, i.e., the channel of the feature map. To achieve this goal, the function must meet two criteria: first, it must be able to operate flexibly, and it must be able to learn the nonlinear relationship between channels; second, it must learn non-exclusive relationships, so that multiple channels can be strengthened (rather than similar one-hot, which can only strengthen a certain channel feature). In order to meet these criteria, the module chooses to use a simple gating mechanism with sigmoid activation.

[0126] In order to meet the above two criteria, the following transformation form is adopted:

[0127] s = F ex (z,W) = σ(g(z,W)) = σ(W2δ(W1z));

[0128] Step 1.5.4, in order to limit the complexity of the model and make it universal, two FC layers are used to parameterize the gating mechanism, i.e. a dimension reduction layer with a dimension reduction rate r, a ReLU, and then a dimension elevation layer, and then go to the channel dimension of the output feature map. The final output of the Block is obtained by using the activation to rescale the feature map. After obtaining s, the final output of the SE Block can be obtained by the following formula:

[0129]

[0130] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, refer to the method embodiment of the present application.

[0131] An example embodiment of the present application provides a student attention evaluation device based on a double-flow sliding attention network, which can be realized as all or part of a terminal by software, hardware or a combination of both, and can also be integrated as an independent module on a server. The student attention evaluation device based on a double-flow sliding attention network in the embodiment of the present application can be applied to a terminal or a cloud, and the device comprises a student attention evaluation device based on a double-flow sliding attention network, which comprises:

[0132] An information collection module is configured to acquire image information of a target student in a teaching period and collect physiological signals of the target student in the teaching period, the physiological signals including electroencephalogram signals and electrooculogram signals; and the image information and the physiological signals are preprocessed;

[0133] An identification module is configured to input the preprocessed image information into a trained head posture recognition model to identify head postures of each image frame; and input the preprocessed physiological signals into a trained physiological signal recognition model to identify physiological indexes of the student at each time point;

[0134] A first evaluation module is configured to determine head posture types of the target student in different time periods according to the head postures of each image frame, and output a first attention evaluation index according to the head posture types and corresponding time durations;

[0135] A second evaluation module is configured to, when the first attention evaluation index is lower than a first attention evaluation index threshold, further output a second attention evaluation index based on the physiological indexes in the time period corresponding to the head posture type;

[0136] An attention evaluation module is used to output the attention evaluation result of the target student during the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student during the teaching period.

[0137] Furthermore, the attention evaluation module is used to output the attention evaluation result of the target student during the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student during the teaching period, including:

[0138] The first evaluation module obtains the number of times and the corresponding duration when the target student's first attention evaluation index is lower than the first attention evaluation index and / or the second evaluation module obtains the number of times and the corresponding duration when the second attention evaluation index is lower than the second attention evaluation index.

[0139] The attention evaluation module calculates the target student's attention evaluation result during the teaching period based on the weighted average of the number of times and the duration.

[0140] It should be noted that the student attention evaluation device based on a dual-stream sliding attention network provided in the above embodiments is only illustrated by the division of the above functional modules when executing the student attention evaluation method based on a dual-stream sliding attention network. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiments and the student attention evaluation method embodiments based on a dual-stream sliding attention network belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0141] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0142] Please see Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0143] like Figure 5 As shown, the electronic device 500 includes a processor 501 and a memory 502.

[0144] In the embodiments of the present application, the processor 501 is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor 501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array).

[0145] The processor 501 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state.

[0146] The memory 502 can include one or more computer-readable storage media, which can be non-transitory. The memory 502 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction for being executed by the processor 501 to implement the method in the embodiments of the present application.

[0147] In some embodiments, the electronic device 500 further includes a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502, and the peripheral device interface 503 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 503 through a bus, a signal line, or a circuit board. Specifically, the peripheral device interface 503 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502.

[0148] In some embodiments of the present application, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations in this regard.

[0149] The display screen 504 is configured to display a UI. The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 504 is a touch display screen, the display screen 504 is further configured to capture touch signals on or above the surface of the display screen 504. The touch signals can be input to the processor 501 as control signals for processing. In this case, the display screen 504 can also be configured to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0150] In some embodiments of the present application, the display screen 504 can be one, arranged on the front panel of the electronic device 500; in some other embodiments of the present application, the display screen 504 can be at least two, arranged on different surfaces of the electronic device 500 or in a folding design; in some other embodiments of the present application, the display screen 504 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 500. Even, the display screen 504 can also be arranged in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 504 can be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0151] The camera 505 is configured to capture images or videos. Optionally, the camera 505 includes a front camera and a rear camera. Generally, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blurring function of the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments of the present application, the camera 505 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0152] The audio circuit 506 can include a microphone and a speaker. The microphone is configured to capture sound waves of a user and the environment, and convert the sound waves into an electrical signal input to the processor 501 for processing. For the purpose of stereo sound capture or reduction, the microphone can be multiple, arranged at different positions of the electronic device 500. The microphone can also be an array microphone or an omnidirectional capture microphone.

[0153] The power supply 507 is configured to supply power to various components in the electronic device 500. The power supply 507 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 507 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery that is charged through a wired line, and the wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0154] The structural block diagram of the electronic device shown in the embodiments of the present application does not constitute a limitation on the electronic device 500, and the electronic device 500 can include more or fewer components than shown, or combine certain components, or adopt a different arrangement of components.

[0155] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method of any of the preceding embodiments. The computer readable storage medium can include, but is not limited to, any type of disk including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic or optical card, a nanosystem (including molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0156] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such an understanding, the above technical solutions essentially or in other words the parts that make contributions to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0157] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A student attention evaluation method based on a dual-flow sliding attention network, characterized in that, The method comprises the following steps: acquiring image information of a target student in a teaching period and collecting physiological signals of the target student in the teaching period, the physiological signals including electroencephalogram signals and electrooculogram signals; preprocessing the image information and the physiological signals; inputting the preprocessed image information into a trained head posture recognition model to recognize head postures of each image frame; inputting the preprocessed physiological signals into a trained physiological signal recognition model to recognize physiological indexes of the student at each time point; judging the head posture types of the target student in different time periods according to the head postures of each image frame, and outputting a first attention evaluation index according to the head posture types and the corresponding durations; if the first attention evaluation index is lower than a first attention evaluation index threshold, further outputting a second attention evaluation index based on the physiological indexes in the time period corresponding to the head posture type; outputting an attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period; wherein the trained head posture recognition model is a double-flow sliding attention network, and the double-flow sliding attention network divides the input image frames into two data flows for processing, specifically including: the structure of the first data flow includes 16-bit DSCr blocks, a 2X2 average pooling layer, two 32-bit DSCr blocks, a 2X2 average pooling layer, two 32-bit DSCr blocks, a Trans, a 2X2 average pooling layer, two 32-bit DSCr blocks, and a Trans connected in sequence; the second data flow replaces the DSCr blocks in the first data flow with DSCt blocks and replaces the average pooling layer in the first data flow with a maximum pooling layer, and the other structures are the same as those of the first data flow; a feature fusion module is composed of element-wise multiplication of features from the two data flows; a prediction head is composed of a 1x1 convolution layer for reducing the number of channels from 32 to 16; a flat array is fed into a linear layer with 6 output units, and a weighted average is taken to calculate the final head posture using three predicted head postures; wherein the DSCr block is represented as DSCr(c)≡[DSC(3×3,c)-BN-ReLU], the DSCt block is represented as DSCt(c)≡[DSC(3×3,c)-BN-Tanh], and c is a parameter, BN represents batch normalization; Trans represents a custom Transformer encoder with 8 heads, 3 encoder layers, 32 expected features in the encoder, and 64 feedforward dimensions.

2. The method of claim 1, wherein, The head posture includes a yaw angle, a pitch angle, and a roll angle of the head, and the head posture types of the target student in each time period are judged based on the yaw angle, the pitch angle, and the roll angle. The head posture types include a straight-to-screen posture and other head posture types.

3. The method of claim 2, wherein, The first attention evaluation index is outputted according to the head posture types and the corresponding durations. obtaining a duration of a corresponding head posture type in each time period, and calculating a first attention evaluation index based on the head posture type and the corresponding duration; if the head posture type is the front screen posture, determining that the first attention evaluation index in the corresponding time period is not less than the first attention evaluation index threshold; if the head posture type is the other head posture type, determining whether the duration of the other head posture type is greater than a posture time threshold, and if greater than the posture time threshold, determining that the first attention evaluation index in the corresponding time period is less than the first attention evaluation index threshold.

4. The method of claim 3, wherein, After determining that the first attention evaluation index in the corresponding time period is less than the first attention evaluation index threshold, the method further includes: obtaining a baseline physiological index of the target student based on historical data, calculating a difference between the physiological index of the target student in a time period of the other head posture type and the baseline physiological index, and calculating an average change rate of the physiological index in the time period of the other head posture type, the average change rate including an electroencephalogram signal change rate and an eye displacement change rate; calculating a second attention evaluation index based on the average change rate and the difference; if the electroencephalogram signal change rate, the eye displacement change rate, and the difference are all greater than corresponding physiological index thresholds, determining that the second attention evaluation index is less than a second attention evaluation index threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The method of outputting an attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period includes: obtaining the number of times and the corresponding duration when the first attention evaluation index of the target student is lower than the first attention evaluation index, and / or the number of times and the corresponding duration when the second attention evaluation index is lower than the second attention evaluation index; calculating an attention evaluation result of the target student in the teaching period based on the number of times and the duration.

6. The method of claim 1, wherein, The method of inputting the preprocessed image information into a trained head posture recognition model to recognize the head posture of each image frame includes: obtaining each image frame of the preprocessed RGB video of the target student, inputting each video frame into two data streams of a trained head posture recognition model for feature fusion, and then inputting the feature fusion result into a fully connected layer of the head posture recognition model to output a rotation matrix; calculating the Euler angle of the image frame based on the rotation matrix.

7. A device for student attention evaluation based on the dual-flow sliding attention network of any one of claims 1-6, characterized in that, The device includes: an information acquisition module configured to obtain image information of a target student in a teaching period, and to acquire physiological signals of the target student in the teaching period, the physiological signals including electroencephalogram signals and electrooculogram signals; and to preprocess the image information and the physiological signals; The recognition module is configured to input the preprocessed image information into a trained head posture recognition model to recognize head postures of each image frame; and input the preprocessed physiological signal into a trained physiological signal recognition model to recognize physiological indexes of the student at each time point. The first evaluation module is configured to determine head posture types of the target student in different time periods according to the head postures of each image frame, and output a first attention evaluation index according to the head posture types and corresponding time durations. The second evaluation module is configured to, when the first attention evaluation index is lower than a first attention evaluation index threshold, further output a second attention evaluation index based on the physiological indexes in the time period corresponding to the head posture type. The attention evaluation module is configured to output an attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period.

8. The student attention evaluation device based on a dual-flow sliding attention network according to claim 7, characterized in that, The attention evaluation module is configured to output an attention evaluation result of the target student in the teaching period based on the first attention evaluation index and / or the second attention evaluation index of the target student in the teaching period, including: The first evaluation module is configured to obtain a number of times and corresponding time durations when the first attention evaluation index of the target student is lower than the first attention evaluation index, and / or the second evaluation module is configured to obtain a number of times and corresponding time durations when the second attention evaluation index is lower than the second attention evaluation index. The attention evaluation module is configured to calculate the attention evaluation result of the target student in the teaching period based on the number of times and the time durations.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the method of any one of claims 1 to 6 when executed by the processor.

Images