A learning attention state evaluation method based on a multi-dimensional feature fusion network

Abstract

The application aims at the problem that learners are difficult to evaluate their attention state in real time, and discloses a learning attention state evaluation method based on a multi-dimensional feature fusion network.The method comprises the following steps: 1) acquiring video resources of learners collected by a binocular imaging device (a short-wave infrared camera and a laser radar scanner) on a desk, and dividing the video resources into multiple frames of images; a hand wearable device acquires blood oxygen saturation and heart rate signals of the learners; 2) positioning a face region and facial feature points of the learners on the SWIR image of the learners; and segmenting a head region 3D point cloud set; 3) inputting the face region SWIR image and the head 3D point cloud set into corresponding feature extraction networks to obtain feature topology graphs; after fusion through a self-attention weighting module, inputting a Cauchy label distribution regression module to obtain a head posture angle of the learners; extracting blood oxygen saturation and heart rate change features to judge a fatigue grade of the learners; 4) comprehensively evaluating an attention state according to the head posture angle of the learners, facial feature point changes and the fatigue degree, and reminding the learners if they are not concentrated; and 5) statistically analyzing the attention state in a learning process, and feeding back a statistical analysis report.The application helps the learners to improve concentration and cultivate good learning habits by comprehensively evaluating and statistically feeding back the attention state of the learners.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, the field of behavior analysis, and in particular to a learning attention state evaluation method based on a multi-dimensional feature fusion network. BACKGROUND

[0002] With the increasing richness and comprehensiveness of learning resource acquisition channels, improving personal ability through self-study is becoming the future trend of learning development. However, in an unsupervised environment, especially in a home environment, the attention of many learners is easily distracted, and the learning efficiency is low. In recent years, artificial intelligence technology has been widely used in many fields due to its convenience and efficiency. In a self-study environment, learners often cannot realize and correct their behavior in time, so artificial intelligence technology, especially posture recognition technology, can be used to supervise the learning of learners in real time, judge whether the attention of learners is concentrated, and thus better help learners develop good learning habits.

[0003] Head posture is an important form of reflecting the attention of learners. By analyzing the changes in the head posture angle of learners during the learning process, and combining the changes in facial feature points, it can effectively determine whether the attention of learners is concentrated. If the head deflection angle of learners is outside the desk or screen during the learning process, or if yawning or even closing eyes frequently occurs, the system can detect and remind learners to concentrate in time. In addition, the blood oxygen saturation and heart rate change features can well reflect the fatigue degree of learners, and the fatigue degree will directly affect the concentration degree of attention. However, head posture estimation also faces some challenges at present:

[0004] (1) During the learning process of learners, hand occlusion, hairstyle occlusion, clothing occlusion, and other head occlusion problems often occur. In addition, indoor scenes often have insufficient lighting problems. These will cause the quality of the collected images to be poor and unable to obtain complete information. Therefore, an image acquisition device that can collect head posture information in multiple dimensions and is not affected by changes in lighting is needed.

[0005] (2) Nowadays, the distribution of training samples in the data set used to train the head posture angle is extremely unbalanced. There are not enough large posture training samples, and many pictures in the training set have the problem of incorrect labeling of the head posture angle. This leads to the inability to train network parameters with strong robustness.

[0006] (3) At present, most methods for head posture estimation only use RGB images or head 3D point cloud data to regress the head posture angle. The angle accuracy of the method based on two-dimensional images is difficult to improve, and the method based on three-dimensional point cloud often has too large a calculation amount. Therefore, a lightweight head posture estimation method that integrates two-dimensional and three-dimensional features is needed. SUMMARY

[0007] In view of the improvement needs of the prior art, the application adopts a binocular imaging device with a short-wave infrared camera and a laser radar scanner, and provides a learning attention state evaluation method based on a multi-dimensional feature fusion network, which can combine the fatigue degree of a learner to supervise in real time whether the learner has attention distraction behaviors such as head deflection angle located in an area outside a desk or a screen, closed eyes, etc., timely remind the learner to concentrate attention, and generate an attention concentration degree report during learning to help the learner develop good learning habits.

[0008] The technical scheme adopted by the application to solve the technical problems is: a learning attention state evaluation method based on a multi-dimensional feature fusion network, comprising the following steps:

[0009] Obtain the video resource of the learner collected by the binocular imaging device on the office table, the binocular imaging device is composed of a short-wave infrared camera and a laser radar scanner, and is divided into multiple images; the hand wearable device obtains the blood oxygen saturation and heart rate signals of the learner;

[0010] Locate the face region and facial feature points of the learner on the SWIR image. Segment the head region 3D point cloud set;

[0011] Input the face region SWIR image and the head 3D point cloud set into the corresponding feature extraction network to obtain the feature topology graph, and after fusion through the self-attention weighting module, input the Cauchy label distribution regression module to obtain the head posture angle of the learner. Extract the blood oxygen saturation and heart rate change features to judge the fatigue level of the learner;

[0012] According to the head posture angle, facial feature point position change and fatigue degree of the learner, the attention state is comprehensively evaluated, and the learner is reminded if the attention is not concentrated;

[0013] Statistical analysis report is fed back.

[0014] According to the above scheme, the face region and facial feature point positioning module process is as follows:

[0015] Step 1.1.1: Adjust the size of each frame of SWIR image of the interactive object to 624x624 pixels, input it into the lightweight Mask R-CNN network pre-trained on the face data set, and obtain the face region ;

[0016] Step 1.2.1: Input the cropped face region SWIR image into the global rough feature extraction network RG-Net, and the network structure of RG-Net can be represented as where conv1 represents a convolutional layer, res represents a residual connection layer, glDSC represents a global channel separable convolution, fc represents a fully connected layer, and regression outputs a final global coarse landmark coordinate vector ;

[0017] Step 1.2.2: Take the output feature map of the res1 layer in the RG-Net network Crop a feature map of size 128 128 centered at the corresponding coarse landmark to obtain a first-level refined feature map . Input the first-level refined feature map into the local refinement network FL-Net to extract a feature vector , and a first-level refined facial landmark coordinate vector ;

[0018] Step 1.2.3: Take the output feature map of the conv1 layer in the RG-Net network Crop a feature map of size 128 128 centered at the corresponding coarse landmark to obtain a second-level refined feature map . Input the second-level refined feature map into the local refinement network FL-Net to extract a feature vector , and a second-level refined facial landmark coordinate vector , which is a sparse facial landmark coordinate vector.

[0019] The head pose two-dimensional feature extraction model comprises a channel separable convolution module, a pixel space Transformer module, a fusion feature topology construction module, and an adaptive graph convolution module. The channel separable convolution module is used to extract local features of a pixel space of a preprocessed SWIR face region image. The pixel space Transformer extracts global feature relationships of the pixel space from the local feature map. The adaptive graph convolution module updates values of graph vertices to obtain a new-dimensional head pose fusion feature topology graph.

[0020] According to the above scheme, the channel separable convolution module processes as follows:

[0021] Step 2.1.1: Input a set of 328 328 pixel value size SWIR face region images into a double-branch channel separable convolution network to extract local features of the images;

[0022] Step 2.1.2: The first branch structure is , wherein the SC1 module structure is [SC, BN, RL], SC represents a channel separable convolution, and local features are extracted by point-by-point convolution for each channel. BN represents normalization processing for a batch of images in the batch input, and batch normalization is performed for C channels respectively. The calculation formula of batch normalization can be expressed as follows:

[0023]

[0024] wherein represents a group of elements corresponding to each channel, i.e., a group of pixel values, represents the average value of the group of pixel values, i.e., , is a very small positive number, which avoids the standard deviation, i.e., the denominator, being 0, , is a network training parameter, which scales and translates the final normalized result. The RL activation function replaces negative value elements with zero, so that the feature map element value is easier to converge. SC_MAX, i.e., patch_max based on SC1, obtains a head pose local feature map .

[0025] Step 2.1.3: The first branch structure is , wherein the SC2 module structure is [SC, BN, TH], and the TH activation function normalizes the element value domain to (-1, 1), so that the network is easier to converge. SC_AVE, i.e., local averaging based on SC2, obtains a head pose local feature map .

[0026] According to the above scheme, the pixel space Transformer module training process is as follows:

[0027] Step 2.2.1: Input the local feature map into the two-branch two-stage pixel space Transformer network to extract pixel space global features and generate a fusion feature map;

[0028] Step 2.2.2: Input into the first branch, and the first branch first-stage structure is , and the second-stage structure is . The pixel space Transformer layer is composed of three pixel space Transformer encoders in series, which extracts pixel space global feature relationships;

[0029] Step 2.2.3: The pixel space Transformer encoder stretches the output feature map of the separable convolution layer SC into a three-dimensional embedding vector in ;

[0030] Step 2.2.4: Assign embedding vectors to each image. Each element, i.e., pixel, has a positional code added, which can be expressed as the following expression.

[0031]

[0032] in , Embedded vector Updated to .Will Input multi-head self-attention mapping module;

[0033] Step 2.2.5: The multi-head self-attention mapping module contains 8 self-attention heads. Each channel obtains a self-mapping weight matrix based on the input. The input is multiplied by the weight matrix and then subjected to a nonlinear transformation to obtain the self-attention mapping for each channel. The multi-head self-attention mapping module ultimately outputs as follows: ;

[0034] Step 2.2.6: Output of the multi-head self-attention mapping module After passing through a residual normalization layer and a fully connected layer, the output of the pixel-space Transformer encoder is obtained. Three pixel-space Transformer encoders are cascaded to finally obtain the head pose fusion feature map. ;

[0035] Step 2.2.7: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation Enter the first Branch, number Branches and the first The branching structures are similar, but different feature maps are extracted based on local averaging. The first-stage structure is as follows: The second stage structure is Finally, the head pose fusion feature map is obtained. .

[0036] According to the above scheme, the fused feature topology graph construction module includes fused feature graph vertices. Construction of the topological connectivity matrix T. The element-wise dot product yields the total fused feature map. Through a fully connected layer Mapping to a low-dimensional fusion feature vector , This represents the number of images in a batch, and a fused feature topology map is constructed for each frame. The fused feature map vertices... The values ​​are the same for a single image. Fusion feature vector , the fusion feature topology graph shares a topology connection matrix with the 3D point cloud topology graph . Construct a fusion feature topology graph .

[0037] The head point cloud segmentation module process is as follows: according to the two-dimensional coordinate information provided by the face region detection frame , compare the dense point cloud set corresponding to each frame of point cloud graph , select the point cloud set in the range from the point cloud graph , get the head region dense point cloud set .

[0038] The head pose three-dimensional feature extraction model includes a facial feature point 3D point cloud topology graph construction module and an adaptive graph convolution module. The facial feature point 3D point cloud topology graph construction module includes a 3D point cloud graph vertex and a topology connection matrix T. The adaptive graph convolution module extracts the weight relationship between the vertices of the topology graph, updates the value of the graph vertex, and obtains a new dimension head pose 3D point cloud topology graph.

[0039] According to the above scheme, the process of the facial feature point 3D point cloud topology graph construction module is as follows:

[0040] Step 3.1.1: According to the two-dimensional coordinate information of the facial feature points , select the corresponding 3D point cloud coordinates from the point cloud set . The value of the 3D point cloud graph vertex is the 3D point cloud coordinate value of the facial key point ;

[0041] Step 3.1.2: Based on the KD-Tree algorithm, find the 5 closest vertices in the Euclidean space of each graph vertex , connect to construct a topology connection matrix , , the number of feature points, , the value is 1, which means that the graph vertices are connected, otherwise it is 0;

[0042] Step 3.1.3: Construct a 3D point cloud topology graph , wherein .

[0043] According to the above scheme, the training process of the adaptive graph convolution module is as follows:

[0044] Step 4.1.1: The network structure of the adaptive graph convolution module is an adaptive graph convolution layer, a batch normalization layer, an RL function activation layer, a 1D convolution layer, a batch normalization layer, an RL function activation layer, and a multiple input feature topology graphThe graph vertex value of The updated feature value of 192 dimensions ; ;

[0045] Step 4.1.2: Adaptive graph convolution layer selects feature map Each vertex And its neighborhood distance closest K vertices constitute a vertex pair, for each vertex pair ( ), M channels are constructed, each channel independently calculates the feature value, and the K vertex pair feature values are concatenated. After channel maximum pooling, the updated graph vertex is obtained. Where K is 6 and M is 192.

[0046] The blood oxygen saturation-ECG signal feature extraction module process is as follows:

[0047] Step 5.1.1: For blood oxygen saturation spo2, calculate the mean square error of the sampling value in a period

[0048] , where N is the number of samples, is the th sampling value, is the average value of the sampling in a period;

[0049] Step 5.1.2: For ECG signal, calculate the standard deviation of the interval between adjacent R waves of two consecutive heartbeats , the ratio of high frequency and ultra-low frequency energy spectrum density in adjacent R wave period . Indicates the interval between two adjacent peaks. According to the spectrum density of ultra-low frequency and high frequency in the interval , the ratio of the spectrum density of ultra-low frequency and high frequency is ;

[0050] Step 5.1.3: Comprehensive analysis The change of the ratio of the spectrum density of ultra-low frequency and high frequency is , If the ratio of the spectrum density of ultra-low frequency and high frequency is , it is judged as fatigue level 1, consciousness is clear, thinking is active. If the ratio of the spectrum density of ultra-low frequency and high frequency is 0.01, , ≥0.8, it is judged as fatigue level 3, consciousness is blurred, thinking cannot be concentrated.

[0051] The self-attention weighting module process is as follows:

[0052] Step 6.1.1: The self-attention weighting module includes self-attention layer, full connection layer, and softmax regression layer. The updated three-dimensional point cloud topology graph and two-dimensional fusion feature topology graph After inputting the self-attention layer, update to , ;

[0053] Step 6.1.2: The full connection layer maps the weighted fusion feature topology graph to a multi-dimensional feature vector, and finally calculates the weighted parameters of , and to obtain the final weighted fusion feature topology graph . and . . .

[0054] The Cauchy label distribution regression module process is as follows:

[0055] Step 7.1.1: Map the weighted fusion feature topology graph to a multi-dimensional feature vector through a full connection layer, and regress the angle of the head pose accurately, calculate the MAE with the real angle as the loss function ;

[0056] Step 7.1.2: For each training set image , convert the actual angle label to a Cauchy label distribution . At the same time, the module trains the network to generate three sets of parameters , and obtain the predicted Cauchy label probability distribution ;

[0057] Step 7.1.4: Calculate the spatial distance and the KL divergence between the predicted Cauchy label probability distribution and the actual Cauchy label distribution as the loss function , and the loss function is weighted to obtain the final loss function .

[0058] According to the above scheme, the optimal network parameters are obtained by training the loss function on the training set in advance, and the short-wave infrared image and 3d point cloud data of the learner are input into the pre-trained multi-dimensional feature fusion self-attention network, so that the real-time head pose angle Yaw, Pitch, Roll of the learner can be obtained. It is judged whether it is located in the attention concentration interval, and the learner's attention concentration degree is comprehensively judged in combination with the face feature point position condition and the learner's fatigue degree. If it is not concentrated, the learner is reminded.

[0059] Overall, compared with the prior art, the present application has the beneficial effects:

[0060] ​(1) The short-wave infrared video image and 3D point cloud data are collected respectively, head posture information is collected from multiple dimensions, and is not affected by illumination changes. The two-dimensional and three-dimensional information of the head posture is comprehensively considered, and more accurate head posture angles can be obtained.

[0061] (2) The multi-dimensional feature fusion self-attention network uses the spatial information of the face key points to construct a topological graph structure, and constructs a head posture topological graph according to the two-dimensional features and the three-dimensional features. The head posture two-dimensional feature extraction combines the convolution operation for extracting local information and the pixel space Transformer for extracting global information, so as to extract more comprehensive local-global two-dimensional fusion features. The Cauchy label distribution regression module fully considers the similarity between adjacent head postures, thereby solving the problem that there are not enough large posture training samples in the training set.

[0062] (3) In order to assist the head posture angle to represent the attention concentration degree of the learner, blood oxygen saturation and electrocardiogram signals are collected, and the change of the corresponding parameters extracted by comprehensive analysis can be used to qualitatively judge the fatigue degree of the learner. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a flowchart of a learning attention state evaluation method based on a multi-dimensional feature fusion network according to an embodiment of the present application

[0064] Figure 2 is a data acquisition schematic diagram in a home environment

[0065] Figure 3 is a multi-dimensional feature fusion network structure schematic diagram according to an embodiment of the present application DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0067] As shown in Figure 1 , the present application is a learning attention state evaluation method based on a multi-dimensional feature fusion network, comprising the following steps:

[0068] Step 1: Obtain the video resource of the learner collected by the binocular imaging device on the desk, the binocular imaging device is composed of a short-wave infrared camera and a laser radar scanner, and is divided into multiple images in time sequence. At the same time, the blood oxygen saturation and heart rate signals of the learner are obtained through the hand wearable device.

[0069] Step 2: Face region and facial feature point positioning is performed on the SWIR image of the learner, and the head region 3D point cloud set is segmented.

[0070] Step 3: The face region SWIR image and the head 3D point cloud set are input into the corresponding head posture two-dimensional and three-dimensional feature extraction network to obtain the feature topology graph. After fusion by the self-attention weighting module, the learner's head posture angle is obtained by inputting the Cauchy label distribution regression module. At the same time, the blood oxygen saturation and heart rate change features are extracted to judge the fatigue level of the learner.

[0071] According to the above scheme, the blood oxygen saturation-ECG signal feature extraction module process is: taking 5 minutes as a cycle, calculating the mean square deviation of the blood oxygen saturation spo2 sampling value in one cycle , where N is the sampling number, is the sampling value, is the sampling average value in one cycle. At the same time, wavelet transform is used to detect all wave peaks of the ECG signal time domain waveform in one cycle, and the interval of two adjacent wave peaks is calculated to obtain the standard deviation of the interval of two adjacent R waves of the continuous heartbeat signal . Fast Fourier transform is used to convert the time domain signal in the cycle to a frequency domain signal, analyze the frequency spectrum, and calculate the spectral density of the ultra-low frequency and high frequency , that is, the second parameter .

[0072] The changes of are comprehensively analyzed. If <0.005, , <0.2, it is judged as fatigue level 1, clear consciousness, and active thinking. If , it is judged as fatigue level 2, consciousness is relatively blurred, and thinking is relaxed. If , it is judged as fatigue level 3, consciousness is blurred, and thinking cannot be concentrated.

[0073] As shown in Figure 2 , the learner is learning at home, and in this scenario, the learner's face video sequence is collected by using a short-wave infrared camera and a laser radar scanner. The multiple frames of SWIR images and 3D point cloud images of the learner collected in this scenario provide important data sources for the head posture estimation module.

[0074] As shown in Figure 3As shown, in this embodiment, the multi-dimensional feature fusion self-attention network includes a face region and facial feature point positioning module, a head point cloud segmentation module, a head posture two-dimensional feature extraction module, a head posture three-dimensional feature extraction module, a self-attention weighting module, and a Cauchy label distribution regression module.

[0075] According to the above scheme, the face region and facial feature point positioning module is processed as follows:

[0076] Step 3.1.1: Adjust the size of each frame of SWIR image of the interactive object to 624x 624 pixels, and input it into the lightweight Mask R-CNN network pre-trained on the face dataset to obtain the face region ;

[0077] Step 3.1.2: According to the face region , the SWIR image of each frame of the interactive object is cropped, and a sparse facial feature point extraction network composed of a global rough feature point extraction network RG-Net and a cascaded local refinement network FL-Net is inputted;

[0078] Step 3.1.3: The cropped face region SWIR image is inputted into RG-Net, and the network structure of RG-Net can be represented as , wherein conv1 represents a convolutional layer, res represents a residual connection layer, glDSC represents a global channel separable convolution, fc represents a fully connected layer, and the final global rough feature point coordinate vector ;

[0079] Step 3.1.4: Take the output feature map of the res1 layer in the RG-Net network , and crop a feature map of size centered at the rough feature point to obtain a first-level refinement feature map . Input into the local refinement network FL-Net, which first reduces the multi-channel feature map to a two-dimensional vector through convolution, and then performs normalization and relu activation function nonlinear transformation, and finally the first-level feature vector

[0080] is regressed by the fully connected layer. ;

[0081] Step 3.1.5: Take the output feature map of the conv1 layer in the RG-Net network , and crop a feature map of size centered at the rough feature point to obtain a second-level refinement feature map , the secondary feature vector is obtained by inputting the local refinement network FL-Net , the secondary refined facial feature point coordinate vector is the final sparse facial feature point coordinate vector. The extraction process can be expressed as follows:

[0082]

[0083] wherein is the output of the global rough feature point extraction network RG-Net, l represents the number of layers, represents the local refinement network FL-Net cascade times, represents the output feature map of the global rough feature point extraction network RG-Net at the layer, represents feature reuse, that is, a size feature map centered on the rough feature point is constructed.

[0084] The head posture two-dimensional feature extraction model includes a channel separable convolution module, a pixel space Transformer module, a fusion feature topology construction module, and an adaptive graph convolution module. The channel separable convolution module converts the SWIR image into a multi-channel local feature map, which is used to extract the local features of the pixel space of the preprocessed SWIR face region image. The pixel space Transformer extracts the global feature relationship of the pixel space from the multi-channel local feature map to generate a pixel space fusion feature map. The fusion feature topology construction module includes the construction of the fusion feature map vertex and the topological connection matrix T. The adaptive graph convolution module extracts the weight relationship between each vertex pair of the topological graph, updates the value of the graph vertex, and obtains a new dimension of head posture fusion feature topology graph.

[0085] According to the above scheme, the process of the channel separable convolution module is as follows:

[0086] Step 3.2.1: The face region window is adjusted to , and the face region window size is adjusted to 328 328, a set of cropped SWIR face region images are input into the double-branch channel separable convolution network to extract the local features of the images;

[0087] Step 3.2.2: inputs the first branch, and the branch structure is , wherein the SC1 module structure is [SC, BN, RL], SC represents a channel separable convolution, which extracts local features by point-by-point convolution for each channel, BN represents normalization processing for a batch of images in the batch input, and the calculation formula of the batch normalization is , represents a set of elements corresponding to each channel, i.e., pixel values. The RL activation function replaces negative value elements with zero, so that the feature map element value is easier to converge. SC_MAX, i.e., Patch_Max based on SC1, obtains the head pose local feature map ;

[0088] Step 3.2.3: input the first branch, and the branch structure is , wherein the SC2 module structure is [SC, BN, TH], and the TH activation function normalizes the element value range to (-1, 1), so that the network is easier to converge. SC_AVE, i.e., Patch_Ave based on SC2, obtains the head pose local feature map . .

[0089] According to the above scheme, the pixel space Transformer module training process is as follows:

[0090] Step 3.3.1: input the local feature map , into the two-branch two-stage pixel space Transformer network to extract pixel space global features and generate a fusion feature map;

[0091] Step 3.3.2: input into the first branch and the second branch, the first-stage structure of the first branch is , and the second-stage structure is . The pixel space Transformer layer is composed of three pixel space Transformer encoders in series, which extracts pixel space global feature relationships;

[0092] Step 3.3.3: the pixel space Transformer encoder stretches the output feature map of the separable convolution layer SC into a three-dimensional embedding vector , wherein ;

[0093] Step 3.3.4: add position encoding to each element, i.e., pixel point, of the embedding vector corresponding to each image, wherein , Embedding vector Update to . Will Input multi-head self-attention mapping module;

[0094] Step 3.3.5: The multi-head self-attention mapping module contains 8-channel self-attention heads, and the self-attention mapping of each channel is The calculation expression is as follows:

[0095]

[0096] Where the self-mapping weight matrix of each channel is Based on the input, access the vector And the key-value vector , input And Dot product to get. The final output of the multi-head self-attention mapping module is ;

[0097] Step 3.3.6: The output of the multi-head self-attention mapping module After passing through the residual normalization layer and the full connection layer, and then through the normalization, the output of the pixel space Transformer encoder is obtained, and the calculation process can be summarized as follows:

[0098]

[0099] Each layer of The pixel points are normalized to standard normal distribution, Representing linear transformation. Three pixel space Transformer encoders are cascaded, and finally the head pose fusion feature map ;

[0100] Step 3.3.7: Input Into the first Branch, the first Branch and the first Branch structure is similar, but based on local average to extract different feature maps, the first stage structure is , the second stage structure is , and finally the head pose fusion feature map .

[0101] According to the above scheme, the fusion feature topology graph construction module includes fusion feature graph vertices And the construction of the topology connection matrix T. Multiply the elements of To get the total fusion feature map , map To a low-dimensional fusion feature vector through a full connection layer , represents the number of batches of images, respectively, each frame of image is constructed into a fusion feature topology graph. The value of the vertex of the fusion feature graph is the same as the single image fusion feature vector , the fusion feature topology graph shares the topology connection matrix T with the 3D point cloud topology graph. The fusion feature topology graph is constructed .

[0102] The head point cloud segmentation module process is as follows: according to the two-dimensional coordinate information provided by the face region detection frame , compare the dense point cloud set corresponding to each frame of point cloud graph , select the point cloud set in the range from the point cloud graph, and obtain the head region dense point cloud set . .

[0103] The head posture three-dimensional feature extraction model includes a face feature point 3D point cloud topology graph construction module and an adaptive graph convolution module. The face feature point 3D point cloud topology graph construction module includes the construction of 3D point cloud graph vertex and topology connection matrix T. The adaptive graph convolution module extracts the weight relationship between each vertex pair of the topology graph, updates the value of the graph vertex, and obtains a new dimension of head posture 3D point cloud topology graph.

[0104] According to the above scheme, the process of the face feature point 3D point cloud topology graph construction module is as follows:

[0105] Step 3.4.1: The 3D point cloud topology graph construction module includes the construction of 3D point cloud graph vertex and topology connection matrix T. According to the two-dimensional coordinate information of the face feature point , select the corresponding 3D point cloud coordinates from the point cloud set . The value of the 3D point cloud graph vertex is the 3D point cloud coordinate value of the face key point ;

[0106] Step 3.4.2: Based on the KD-Tree algorithm, find the 5 nearest vertices of each graph vertex in the Euclidean space, and connect them to construct the topology connection matrix , is the number of feature points, and the value of 1 represents that the graph vertices are connected, otherwise 0;

[0107] Step 3.1.3: Construct the 3D point cloud topology graph , wherein .

[0108] According to the above scheme, the adaptive graph convolution module training process is as follows:

[0109] Step 3.5.1: The network structure of the adaptive graph convolution module is an adaptive graph convolution layer, a batch normalization layer, an RL function activation layer, a 1-dimensional convolution layer, a batch normalization layer, an RL function activation layer, an adaptive graph convolution layer, which extracts the weight relationship between each vertex pair of the topological graph, and updates the value of the graph vertex. The 1-dimensional convolution layer further extracts the relationship between the sequences, and the batch normalization operation and the RL activation function make the network more easily convergent. Finally, the feature values of the multiple input topological graphs are updated to 192-dimensional feature values . ;

[0110] Step 3.5.2: The adaptive graph convolution layer selects the feature map of each vertex and the K vertices closest to the neighborhood of the vertex to form a vertex pair. For each vertex pair , M channels are constructed, and the feature value is calculated independently for each channel. The calculation process is as follows:

[0111]

[0112] where represents the concatenation of A and B vectors, represents the dot product, represents the MLP layer, represents the RL function nonlinear activation, which converts negative elements to 0;

[0113] Step 3.5.3: The feature value of each vertex pair is updated to an M-dimensional feature vector . The feature values of K vertex pairs are concatenated , and the updated graph vertex is obtained through channel maximum pooling. Where K is 6 and M is 192.

[0114] According to the above scheme, the self-attention weighting module process is as follows:

[0115] Step 3.6.1: The self-attention weighting module includes a self-attention layer, a fully connected layer, and a softmax regression layer. The three-dimensional point cloud topological graph and the two-dimensional fusion feature topological graph are input into the self-attention layer and updated to , . The calculation process is as follows:

[0116]

[0117] Step 3.6.2: The calculation process of the full connection layer can be expressed as follows, where represents a linear transformation, and is mapped to a vector of dimension

[0118] Finally, the weighted parameters of and are calculated using the function, and the calculation process can be represented as:

[0119] According to the above scheme, the Cauchy label distribution regression module process is as follows:

[0120] Step 3.6.3: The weighted fusion feature topology graph is mapped to a multi-dimensional feature vector through the full connection layer, and the angle of the accurate head pose is regressed. The MAE with the real angle is calculated as the loss function .

[0121] Step 3.6.4: Considering that for the same head pose angle, the head pose similarity along the three directions of Yaw, Pitch, and Roll is different, the deflection angles of the three directions are respectively divided into 46, 100, and 62 segments, i.e., the angle is encoded into the corresponding label set .

[0122] Step 3.6.5: For each training set image , the actual angle label is converted into a Cauchy label distribution . Where , the element value can be represented as

[0123] where represents the th label, represents the encoded value corresponding to the real yaw angle, and the label standard deviation is set to 4. , the element value can be represented as

[0124] where represents the th label, represents the encoded value corresponding to the real pitch angle, and the label standard deviation is set to 10. , the element value can be represented as

[0125] ​​ wherein represents the first label, represents the encoding value corresponding to the real roll angle, and the label standard deviation is set to 6.

[0126] Meanwhile, the module trains the network to generate three sets of parameters , respectively corresponding to the three sets of label sets . The predicted Cauchy label probability distribution is obtained.

[0127] The spatial distance and the KL divergence between the predicted Cauchy label probability distribution and the actual Cauchy label distribution are calculated. The spatial distance and the KL divergence are weighted as the loss function .

[0128] Step 3.6.6: Calculate the spatial distance and the KL divergence

[0129]

[0130] The final loss function . According to the above scheme, the optimal network parameters are obtained by training the loss function on the training set in advance. The short-wave infrared image and 3D point cloud data of the learner are input into the pre-trained multi-dimensional feature fusion self-attention network, and the final head pose angles (Yaw, Pitch, Roll) are obtained.

[0131] Step 4: According to the head pose angles and facial feature point positions of the learner at different times, combined with the changes of blood oxygen saturation and electrocardiogram signals, the fatigue level is determined, and the attention state of the learner is comprehensively evaluated. At the same time, it is judged whether it is located in the attention dispersion interval. If it is located in the interval, the learner's attention is not concentrated at this time, otherwise the attention is concentrated.

[0132] Table 1 Comprehensive evaluation rule of learner attention state

[0133] Those skilled in the art will readily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A learning attention state evaluation method based on a multi-dimensional feature fusion network, characterized in that, The method comprises the steps of: acquiring a learner video resource collected by a binocular imaging device on an office desk, the binocular imaging device being composed of a short-wave infrared camera and a laser radar scanner and being divided into multiple images; a hand wearable device acquires a blood oxygen saturation and a heart rate signal of the learner; locating a face region and a facial feature point of the SWIR image of the learner, and segmenting a head region 3D point cloud set; inputting the face region SWIR image into a head posture two-dimensional feature extraction model to extract a fusion feature topology graph; inputting the head 3D point cloud set into a head posture three-dimensional feature extraction model to extract a 3D point cloud topology graph; after the two feature topology graphs are fused through a self-attention weighting module, the head posture angle of the learner is obtained through a Cauchy label distribution regression module; blood oxygen saturation and heart rate change features are extracted to judge the fatigue level of the learner; the head posture two-dimensional feature extraction model comprises a channel separable convolution module, a pixel space Transformer module, a fusion feature topology graph construction module and an adaptive graph convolution module; the head posture three-dimensional feature extraction model comprises a facial feature point 3D point cloud topology graph construction module and an adaptive graph convolution module; according to the head posture angle, the facial feature point position change and the fatigue level of the learner, the attention state is comprehensively evaluated, and the learner is reminded if the attention state is not concentrated; the attention state during the learning process is counted, and a statistical analysis report is fed back.

2. The method of claim 1, wherein, The process of locating the face region and the facial feature point of the SWIR image of the learner is as follows: Step 1.1.1: adjust the size of each frame of SWIR image of the interactive object to 624*624 pixels, input into the pre-trained lightweight Mask R-CNN network, and obtain the face region; Step 1.2.1: input the cropped face region SWIR image into the global rough landmark extraction network RG-Net, the structure of which is wherein res represents a residual connection layer, glDSC represents a global channel separable convolution, fc represents a fully connected layer, and the final global rough landmark coordinate vector is regressed ; Step 1.2.2: Take the output feature map of the res1 layer of RG-Net , crop the feature map with the size of centered on the rough feature point , input the local refinement network FL-Net to extract the feature vector , and the first-level refined face feature point coordinate vector is ; Step 1.2.3: Take the output feature map of the conv1 layer of RG-Net , crop a feature map with size centered at the coarse feature point , where is the coordinate of the coarse feature point, are the height and width of the feature map respectively, input the local refinement network FL-Net to extract the feature vector , , which is the sparse facial feature point coordinate vector. 3.The learning attention state evaluation method based on the multi-dimensional feature fusion network of claim 1, wherein, The training process of the channel separable convolution module is as follows: Step 2.1.1: inputting the SWIR face region image into a dual-branch channel separable convolutional network to extract local features of the two-dimensional image, where N is the number of image batches, is the image resolution, and C is the number of channels. Step 2.1.2: Channel separable convolution module The branch structure is Wherein the SC1 module structure is [SC, BN, RL], SC represents channel separable convolution, point-by-point convolution is performed on each channel to extract local features, BN represents normalization processing on the batch input image, RL is an activation function that replaces negative elements with zero, SC_MAX is local maximization on the basis of SC1 to obtain a head pose local feature map Wherein is the feature map resolution, is the feature channel number; Step 2.1.3: Channel separable convolution module The branch structure is , wherein the SC2 module structure is [SC, BN, TH], the TH activation function normalizes the elements to (-1, 1), and SC_AVE is local averaging on the basis of SC2, and finally the head pose local feature map is obtained . 4.The learning attention state evaluation method based on the multi-dimensional feature fusion network of claim 3, wherein, The training process of the pixel space Transformer module is as follows: Step 2.2.1: The input pixel space Transformer module is Input pixel space Transformer module Branch, first stage Branch first stage is Wherein is the local maximum pooling of the feature map, and the second stage is The pixel space Transformer layer is composed of three pixel space Transformer encoder cascades, which extract the global feature relationship of the pixel space, and the pixel space Transformer encoder is composed of a pixel space Transformer module and a pixel space Transformer module is a local maximum operation after a channel separable convolution; Step 2.2.2: The pixel space Transformer encoder encodes the output feature map of the SC layer Stretching into a three-dimensional vector And add position encoding , will Input multi-head self-attention mapping module, finally output , n is the number of self-control heads; By inputting and self-mapping weight matrix point multiplication and nonlinear transformation, After the residual normalization layer and the fully connected layer, the output of the Transformer encoder is obtained, and the three pixel space Transformer encoders are cascaded to finally obtain the head pose fusion feature map , wherein is the feature map resolution; Step 2.2.3: the head pose fusion feature map is obtained by performing local average pooling on the local maximum feature map obtained in step 2.2.2 Input pixel space Transformer module Branch, the first Branch and the second Branch structure differ in that the local feature processing operation adopts local average pooling after channel separable convolution instead of local maximum pooling after channel separable convolution , and the pooling mode adopts local average pooling instead of local maximum pooling, finally obtaining the head pose fusion feature map . 5.The learning attention state evaluation method based on the multi-dimensional feature fusion network of claim 4, wherein, The fusion feature topology graph construction module includes a fusion feature graph vertex and the construction of a topology connection matrix T, wherein is the vertex number, is the dimension of the fusion feature vector, and the element of is mapped to a low-dimensional fusion feature vector through a full connection layer after point multiplication , wherein is the number of image batches, is the number of channels of the low-dimensional fusion feature vector, and the value of the head pose fusion feature graph vertex is a single image fusion feature vector is the first element, and the head pose fusion feature topology graph shares a topology connection matrix with a 3D point cloud topology graph . 6.The learning attention state evaluation method based on the multi-dimensional feature fusion network of claim 1, wherein, The process of the facial feature point 3D point cloud topology graph construction module is as follows: Step 3.1.1: The 3D point cloud topology graph construction module includes 3D point cloud graph vertices and the construction of the topology connection matrix T, wherein is the vertex number, is the dimension of the 3D point cloud coordinates, according to the two-dimensional coordinate information of the facial feature points , the corresponding 3D point cloud coordinates are selected from the point cloud set , and the value of the 3D point cloud graph vertex is the 3D point cloud coordinate value of the facial key point. Step 3.1.2: Finding the nearest 5 vertices of each vertex based on KD-Tree algorithm Connect the nearest 5 vertices in Euclidean space to construct the topological connection matrix , The number of feature points, The value of 1 represents the connection of the graph vertices, and vice versa. 7.The learning attention state evaluation method based on the multi-dimensional feature fusion network of claim 1, wherein, The process of the adaptive graph convolution module is as follows: Step 4.1.1: Select feature map Each vertex And its neighborhood distance closest K vertex Make vertex pair, update feature value, n is vertex serial number; Step 4.1.2: For each vertex pair ( Construct M channels, and calculate the eigenvalues ​​of each channel independently. , This represents a concatenation of vectors A and B. Represents the dot product. Represents the MLP layer. Representing the non-linear activation of the RL function, negative elements are converted to 0. The eigenvalues ​​of all vertex pairs are concatenated and then subjected to channel max pooling to obtain the updated M-dimensional graph vertices. . 8.The learning attention state evaluation method based on the multi-dimensional feature fusion network of claim 1, wherein, The extracted blood oxygen saturation and heart rate variation features are blood oxygen saturation-ECG signal feature extraction modules, including calculating the mean square deviation of blood oxygen saturation spo2 sampling points in a cycle and the standard deviation of the interval between adjacent R waves of the heartbeat signal , the ratio of high frequency to ultra-low frequency energy spectral density in adjacent R wave cycles , wherein is the ultra-low frequency energy spectral density, is the high frequency energy density.

Images