Learning state intervention method and system for AI study room
Through intelligent composite cameras and head posture extractors, real-time monitoring and intervention of students' learning status is solved, and the problem of inaccurate learning status monitoring in the existing technology is improved, and self-study efficiency and educational quality are improved.
Patent Information
- Application Number
- CN202411036059.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
AI Technical Summary
The existing learning status intervention methods cannot accurately monitor students' learning status, and lack an intervention mechanism for learning status, which affects the self-study effect.
The intelligent composite camera is used to collect self-study video data of students' complete head posture, combine infrared cameras and depth cameras, calculate the face loss function through the head posture extractor and feature fusion diagram, adjust the camera angle in real time, monitor the students' learning status, and Implement intervention based on learning status.
It improves the monitoring accuracy and intervention effect of learning status, enhances students' self-study efficiency and education quality, reduces false alarms, and realizes personalized rest plan and study habit optimization.
Smart Images

Figure CN118968555A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a learning state intervention method and system for an AI study room. Background Art
[0002] In the field of modern education, especially in the context of intelligent education, improving students' self-study efficiency has become an important topic. However, existing learning status intervention methods mainly rely on traditional manual intervention or intervention using simple behavior recognition systems, but these methods often cannot fully and accurately monitor students' learning status and lack feedback mechanisms, resulting in the inability to intervene in students' adjustment of their learning status, which in turn affects the effectiveness of self-study.
[0003] At present, existing researchers use head posture to calculate concentration, so as to monitor and analyze students' self-study status in real time. However, on the one hand, traditional head posture is usually based on two-dimensional image processing and lacks depth information, which makes it difficult to accurately capture students' three-dimensional head posture in complex learning environments; on the other hand, traditional head posture acquisition methods often perform poorly when dealing with problems such as lighting changes and occlusion, further limiting their application scope and accuracy; in addition, existing research only focuses on studying students' learning status, but does not consider intervening in students' learning status so that students can adjust their learning status independently. Summary of the invention
[0004] The present invention provides a learning status intervention method and system for an AI study room, which are used to solve the problem that traditional methods for monitoring learning status are inaccurate and cannot intervene in students' learning status, thereby improving students' self-study efficiency and education quality.
[0005] On the one hand, the present invention provides a method for intervening in the learning state of an AI study room, the method comprising:
[0006] Using a composite camera to collect learning material area data and self-study video data including the student's complete head posture in real time; wherein the composite camera includes an integrated infrared camera and a depth camera; the self-study video data includes infrared visual data and depth space data;
[0007] Preprocessing the self-study video data to obtain head area data and student face data;
[0008] Inputting the head region data into a head posture extractor to obtain head posture angle data;
[0009] The student's learning status is monitored according to the student's facial data, the learning material area data and the head posture angle data, and corresponding intervention is implemented based on the learning status.
[0010] Furthermore, a composite camera is used to collect self-study video data including the complete head posture of the students in real time, including:
[0011] Inputting the collected current frame image into a feature extraction network including a plurality of blocks connected in series, and correspondingly obtaining a plurality of feature maps with different resolutions;
[0012] Fusing a plurality of the feature maps to obtain a feature fusion map;
[0013] Calculate the face loss function based on the feature fusion graph;
[0014] The acquisition angle of the composite camera is adjusted to minimize the face loss function, and the current acquisition angle is used as the camera acquisition angle of the next frame.
[0015] Further, calculating the face loss function based on the feature fusion graph includes:
[0016] Based on the feature fusion map, obtain the coordinates of the center point of the face, the center point offset, and the height and width of the face on the feature fusion map;
[0017] The face center point classification loss, the center point offset loss, and the face size loss are calculated respectively to obtain the face loss function.
[0018] Further, the preprocessing includes:
[0019] Performing data synthesis on the infrared visual data to obtain first visual data;
[0020] Performing shadow removal on the depth space data to obtain first space data;
[0021] Matching the resolution of the first visual data and the first spatial data to obtain second visual data and second spatial data accordingly;
[0022] Converting pixel coordinates of the second spatial data into coordinates in a world coordinate system to obtain three-dimensional spatial information, and mapping the three-dimensional spatial information to the second visual data to obtain preliminary fusion data;
[0023] The face region is segmented based on the preliminary fusion data to obtain the head region data and the student face data.
[0024] Furthermore, the construction of the head posture extractor includes:
[0025] Construct a Token, create a regional Token according to the head region data, and connect the visual Token in the Vit model with the regional Token through Concat;
[0026] Construct a Transformer encoder by stacking multiple identical Blocks; wherein a Block includes a multi-head self-attention module and a multi-layer perception module, and the multi-head self-attention module and the multi-layer perception module are connected through layer normalization and jump connection;
[0027] A Token-based head posture extractor is constructed, and multiple groups of regional Tokens corresponding to the outputs of multiple Blocks are converted into multiple rotation matrices for training and prediction to obtain the head posture extractor.
[0028] Furthermore, creating a region Token according to the head region data includes: dividing the head region data into a plurality of basic regions, and constructing the same number of d-dimensional learnable vectors to represent the plurality of basic regions.
[0029] Further, monitoring the student's learning status according to the student's face data, the learning material area data and the head posture angle data includes:
[0030] Based on the student face data and the learning material area data, obtain the distance between the student face and the learning material, and perform a first analysis;
[0031] Based on the head posture angle data and the learning material area data, the probability of the student's head attention being on the learning material area is obtained, and a second analysis is performed;
[0032] The student's learning status is obtained based on the first analysis results and the second analysis results.
[0033] Furthermore, the probability of obtaining the student's head attention on the learning material area includes:
[0034] Use Gaussian probability to determine the probability that the student's head attention is on the learning material area:
[0035]
[0036] Among them, (i, j) represents the coordinates of the center point of the AI study room in the composite camera coordinate system; (x, y) represents the focus of the student's attention, which is the coordinates of the student's head facing a certain projection point on the composite camera plane; w and h represent the width and height of the effective field of view projected on the composite camera plane with the student's head as the center; σ w , σ h They represent the correlation coefficients of w and h, respectively, and are used to control the distribution of Gaussian probability.
[0037] Further, implementing corresponding intervention based on the learning state includes:
[0038] A prompt device is arranged in the AI study room to issue different prompts according to the monitored learning status of the students; wherein the prompt device includes a learning reminder unit, a rest reminder unit and an emergency identification unit.
[0039] On the other hand, the present invention also provides a learning status intervention system for an AI study room, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the above methods.
[0040] In general, the present invention provides a method and system for intervening in the learning state of an AI study room. The technical solution conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0041] (1) Use an intelligent composite camera to collect self-study video data including the complete head posture of students. The composite camera includes an integrated infrared camera and a depth camera. The depth camera is used to capture deep spatial data, which adds depth information. It can not only obtain three-dimensional data, but also accurately capture the stereoscopic head posture of students in a complex learning environment; it is not affected by the environment such as light intensity and object occlusion, and can further retain more head detail information when facial features are blocked, so as to make the monitored learning status more accurate. In addition, corresponding intervention is implemented based on the learning status, thereby improving students' self-study efficiency and education quality.
[0042] (2) By adjusting the acquisition angle of the composite camera in real time through the face loss function, more effective and higher-quality infrared visual data and depth space data can be obtained. It also improves the accuracy of the head posture angle data to a certain extent, thereby improving the accuracy of the monitored learning state.
[0043] (3) The original visual token of the ViT model is used to create a regional token based on the head area data, and the visual token and the regional token are concatenated through Concat to obtain a higher-dimensional token vector, which greatly improves the accuracy and output efficiency of the head posture angle data.
[0044] (4) The learning status of students is monitored based on their facial data, learning material area data, and head posture angle data. At the same time, Gaussian probability is further used to determine the probability that the student's head attention is on the learning material area. This can greatly reduce false alarms, improve the accuracy of monitoring the learning status, and prevent misdetection.
[0045] (5) By adding a reminder module associated with the learning status, not only can appropriate reminders be given according to the student's learning status, but also the student's learning time, concentration score, and the frequency of changes in head posture can be comprehensively considered to intelligently customize a personalized rest plan for each student. In addition, the student's rest and study time can be recorded and saved, and the rest interval algorithm can be continuously optimized to make it more in line with the student's personal needs, helping students establish scientific learning and rest habits and achieve a balance between long-term learning effects and health status. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 It is a schematic diagram of the principle of a method for intervening in the learning state of an AI study room provided by the present invention;
[0048] Figure 2 It is a schematic diagram of an AI study room scene of a learning state intervention method of an AI study room provided by the present invention;
[0049] Figure 3 It is a schematic diagram of constructing a head posture extractor of a learning state intervention method of an AI study room provided by the present invention;
[0050] Figure 4 It is a model network diagram of a head posture extractor of a learning state intervention method of an AI study room provided by the present invention;
[0051] Figure 5 It is a schematic diagram of a learning status intervention method for an AI study room provided by the present invention for intervening according to the learning status. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings and embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0053] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method, step or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such method, step or system. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the method, step or system including the element.
[0054] The present invention provides a learning status intervention method and system for an AI study room, which aims to use a composite camera to collect relevant data, input the data into a pre-built head posture extractor, evaluate the students' learning status by analyzing their head posture, and implement corresponding intervention based on the learning status, thereby improving learning efficiency.
[0055] On the one hand, the present invention provides a method for intervening in the learning state of an AI study room, such as Figure 1 As shown, the method includes:
[0056] Step 100: Use a composite camera to collect learning material area data and self-study video data including the student's complete head posture in real time.
[0057] The composite camera includes an integrated infrared camera and a depth camera. That is, the intelligent composite camera DEPTH-IR integrated by the infrared camera and the depth camera in the AI study room is used to synchronously obtain the complete posture information of the students. The intelligent composite camera will automatically rotate the shooting angle according to the student's face and zoom in and out the lens for students at different distances, thereby ensuring accurate collection of relevant data in the study room scene.
[0058] Two different cameras, infrared and depth, are set up. The infrared camera is used to collect infrared visual information, and the depth camera is used to collect deep spatial information. The integrated intelligent composite camera can collect students' visual information and spatial information at the same time, thereby fusing the two types of information and improving the accuracy of data collection.
[0059] The learning material area data refers to the area covered by the learning content in the book or the location area covered by the learning content on the computer or tablet.
[0060] It should be noted that the AI study room can be a study room for one person or multiple people. When the AI study room is for one person, the composite camera includes a set of infrared cameras and depth cameras, and the collected data is the relevant data of the current student; when the AI study room is for multiple people, the composite camera includes multiple sets of one-to-one corresponding infrared cameras and depth cameras, correspondingly collecting relevant data of multiple students.
[0061] like Figure 2As shown, the composite camera is preferably set on the front wall of the AI study room, or it can be set on the desktop of the AI study room, so as to collect the self-study video data of the students' complete head posture. Among them, the self-study video data includes infrared visual data and deep space data.
[0062] As a preferred embodiment, a composite camera is used to collect self-study video data including the complete head posture of the student in real time, including:
[0063] Step 101: input the captured current frame image into a feature extraction network including a plurality of blocks connected in series, and obtain a plurality of feature maps with different resolutions accordingly.
[0064] It should be noted that the feature extraction network includes multiple blocks, and each block outputs feature maps of different resolutions. For example, the feature extraction network consists of six blocks connected in series, namely: Block_1, Block_2, Block_3, Block_4, Block_5, Block_6, then the resolution of each block output is 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64 of the original image feature map, respectively represented as: P1, P2, P3, P4, P5, P6.
[0065] Step 102: fuse multiple feature maps to obtain a feature fusion map. For example, feature fusion pyramid structure can be used to fuse feature maps P1, P2, P3, P4, P5, and P6 to obtain feature fusion map P.
[0066] Step 103: Calculate the face loss function based on the feature fusion graph.
[0067] As an embodiment, firstly, according to the feature fusion map P, the coordinates of the face center point, the center point offset, and the face width and height on the feature fusion map P can be directly obtained, and then the face center point classification loss, the center point offset loss and the face size loss are calculated respectively, and finally the face loss function is obtained.
[0068] Cross entropy can be used as the face center point classification loss, specifically:
[0069]
[0070] Among them, p∈[0,1] represents the predicted value, y∈{0,1} represents the true value label, and y=1 indicates that the sample is the center point of the student's face.
[0071] Since the number of faces in the data image is much smaller than the pixels, the focus loss is used to make the face detection neural network more focused on training positive samples. The face center point classification loss after adding the focus loss can also be used, specifically:
[0072]
[0073] Due to the nature of the convolution layer, each point also contains the features of the surrounding points, and the predicted value of the pixel closer to the center point should be closer to 1. Therefore, the center point heat map loss based on the Gaussian function can also be added to the loss function. The Gaussian heat map formula is: (i, j) represents the horizontal and vertical coordinates of each pixel in the image, M ij ∈[0, 1] represents the value of each point in the Gaussian heat map, (x, y) represents the coordinates of the center point of the student's face in the image, w and h represent the width and height of the student's face respectively, and Represent the adaptive variance related to face width and face height respectively.
[0074] Since the Gaussian heatmaps of different faces may overlap, a function can be used to determine the value of each point on the entire Gaussian heatmap. The Gaussian heatmap increases the probability of predicting pixels close to the center as positive samples and suppresses the loss center of negative samples close to the center. The center point classification loss L can be calculated using the focal loss and Gaussian heatmap loss center , specifically:
[0075]
[0076] Among them, p ij Represents the predicted value of each pixel in the image, M ij represents the value of each point in the Gaussian heat map, (x, y) represents the coordinates of the center point of the student's face in the image, w and h represent the width and height of the student's face respectively, a represents the importance attached to the high-value area in the Gaussian heat map; β represents the penalty for low prediction values.
[0077] Due to the error caused by enlarging the bounding box back to the original image, it is necessary to add center point offset prediction to make up for the lack of fine-grained information.
[0078] The SmoothL1 loss function can be used to calculate the center point offset loss L offset and face size loss L scale .
[0079] As an embodiment, the hyperparameter α is set to 6, and the offset within a plurality of pixels is mapped to the interval [0, 1]. Then, the center point offset loss can be:
[0080]
[0081] in, is the predicted value of the center point offset, τ is the true value label, and the negative sample is 0.
[0082] As an embodiment, the face size loss may be:
[0083]
[0084] in, is the predicted value of the center point offset, s0 is the true value label. Since the face detection neural network completes the prediction on the feature map sampled n times, the size value needs to be divided by n.
[0085] The face loss function can be calculated by weighting the face center classification loss, center offset loss, and face size loss to obtain the total loss function of the face detection neural network. Specifically:
[0086]
[0087] in, and They represent the weights of face center classification loss, center offset loss, and face size loss. The weights of the three losses are generally determined according to their importance. For example, face center classification loss is the main task of face detection neural network, so the weight can be Set to 1.00; the center point offset loss has a relatively small effect on the result, so the weight can be Set to 0.10; the range of facial size changes of students during self-study is small, so the weight can be Set to 0.01.
[0088] Step 104: Adjust the acquisition angle of the composite camera to minimize the face loss function, and use the current acquisition angle as the camera acquisition angle for the next frame. That is, the composite camera will compare the loss of images at different angles and then select the best acquisition angle. For example, according to the feature fusion graph P = (p1, p2, ..., p n ) to compare the face loss function, and take the minimum value of the face loss function min(p1, p2, ..., p n ) is taken as the optimal acquisition angle.
[0089] Step 200: Preprocess the self-study video data to obtain head area data and student face data.
[0090] Specifically, the self-study video data is converted into image data for each frame, and the infrared image and depth image are cropped to the same size. The two types of cropped images are preprocessed, the infrared image and the depth image are fused, and the background removal and facial area segmentation are performed, so as to effectively extract and separate the accurate student facial area from the infrared image and the depth image, and then obtain the head area data and the student face data, providing high-quality image data for subsequent head posture analysis and learning status monitoring.
[0091] The student face data includes face position data or face size data, that is, the position of any fixed point on the student face in three-dimensional space coordinates or the size of the face frame, for example, it can be the middle point of the two eyes, the center point of the eyebrows, the tip of the nose or the center point of the forehead, etc.; it can also be the center point on the student face plane.
[0092] As an embodiment of the present invention, the preprocessing includes:
[0093] Step 201: synthesize the infrared visual data to obtain first visual data; remove shadows from the depth space data to obtain first space data.
[0094] The first visual data may be obtained by performing principal component analysis on the infrared visual data to highlight important features in the image; its covariance matrix may be:
[0095]
[0096] Among them, C is the covariance matrix of infrared vision data, X is the standardized data matrix, N is the number of pixels, and X T is the transpose of X.
[0097] The first spatial data may be obtained by removing the depth shadow using a depth image enhancement method. In order to reduce the interference of the depth shadow on the head posture estimation, the process of removing the depth shadow may be:
[0098] I depth =I depth -min(I depth );
[0099] Among them, I depth Represents depth image data, min(I depth ) represents the minimum depth value in the depth image.
[0100] Step 202: performing resolution matching on the first visual data and the first spatial data to obtain second visual data and second spatial data accordingly.
[0101] In order to spatially align the infrared visual data and the depth space data so that the infrared radiation intensity and the depth value of each pixel can be correctly corresponded subsequently, the present invention upsamples the first spatial data and downsamples the first visual data to match the resolution of the depth data and the infrared data.
[0102] As an embodiment, the low-resolution first spatial data is expanded to the same resolution as the RGB data using a bilinear interpolation method, and then the first visual data is downsampled using a principal component analysis method.
[0103] The matching process is:
[0104]
[0105] Among them, I depth Represents the original depth image, that is, the first spatial data, is the upsampled depth image, that is, the second spatial data; N and M are the width and height of the original depth image, respectively, K is the bilinear interpolation kernel, and x′ and y′ are the pixel coordinates in the upsampled depth image.
[0106] Step 203: Convert the pixel coordinates of the second spatial data into coordinates in the world coordinate system to obtain three-dimensional spatial information, and map the three-dimensional spatial information to the second visual data to obtain preliminary fusion data.
[0107] It should be noted that the pixel coordinates of the second spatial data are converted into coordinates in the world coordinate system to obtain three-dimensional spatial information, more specifically:
[0108] First, the pixel coordinates of the second spatial data Convert to normalized coordinates in the camera coordinate system The conversion method is:
[0109]
[0110] Among them, K represents the depth camera intrinsic parameter.
[0111] Then, the normalized coordinates in the camera coordinate system are Convert to the coordinates in the camera coordinate system The conversion method is: Wherein, d represents the depth value of the pixel in the second spatial data.
[0112] Finally, the camera coordinates are transformed is the coordinate in the world coordinate system The conversion method is: Then we can get three-dimensional space information [X depth , Y depth , Zdepth ], and [X depth , Y depth , Z depth ] is the three-dimensional coordinate corresponding to the pixel in the second spatial data, where R and t represent the external parameters of the depth camera.
[0113] The three-dimensional space information is mapped to the second visual data to obtain preliminary fusion data. The mapping method can be:
[0114]
[0115] Among them, (X depth , Y depth , Z depth ) is the three-dimensional space coordinate, (u inf , v inf ) is the pixel coordinate in the second visual data, K inf is the internal parameter matrix of the infrared camera, Z inf is the depth value of the infrared camera.
[0116] Step 204: Perform face region segmentation based on the preliminary fusion data to obtain head region data and student face data.
[0117] The face region segmentation method can use the fine-tuned FocalNet-H model to perform facial region segmentation, output the segmentation results and optimize the model accuracy according to the loss function.
[0118] Among them, the optimized loss function in the FocalNet-H model can be:
[0119]
[0120] Among them, λ coord is the weight of the positioning loss, S is the size of the grid, B is the number of prediction boxes for each grid, is the indicator function of the object confidence, is the position and size of the prediction box, (b x , b y , b w , b h ) is the actual position and size of the frame.
[0121] Step 300: Input the head region data into the head posture extractor to obtain the head posture angle data, which is the three Euler angles of the head posture.
[0122] It should be noted that the head posture extractor is a Transformer head posture estimation method that uses the ResNet network for feature extraction and integrates regional clues. However, traditional head posture is usually based on two-dimensional image processing and lacks depth information, which makes it difficult to accurately capture the three-dimensional head posture of students in complex learning environments; and traditional head posture acquisition methods often perform poorly when dealing with problems such as lighting changes and occlusion, further limiting its application scope and accuracy.
[0123] Therefore, the present invention further proposes a head posture extractor to solve the problems of the head posture extractor in the prior art. Specifically, the construction of the head posture extractor is as follows: Figure 3 As shown, including:
[0124] Step 301: Construct a Token, create a regional Token based on the head region data, and connect the visual Token in the Vit model with the regional Token through Concat.
[0125] The present invention not only utilizes the original visual token in the ViT model, but also creates a regional token according to the head region data features, and concatenates the visual token and the regional token through Concat.
[0126] To construct a visual token, the feature map can be adjusted to a one-dimensional vector, the one-dimensional vector is projected to obtain a visual token, and position embedding is added to the visual token to preserve the spatial relationship.
[0127] For example, the ResNet feature extractor outputs a set of feature maps of size H×W×C. The feature maps are resized to P h ×P w ×C, and then project the one-dimensional vector to obtain a visual mark; this process can be expressed as:
[0128]
[0129] Where p represents a one-dimensional patch vector, and v represents a visual marker with a dimension of d. Considering that spatial relationships are crucial for accurate head pose estimation, the present invention also adds position embedding to the visual marker to preserve the spatial relationship:
[0130] [visual]={v1+pos, v2+pos,...,v n +pos}; where n represents the number of tokens.
[0131] Creating a region token based on the head region data includes: dividing the head region data into a plurality of basic regions, and constructing the same number of d-dimensional learnable vectors to represent the plurality of basic regions.
[0132] That is to say, the construction of regional tokens first divides the head region data image into several basic regions. In a specific orientation region, these orientations have high similarity in the head posture data features. After quantitative analysis of the basic region division, the same number of d-dimensional learnable vectors are constructed to represent multiple basic regions. These learning vectors are symbolized as [reg].
[0133] Concat concatenates visual tokens and regional tokens into a higher-dimensional token vector.
[0134] Specifically, Concat(R, V) = [R, V]; wherein, R = [reg], V = [visual], and [R, V] means concatenating the two vectors R and V to form a vector with greater information content.
[0135] Step 302: Construct a Transformer encoder by stacking multiple identical Blocks.
[0136] Among them, Figure 4 As shown in the figure, a Block includes a multi-head self-attention module and a multi-layer perception module, and the multi-head self-attention module and the multi-layer perception module are connected through layer normalization and jump connection.
[0137] It should be noted that the self-attention module can calculate the output through the query matrix, key matrix and value matrix. The multi-head self-attention module splices multiple self-attention operations and then projects them. The operations of Block include layer normalization and jump connection before passing through the multi-layer perception module.
[0138] The self-attention module can be:
[0139]
[0140] Among them, W Q , W k and W v ∈R d×d Represent the query matrix, key matrix and value matrix respectively; R t represents the output of the t-th layer Block, θ represents the scaling factor, and in SA, s is equal to the dimension d of the token. MHSA is an extension of SA with h self-attention operations, called a multi-head self-attention module. In MHSA, θ is usually set to d·h. Therefore, MHSA can be expressed as:
[0141] MHSA(R t )=[SA1(R t ); SA2(R t);...;SA h (R t )]W p ;
[0142] in,
[0143] Block can be represented as:
[0144]
[0145] The MHLP module consists of two linear projections with a Tanh (·) activation function and a dropout operation in between.
[0146] Step 303: construct a token-based head posture extractor, convert multiple groups of regional tokens corresponding to the output of multiple blocks into multiple rotation matrices for training and prediction, and obtain a head posture extractor.
[0147] If a set of regional tokens is represented as Where k is the number of region tokens, and the region tokens are converted into rotation matrices for training and prediction.
[0148] Specifically, linear projection is performed on the regional token to obtain a 6D representation of the head posture, and then the Gram-Schmidt process is applied to generate a 9D rotation matrix. The process can be expressed as:
[0149]
[0150] Where W is the linear projection matrix; is the predicted rotation angle matrix of the i-th basic direction region; F GS (·) represents the Gram-Schmidt process,
[0151] F GS (q1,q2)=[p1,p2,p3],
[0152] q1,q2∈R3 represents the three-dimensional column vector of the rotation matrix, p i A three-dimensional column vector representing a rotation matrix,
[0153]
[0154] The predicted rotation matrix is obtained through the MHLP module and the vector of rotation matrices.
[0155] After a group of regional tokens are converted, a rotation matrix can be generated. After multiple groups of regional tokens corresponding to multiple blocks are converted, a group of rotation matrices can be generated. Where k is the number of regional tokens.
[0156] The final predicted rotation matrix is the head posture angle data. To obtain the final predicted rotation matrix, The concatenated flattening, as the input to the MHLP head, can be expressed as:
[0157]
[0158] in, A vector representing a flattened rotation matrix; W i and b i are the weight matrix and bias vector of the MHLP module respectively.
[0159] It should be noted that the overall loss function L of the head posture extractor is overall It is calculated based on multiple losses including distance loss, regional token loss, and prediction loss.
[0160] As an example, the overall loss function L overall Obtained based on geodesic distance loss, regional Token loss, and prediction loss.
[0161] More specifically,
[0162]
[0163] Among them, α, β, and γ are hyperparameters that balance different losses.
[0164] As another example, the overall loss L overall It can also be obtained based on regional Token loss and prediction loss.
[0165] More specifically,
[0166]
[0167] in, It is a hyperparameter that balances prediction loss and regional Token loss.
[0168] Geodesic distance loss During the training phase, the intermediate rotation matrix and the final predicted rotation matrix are used to calculate the back-propagation loss. During the prediction phase, only the predicted rotation matrix is output. As the model prediction. The prediction result of the proposed FusionViT model is a rotation matrix, expressed as Assuming that the true value rotation matrix is A and using the geodesic distance as the loss function, the geodesic distance loss can be expressed as:
[0169]
[0170] Regional Token Loss L reg , can be encoded into the region Token through the mean square error. Region Token loss L reg Specifically, it can be expressed as:
[0171]
[0172] Where k is the number of basic regions, A is the geodesic true rotation matrix, is the predicted rotation matrix, I(A,i) is the identity function of whether the ground truth head pose is located in the i-th basic area,
[0173]
[0174] Prediction loss L pred , the predictions from the regional tokens are aggregated to form the final prediction of the Fusion ViT model. The prediction loss is:
[0175]
[0176] in, is the model's prediction.
[0177] Step 400: Monitor the student's learning status based on the student's facial data, learning material area data, and head posture angle data, and implement corresponding intervention based on the learning status.
[0178] That is, the student's attention and concentration can be analyzed in real time based on the obtained head posture angle data, the student's learning status can be identified, and appropriate learning intervention can be carried out on the student based on the student's learning status. In addition, the student's learning status data can be transmitted to the cloud in real time for teachers, parents and students to view and analyze.
[0179] After obtaining the student's facial data and head posture angle data, the student's learning status is analyzed based on the learning material area data.
[0180] As an embodiment, monitoring the learning status of students based on student face data, learning material area data, and head posture angle data includes:
[0181] Step 401: Based on the student face data and the learning material area data, obtain the distance between the student face and the learning material, and perform the first analysis.
[0182] The distance between the student and the learning screen is determined based on the student's face data and the learning material area data, and a preliminary inference is made as to whether the student is distracted. The face position data can be used to calculate the distance between the face and the screen, and the face size of the face frame can be used to reflect the distance between the student's head and the learning material.
[0183] It should be noted that when students concentrate on doing homework, the distance between their faces and learning materials will be smaller, while when students start to lose focus, they may move away from the learning materials to take a rest, resulting in a larger distance. When the distance is less than the threshold F, it is considered that the student is too far away from the learning materials, is not thinking about the questions seriously, and may be in a distracted state. Therefore, the composite camera can reflect the student's learning status based on the distance between the student and the learning materials.
[0184] For example, due to the limited space in an independent AI study room, the optimal posture for students to solve problems is about 30 to 100 centimeters away from the learning materials; when the distance exceeds 100 centimeters, the students are no longer within the effective range of the desktop, and when the distance is too close, the learning posture may be problematic; if the threshold is set to 30 to 100, then when the distance is not within this range, it is considered to be a distracted state.
[0185] Step 402: Based on the head posture angle data and the learning material area data, obtain the probability that the student's head attention is on the learning material area, and perform a second analysis.
[0186] The head posture angle data can be used to obtain the three-dimensional Euler rotation angle of the student's head, that is, the orientation of the student's head. When students focus on things, their heads always face the focus of attention. Therefore, by analyzing the head posture, it is possible to determine whether the student is focused on learning. When students are studying hard, they may experience a certain range of head orientation changes. For example, when reading a book, the student's attention will change with the content of the book, and the head may face left or right. He is not completely staring at the desk, but he is indeed studying hard. Therefore, when the student's head is not within a certain angle range, he may be in a serious hearing state.
[0187] Therefore, as an embodiment of the present invention, the three-dimensional Euler rotation angle data and Gaussian probability distribution of head posture are used to obtain the probability that the student's head attention is on the learning material area, and further analyze whether the student is focused.
[0188] More specifically, Gaussian probability is used to determine the probability that the student's head attention is on the learning material area:
[0189]
[0190] Among them, (i, j) represents the coordinates of the center point of the AI study room in the composite camera coordinate system; (x, y) represents the focus of the student's attention, which is the coordinates of the student's head facing a certain projection point on the composite camera plane; w and h represent the width and height of the effective field of view projected on the composite camera plane with the student's head as the center; σ w , σ h They represent the correlation coefficients of w and h, respectively, and are used to control the distribution of Gaussian probability. The range of probability P is (0, 1). Preferably, σ w =0.25w,σ h =0.25h.
[0191] As an example, the concentration score of the second analysis may be calculated as follows:
[0192] S = 100 × P;
[0193] However, during the learning process, students may experience constant changes in head orientation, but they are indeed studying hard. For example, when taking notes, students may switch their head direction back and forth between the textbook and the notebook, which can easily be mistaken for low concentration. In order to reduce false positives, the present invention records the data of the first n seconds during detection and uses a formula to calculate the weighted average of the attention score within m seconds.
[0194] As an embodiment, the concentration score of the second analysis may also be calculated as follows: Where K is the number of detections recorded in the previous n seconds, S(k) is the concentration score of the Kth record in chronological order, is the weight assigned to each time step. The closer the data is to the current time step, the greater the weight. After obtaining the concentration score, it is compared with the set concentration threshold for analysis.
[0195] Step 403: Obtain the student's learning status according to the first analysis result and the second analysis result.
[0196] It should be noted that only when the first and second analysis results are both concentration, does it mean that the student’s learning state is concentration; all other results are distraction.
[0197] In order to improve accuracy, the student's data over a period of time can also be recorded and analyzed to calculate the student's weighted status score. Figure 5 As shown, the students' learning status is divided into three situations: resting, concentrating, and distracted. The students' learning data is uploaded to the cloud for storage, so that the system can make timely intervention reminders and facilitate students to observe their learning status in real time.
[0198] When no one is detected, it is judged as a resting state. When someone is detected, it means that the student has entered the AI study room. The student's learning status will be monitored based on the student's facial data, learning material area data, and head posture angle data, and the monitoring results will be recorded in the remote cloud. For example, if the student's concentration score is above 60 points, it is considered normal learning; if it is below 60 points, it is considered distracted.
[0199] As an embodiment, implementing corresponding intervention based on the learning status includes: setting a prompt device in the AI study room to issue different prompts according to the monitored learning status of the students; wherein the prompt device includes a learning reminder unit, a rest reminder unit and an emergency identification unit.
[0200] It should be noted that when the AI study room is a study room with an independent space, that is, it can only accommodate one person, the prompt device can be set at any position in the AI study room, it can be set on the front wall or on the desk of the AI study room; when the AI study room is a study room that can accommodate multiple people, it is preferably set on the desk corresponding to each student.
[0201] The learning reminder unit is used to remind students of their studies.
[0202] For example, Figure 5 As shown in the figure, when a student is identified as distracted and the state lasts for more than 10 seconds, the reminder light will turn red to remind the student to study. If the student continues to be distracted for more than 1 minute, the reminder light will continue to flash. During the self-study process, students can turn off the reminder light and enter the rest state when they want to take a break. At this time, the system will no longer collect the student's learning data.
[0203] The break reminder unit is used to remind students to take a break.
[0204] The rest reminder unit will comprehensively consider the student's study time, concentration score, and the frequency of head posture changes, and intelligently customize a personalized rest plan for each student. It will also track the student's continuous study time ST. When the preset study time threshold is reached, the rest reminder unit will automatically turn off the reminder light or light up green to remind the student to take a short break.
[0205] In addition, the rest reminder unit will monitor the student's concentration score. If the score drops rapidly, it indicates that the student may feel tired and will enter the rest mode in advance. The rest reminder unit will record the student's rest mode and feedback in order to continuously optimize the rest interval algorithm to make it more suitable for the student's personal needs, and help students establish scientific study and rest habits to achieve a balance between long-term learning effects and health status.
[0206] Emergency recognition unit to predict potential emergency situations.
[0207] The emergency recognition unit can predict potential emergencies by analyzing students' head posture, behavior patterns, and abnormal movements. If the emergency recognition unit identifies a student's sudden violent movement or abnormally long period of stillness, this may indicate that the student is in an emergency. The emergency recognition unit will be activated to trigger a multi-mode alarm system, which includes the continuous flashing of the indicator light and the sound of the alarm to attract the attention of people around. At the same time, the emergency recognition unit will send an emergency notification to the management staff in the study room through the automated communication system, and provide specific location information and real-time status data of the students to ensure that quick and effective action can be taken in an emergency to protect the safety and health of students.
[0208] In addition, the prompt device also includes a storage unit for storing all the data of the student collected by the composite camera in units of different students. Students can log in to the client at any time to view their learning data, and a dynamic learning status dashboard will be provided, including learning trend charts, concentration heat maps and rest frequency analysis to identify students' learning habits and efficiency patterns, and use predictive algorithms to provide students with personalized learning suggestions and improvement strategies based on their learning performance and historical data, so as to promote students' personalized learning development.
[0209] On the other hand, the present invention also provides a learning status intervention system for an AI study room, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above methods. The technical features of the system are consistent with those of the above methods, and will not be described one by one here.
[0210] It should be noted that, for the above-mentioned various embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0211] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0212] In the several embodiments provided in this application, it should be understood that the disclosed method or system can be implemented in other ways. For example, the above-described embodiments are only schematic, and the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.
[0216] A person skilled in the art can understand that all or part of the various circuits in the above-mentioned embodiments can be completed by entering a program to instruct related hardware. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0217] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.
[0218] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0219] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for intervening in the learning state of an AI study room, characterized in that: The method comprises: Using a composite camera to collect learning material area data and self-study video data including the student's complete head posture in real time; wherein the composite camera includes an integrated infrared camera and a depth camera; the self-study video data includes infrared visual data and depth space data; Preprocessing the self-study video data to obtain head area data and student face data; Inputting the head region data into a head posture extractor to obtain head posture angle data; The student's learning status is monitored according to the student's facial data, the learning material area data and the head posture angle data, and corresponding intervention is implemented based on the learning status.
2. A method for intervening in the learning state of an AI study room according to claim 1, characterized in that: Use a composite camera to collect self-study video data including the student's complete head posture in real time, including: Inputting the collected current frame image into a feature extraction network including a plurality of blocks connected in series, and correspondingly obtaining a plurality of feature maps with different resolutions; Fusing a plurality of the feature maps to obtain a feature fusion map; Calculate the face loss function based on the feature fusion graph; The acquisition angle of the composite camera is adjusted to minimize the face loss function, and the current acquisition angle is used as the camera acquisition angle of the next frame.
3. A method for intervening in the learning state of an AI study room according to claim 2, characterized in that: Calculating the face loss function based on the feature fusion graph includes: Based on the feature fusion map, obtain the coordinates of the center point of the face, the center point offset, and the height and width of the face on the feature fusion map; The face center point classification loss, the center point offset loss, and the face size loss are calculated respectively to obtain the face loss function.
4. The method for intervening in the learning state of an AI study room according to claim 1, characterized in that: The pre-processing comprises: Performing data synthesis on the infrared visual data to obtain first visual data; Performing shadow removal on the depth space data to obtain first space data; Matching the resolution of the first visual data and the first spatial data to obtain second visual data and second spatial data accordingly; Converting pixel coordinates of the second spatial data into coordinates in a world coordinate system to obtain three-dimensional spatial information, and mapping the three-dimensional spatial information to the second visual data to obtain preliminary fusion data; The face region is segmented based on the preliminary fusion data to obtain the head region data and the student face data.
5. The method for intervening in the learning state of an AI study room according to claim 1, characterized in that: The construction of the head posture extractor includes: Construct a Token, create a regional Token according to the head region data, and connect the visual Token in the Vit model with the regional Token through Concat; Construct a Transformer encoder by stacking multiple identical Blocks; wherein a Block includes a multi-head self-attention module and a multi-layer perception module, and the multi-head self-attention module and the multi-layer perception module are connected through layer normalization and jump connection; A Token-based head posture extractor is constructed, and multiple groups of regional Tokens corresponding to the outputs of multiple Blocks are converted into multiple rotation matrices for training and prediction to obtain the head posture extractor.
6. A method for intervening in the learning state of an AI study room according to claim 5, characterized in that: Creating a region Token according to the head region data includes: dividing the head region data into a plurality of basic regions, and constructing the same number of d-dimensional learnable vectors to represent the plurality of basic regions.
7. A method for intervening in the learning state of an AI study room according to claim 5, characterized in that: Monitoring the student's learning status according to the student's face data, the learning material area data, and the head posture angle data includes: Based on the student face data and the learning material area data, obtain the distance between the student face and the learning material, and perform a first analysis; Based on the head posture angle data and the learning material area data, the probability of the student's head attention being on the learning material area is obtained, and a second analysis is performed; The students' learning status is obtained based on the first analysis results and the second analysis results.
8. A method for intervening in the learning state of an AI study room according to claim 7, characterized in that: The probability of getting the student's head attention on the learning material area includes: Use Gaussian probability to determine the probability that the student's head attention is on the learning material area: Among them, (i, j) represents the coordinates of the center point of the AI study room in the composite camera coordinate system; (c, y) represents the focus of the student's attention, which is the coordinates of the student's head facing a certain projection point on the composite camera plane; w and h represent the width and height of the effective field of view projected on the composite camera plane with the student's head as the center; σ w , σ h Respectively represent the correlation coefficients of w and h, which are used to control the distribution of Gaussian probability.
9. The method for intervening in the learning state of an AI study room according to claim 1, characterized in that: Implementing corresponding interventions based on the learning status includes: A prompt device is arranged in the AI study room to issue different prompts according to the monitored learning status of the students; wherein the prompt device includes a learning reminder unit, a rest reminder unit and an emergency identification unit.
10. A learning status intervention system for an AI study room, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.
Citation Information
Cited By
Multi-source data-based conference participant state analysis and dynamic intervention method and system
CN121074774A
