A learning concentration monitoring method based on a Transformer network
By employing a multimodal data fusion method based on Transformer networks, combining head posture, ambient sound, and skin conductance signals, the accuracy of student attention assessment in online teaching was solved, enabling a comprehensive evaluation of learning emotions and attention throughout the entire process.
Patent Information
- Application Number
- CN202211596338.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-12
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2042-12-12
AI Technical Summary
Existing technologies struggle to accurately assess students’ learning focus in online teaching environments, especially head posture estimation, which is limited by a lack of detailed information and the influence of lighting changes, leading to biased assessment results.
A multimodal data fusion method based on Transformer networks is adopted, which combines head pose RGB images, ambient sound data and skin conductance signals. Attention monitoring is learned through feature extraction, encoding and multi-layer Transformer networks, and comprehensive evaluation is carried out using complementary data.
It enables a comprehensive evaluation of students' learning emotions and focus throughout the entire process, improving the accuracy and reliability of the assessment and allowing for a more accurate acquisition of students' emotional focus.
Smart Images
Figure CN116127350B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of pattern recognition and information processing technology, and more specifically, to a method, system, device, and storage medium for monitoring learning focus based on Transformer networks. Background Technology
[0002] Learning emotional focus is an important means of assessing student learning effectiveness and quality, and it is a key issue in studying learners' learning states. Basic technologies based on image and audio behavior recognition are widely used to assess students' learning emotional focus, analyze their learning states, and help them improve their concentration. However, due to the increasing informatization of the world today, interference from computer equipment and the environment makes students more prone to anxiety and a decline in their ability to concentrate, especially in online teaching scenarios lacking teacher supervision, where assessing student focus during the learning process becomes even more difficult. Therefore, a method for quantitatively assessing students' learning emotional focus is urgently needed.
[0003] Currently, head pose estimation is often used to assess students' concentration. However, the difficulty of head pose estimation lies in the fact that the head is a rigid body with little detailed information other than facial expressions; information varies greatly at different distance scales and is easily affected by changes in lighting in the learning environment. These factors can all lead to biases in the assessment of concentration. Summary of the Invention
[0004] To address at least one deficiency or improvement need in the existing technology, this invention provides a learning focus monitoring method, system, device, and storage medium based on Transformer networks. By introducing complementary multimodal data and temporal information for learning focus assessment, it can achieve comprehensive and full-process evaluation of students' learning emotions and focus, thereby improving the accuracy and reliability of the results.
[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for monitoring learning attention based on Transformer networks is provided, the method comprising the following steps:
[0006] Acquire RGB images of the head posture of the monitored object, environmental sound data, and skin conductance signals during the monitoring period, and divide the monitoring period into multiple time-domain segments according to time sequence;
[0007] The RGB image of the head pose, the ambient sound data, and the electrodermal signal corresponding to each time domain segment are input into the trained multimodal information classification model to obtain the head pose angle, ambient sound category, and emotional state category.
[0008] The head posture angle, ambient sound category, and emotional state category are encoded and weighted and summed according to preset weights to obtain the emotional focus score of the monitored object in each time domain segment.
[0009] The emotional focus scores of each time segment are merged according to preset rules to obtain the focus estimation results of the monitored subject during the monitoring period.
[0010] Furthermore, in the above-mentioned learning attention monitoring method, the multimodal information classification model includes a head pose recognition model, used to predict the head pose angle based on the head pose RGB image; the head pose recognition model includes:
[0011] A feature extraction network is used to extract multi-scale feature vectors from the input head pose RGB image;
[0012] The first encoding layer is used to map the multi-scale feature vectors to the same dimension through a linear transformation, generate a one-dimensional feature vector, and embed randomly initialized learnable parameters.
[0013] A multi-layer transformer network, which uses an attention mechanism to operate on the one-dimensional feature vector output by the first encoding layer to obtain the final head pose feature vector;
[0014] The first classification network performs classification calculations based on the head posture feature vector to obtain the head posture angle of the monitored object.
[0015] Furthermore, in the above-mentioned learning attention monitoring method, the multimodal information classification model includes an environmental sound classification model, used to predict the environmental sound category based on environmental sound data; the environmental sound classification model includes:
[0016] The second encoding layer is used to encode the environmental sound data and generate a one-dimensional sound feature vector.
[0017] A multi-layer transformer network, based on an attention mechanism, operates on the one-dimensional sound feature vector output by the second coding layer to obtain the final environmental sound feature vector;
[0018] The second classification network performs classification calculations based on the environmental sound feature vectors to obtain the environmental sound category of the environment in which the monitored object is currently located.
[0019] Furthermore, in the above-mentioned learning attention monitoring method, the multimodal information classification model includes a skin conductance emotion classification model, used to predict the emotion state category based on skin conductance signals; the skin conductance emotion classification model includes:
[0020] The third coding layer is used to encode the skin conductance signal and generate a one-dimensional physiological feature vector.
[0021] A multi-layer transformer network, based on an attention mechanism, operates on the one-dimensional physiological feature vector output by the third encoding layer to obtain the final physiological emotion feature vector;
[0022] The third classification network performs classification calculations based on the physiological emotion feature vectors to obtain the emotional state category of the monitored object.
[0023] Furthermore, in the above-mentioned learning attention monitoring method, the head posture recognition model, the ambient sound classification model, and the skin conductance emotion classification model share the same multi-layer transformer network.
[0024] Furthermore, in the above-mentioned learning attention monitoring method, the first classification network includes three consecutive fully connected layers;
[0025] The output of the first fully connected layer is connected to the second and third fully connected layers respectively; the output of the second fully connected layer is connected to the third fully connected layer.
[0026] Furthermore, in the above-mentioned learning focus monitoring method, the step of fusing the emotional focus scores of each time segment according to a preset rule includes:
[0027] Weights are assigned to the learning content corresponding to each time segment based on its importance, and the emotional focus scores corresponding to each time segment are weighted and summed according to the weights.
[0028] According to a second aspect of the present invention, a learning attention monitoring system based on Transformer networks is also provided, the system comprising:
[0029] The data acquisition module is used to acquire the head posture RGB image, environmental sound data and skin conductance signal of the monitored object during the monitoring period, and divide the monitoring period into multiple time domain segments according to the time sequence.
[0030] The classification module is used to input the corresponding head pose RGB image, ambient sound data and skin conductance signal in each time domain segment into the trained multimodal information classification model to obtain the head pose angle, ambient sound category and emotional state category.
[0031] The focus calculation module is used to encode the head posture angle, ambient sound category and emotional state category and perform weighted summation according to preset weights to obtain the emotional focus score of the monitored object in each time domain segment.
[0032] The results output module is used to merge the emotional focus scores of each time segment according to preset rules to obtain the focus estimation results of the monitored object during the monitoring period.
[0033] According to a third aspect of the present invention, a computer device is also provided, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of any of the above-described learning attention monitoring methods.
[0034] According to a fourth aspect of the invention, a storage medium is also provided that stores a software program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of any of the above-described learning attention monitoring methods.
[0035] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:
[0036] (1) This invention introduces complementary multimodal data and time-domain information to evaluate learning focus, which can achieve a comprehensive and full-process evaluation of students' learning emotions and focus; the Transformer architecture can more accurately obtain the multimodal information classification results of learners, and can more accurately obtain and evaluate students' emotional focus, thereby helping students' development.
[0037] (2) The multimodal information fusion index system adopted in this invention can reasonably evaluate learners’ emotional focus. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating a learning attention monitoring method based on Transformer networks provided in this embodiment.
[0040] Figure 2 This is a schematic diagram of the data acquisition scenario provided in this embodiment;
[0041] Figure 3 This is a schematic diagram of the network structure of the multimodal information classification model provided in this embodiment. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0043] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0044] Furthermore, to avoid obscuring the understanding of the invention by those skilled in the art, well-known or widely used techniques, elements, structures, and processes may not be described or shown in detail. Although the accompanying drawings illustrate exemplary embodiments of the invention, the drawings are not necessarily drawn to scale, and specific features may be enlarged or omitted to better illustrate and explain the invention.
[0045] Figure 1 This is a flowchart illustrating a learning attention monitoring method based on Transformer networks provided in this embodiment. Please refer to [link / reference]. Figure 1 The method mainly includes the following steps:
[0046] S1 acquires the head posture RGB image, ambient sound data and skin conductance signal of the monitored object during the monitoring period, and divides the monitoring period into multiple time domain segments according to the time sequence.
[0047] In this embodiment, the monitored object is the learner who is currently engaged in online learning.
[0048] Figure 2 This is a schematic diagram of the data acquisition scenario provided in this embodiment, such as... Figure 2 As shown, an RGB camera installed on the learning video playback device can capture the learner's head posture RGB image, an ambient sound recording device can record ambient sound information during the learning process, and a contact-type ductal resonance imaging (DRI) device can collect DR signals during the learning process. Then, the acquired three types of information are preprocessed and segmented according to time sequence.
[0049] Specifically, an RGB camera installed on the learning video playback device captures RGB images of the learner's head posture. A suitable visual capture interval can be preset, triggering the RGB camera to collect the learner's head posture information at preset intervals. Then, the captured head posture RGB images undergo processing such as resolution compression and image scaling.
[0050] Ambient sounds are recorded using a dedicated ambient sound recording device placed near the learner. Then, after signal noise reduction and format conversion, the ambient audio information of the learner during the learning process is extracted.
[0051] Electrodermal contact patches are used to capture human skin electrical signals. These signals are then processed through noise reduction and format conversion to extract the learner's electrodermal signal information.
[0052] The information recorded throughout the process is divided into several time-domain segments according to the time-series segmentation rules, given a fixed segmentation interval ε∈(5±2)s.
[0053] S2 inputs the corresponding head pose RGB image, ambient sound data and skin conductance signal in each time domain segment into the trained multimodal information classification model to obtain head pose angle, ambient sound category and emotional state category.
[0054] Figure 3 This is a schematic diagram of the network structure of the multimodal information classification model provided in this embodiment; please refer to [link / reference]. Figure 3 In this embodiment, the multimodal information classification model includes a head posture recognition model, an ambient sound classification model, and a skin conductance emotion classification model. The head posture recognition model is used to predict the head posture angle based on the head posture RGB image; the ambient sound classification model is used to predict the ambient sound category based on the ambient sound data; and the skin conductance emotion classification model is used to predict the emotion state category based on the skin conductance signal.
[0055] In one optional implementation, the head pose recognition model includes a feature extraction network, a first encoding layer, a multi-layer transformer network, and a first classification network; wherein,
[0056] The feature extraction network is used to extract multi-scale feature vectors from the input head pose RGB image. In this embodiment, the feature extraction network uses a ResNet-50 pre-trained network to extract multi-scale features from the head pose image. This model has a 50-layer structure. The input is a 3×224×224 tensor. After passing through 50 convolutional blocks, the output is a tensor with a size of 2048×7×7. The tensors of the last three stages are selected and concatenated using the concat function in the PyTorch framework to output multi-scale feature vectors.
[0057] The specific processing procedure is as follows:
[0058] For an image of a head, x∈R H×W×C First, the data is input into ResNet to obtain multi-scale feature vectors. The formal expression of ResNet's operations is as follows:
[0059] C=F(x,{W i})+x (1)
[0060] Where x is the input image, C is the feature vector, and F is a multi-layer fully connected layer, the specific expression of F is as follows:
[0061] F=W2σ(W1x) (2)
[0062] Where σ is the ReLU activation function, defined as follows:
[0063]
[0064] After processing by ResNet, the final three stages, C1, C2, and C3, are multi-scale feature maps, as described below:
[0065]
[0066] The first encoding layer is used to map the multi-scale feature vectors to the same dimension through a linear transformation, generate a one-dimensional feature vector, and embed randomly initialized learnable parameters.
[0067] The multi-scale feature map output by the feature extraction network is input into the first encoding layer, which flattens it into a one-dimensional vector, as expressed by the formula:
[0068]
[0069] in, This is a flattening operation. Linear renormalization transforms vectors of different dimensions into vectors of the same dimension d = 256, expressed by the formula:
[0070] E:p′ i →v i ∈R d (6)
[0071] E i For linear transformation renormalization operation; v i To renormalize and obtain a one-dimensional multi-scale eigenvector.
[0072] Then, the multi-scale feature vectors are concatenated together and 2D positional encoding is added to form a visual vector; where v iLet i = 1, 2, 3, ..., L be a multi-scale one-dimensional feature vector. Concatenate and add 2D positional encoding pe. i The operation to obtain the visual vector is as follows:
[0073] [visual]={v1+pe1,v2+pe2,…,v L +pe L} (7)
[0074] The visual vector is concatenated with the head pose token to form T, which is then used as the input to a multi-layer transformer network.
[0075] T={[visual],[headposetoken]} (8)
[0076] Here, [head pose token] represents a randomly initialized learnable parameter.
[0077] The multi-layer transformer network operates on the one-dimensional feature vector output by the first encoding layer based on the attention mechanism to obtain the final head pose feature vector;
[0078] This multi-layer transformer network consists of multiple transformer blocks, and the computation steps for each transformer block are as follows:
[0079] Step (1): Layer normalization operation
[0080]
[0081]
[0082]
[0083] Among them, a l This is the original output of the l-th layer. This is the result obtained after layer normalization.
[0084] Step (2): First, calculate the single-head self-attention (SA). The specific calculation method is as follows:
[0085]
[0086] Among them, W Q W k W V is a learnable parameter in the network, and h is the number of heads for self-attention operations. Preferably, h is set to 8, i.e., 8-head self-attention.
[0087] Step (3): Calculate the multi-head self-attention. The specific calculation method is as follows:
[0088] MSA(T)=[SA1(T);SA2(T);SA3(T);…,SA h (T)]W P (13)
[0089] In this case, there is a residual connection between the output of the multi-head self-attention system and the original input, which can be expressed by the formula:
[0090] z′ l =MSA(LN(z) l-1 ))+z l-1 (14)
[0091] Step (4): Perform another layer normalization operation, and then pass it through a fully connected neural network. The output of the fully connected neural network also has a residual connection with the original input. This process is described as follows:
[0092] z l =MLP(LN(z′) l ))+z′ l (15)
[0093] This completes the computation of a single Transformer block.
[0094] Step (5): Repeat the data through M Transformer blocks to obtain the final head pose feature vector; preferably, M is set to 12.
[0095] The first classification network is used to perform classification calculations based on the head posture feature vector to obtain the head posture angle of the monitored object.
[0096] The head pose feature vector output from the multi-layer transformer is used as the input to the first classification network. Different head pose angles are calculated through fully connected layers and the activation function softmax. In this embodiment, the first classification network includes three consecutive fully connected layers, which are also interconnected. The output of the first fully connected layer is connected to the second and third fully connected layers respectively. After passing through the first layer, the parameters are added to the second and third fully connected layers. The output of the second fully connected layer is connected to the third fully connected layer. After passing through the second layer, the parameters are added to the third fully connected layer, achieving dense connections between the three layers, and finally outputting three Euler angles. The specific calculation process is as follows:
[0097]
[0098] in, is the j-th neuron in the fully connected layer; w and b are the connection parameters between the j-th neuron in the fully connected layer and the hidden layer above; σ is the activation unit, and its calculation formula is as follows:
[0099]
[0100] Applying a softmax operation to the output of the third fully connected layer yields the three Euler angles of the head pose, which represent the head pose classification result A. cls The softmax operation is performed as follows:
[0101]
[0102] In this embodiment, the training process of the head pose recognition model is as follows:
[0103] Step 1.1: Construct the training sample set and the test sample set;
[0104] Step 1.2: Data augmentation of the test set samples. Data preprocessing plays a crucial role in training the neural network. Data augmentation methods include: enhancing RGB images through rotation, translation, scaling, etc., thereby improving the robustness of the model; at the same time, these operations also provide a large number of fake samples for model training.
[0105] Step 1.3: Train the head pose recognition model using the training sample set. Calculate the loss by comparing the Gaussian distribution of the predicted head pose of the training samples with the Gaussian distribution of the actual head pose using the cross-entropy loss function. Use the AdamW optimizer for gradient optimization. Set the initial learning rate to 0.01. After several epochs of training, reduce the learning rate using cosine annealing to make the network learn more stably until the loss value no longer decreases.
[0106] Step 1.4: Fine-tune the head pose recognition model using the test sample set, with the learning rate set to 5e-6.
[0107] In one optional implementation, the ambient sound classification model includes a second coding layer, a multi-layer transformer network, and a second classification network;
[0108] The second coding layer is used to encode the environmental sound data and generate a one-dimensional sound feature vector;
[0109] A multi-layer transformer network uses an attention mechanism to process the one-dimensional sound feature vector output from the second coding layer to obtain the final environmental sound feature vector.
[0110] The second classification network performs classification calculations based on the environmental sound feature vectors to obtain the environmental sound category of the environment in which the monitored object is currently located. Specifically, the second classification network calculates different environmental sound categories by passing the environmental sound feature vectors through a fully connected layer and an activation function softmax.
[0111] In this embodiment, the training process of the environmental sound classification model is as follows:
[0112] Step 2.1: Construct the training sample set and the test sample set;
[0113] Step 2.2: Train the ambient sound classification model using the training sample set. Calculate the loss by comparing the predicted ambient sound category and the real ambient sound category of the training samples using the cross-entropy loss function. Use the AdamW optimizer for gradient optimization. Set the initial learning rate to 0.01. After several epochs of training, reduce the learning rate using cosine annealing to make the network learn more stably until the loss value no longer decreases.
[0114] Step 2.3: Fine-tune the environmental sound classification model using the test sample set, with the learning rate set to 5e-6.
[0115] The skin conductance emotion classification model includes a third encoding layer, a multi-layer transformer network, and a third classification network; among them,
[0116] The third coding layer is used to encode the skin conductance signal and generate a one-dimensional physiological feature vector;
[0117] A multi-layer transformer network uses an attention mechanism to process the one-dimensional physiological feature vector output by the third encoding layer to obtain the final physiological emotion feature vector.
[0118] The third classification network performs classification calculations based on the physiological emotion feature vectors to obtain the emotional state category of the monitored object. The physiological emotion feature vectors obtained by the third classification network are used to calculate different emotional representations through a fully connected layer and an activation function softmax. In this embodiment, the emotional state categories are divided into three categories: positive, neutral, and negative.
[0119] In this embodiment, the training process of the skin conductance emotion classification model is as follows:
[0120] Step 3.1: Construct the training sample set and the test sample set;
[0121] Step 3.2: Train the skin conduction emotion classification model using the training sample set. Calculate the loss by comparing the predicted emotion category and the true emotion category of the training samples using the cross-entropy loss function. Use the AdamW optimizer for gradient optimization. Set the initial learning rate to 0.01. After several epochs of training, reduce the learning rate using cosine annealing to make the network learn more stably until the loss value no longer decreases.
[0122] Step 3.3: Fine-tune the skin conductance emotion classification model using the test sample set, with the learning rate set to 5e-6.
[0123] In a preferred embodiment, the head pose recognition model, ambient sound classification model, and skin conductance emotion classification model share the same multi-layer transformer network. The processing of ambient sound and skin conductance information is similar to that of head pose RGB image data, and so on, to derive the ambient sound category B. cls and emotional state category C cls .
[0124] S3 encodes the head posture angle, ambient sound category, and emotional state category, and performs a weighted summation according to preset weights to obtain the emotional focus score of the monitored object in each time domain segment.
[0125] First, the Euler angles of head pose output by the head pose recognition model, the environmental sound category output by the environmental sound classification model, and the skin conductance emotion category output by the skin conductance emotion classification model are encoded.
[0126]
[0127] Wherein, Ψ(·) is the encoding function; preferably, one-hot encoding is used as the encoding function to encode the head posture angle, ambient sound category and emotional state category.
[0128] Then, the encoded head posture angle, ambient sound category, and emotional state category are weighted and scored to obtain the emotional focus score for the i-th time segment. This process is expressed by the following formula:
[0129] S i =αA code +βB code +γC code (19)
[0130] Among them, A code B code C codeThese are the head posture angle category code, the ambient sound category code, and the skin conductance signal category code, respectively; α, β, and γ are the corresponding weights. After calculation, the given weight ranges are α∈[0.5±0.15], β∈[0.25±0.15], and γ∈[0.25±0.15].
[0131] S4 integrates the emotional focus scores of each time segment according to preset rules to obtain the focus estimation results of the monitored subject during the monitoring period.
[0132] In an optional embodiment, weights are assigned to the learning content corresponding to each time segment based on its importance, and the emotional focus scores corresponding to each time segment are weighted and summed according to the weights.
[0133] In this embodiment, fragmented learning emotional focus is obtained at time intervals ε∈(5±2)s, and a learning emotional focus score S is calculated every ε∈(5±2)s throughout the entire learning process. i , all S i The emotional focus score for the entire stage is obtained by summing the results according to certain weights, as follows:
[0134]
[0135] The value of λ can be set empirically. In a specific example, the range of the parameter λ is given as follows:
[0136]
[0137] The final score, S, represents the emotional focus score throughout the entire learning process.
[0138] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0139] This embodiment also provides a learning attention monitoring system based on Transformer networks. This system can be implemented using software and / or hardware and can be integrated into a computer device. Specifically, the system includes a data acquisition module, a classification module, an attention calculation module, and a result output module.
[0140] The data acquisition module is used to acquire the head posture RGB image, environmental sound data and skin conductance signal of the monitored object during the monitoring period, and divide the monitoring period into multiple time domain segments according to the time sequence;
[0141] The classification module is used to input the corresponding head pose RGB image, ambient sound data and skin conductance signal in each time domain segment into the trained multimodal information classification model to obtain the head pose angle, ambient sound category and emotional state category.
[0142] The focus calculation module is used to encode the head posture angle, ambient sound category and emotional state category, and perform weighted summation according to preset weights to obtain the emotional focus score of the monitored object in each time domain segment.
[0143] The results output module is used to merge the emotional focus scores of each time domain segment according to preset rules to obtain the focus estimation results of the monitored subject during the monitoring period.
[0144] Specific limitations regarding the learning focus monitoring system can be found in the limitations of the learning focus monitoring method described above, and will not be repeated here. Each module in the aforementioned learning focus monitoring system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0145] This embodiment also provides a computer device, which includes at least one processor and at least one memory. The memory stores a computer program. When the computer program is executed by the processor, it causes the processor to perform the steps of the learning attention monitoring method. The specific steps are described above and will not be repeated here. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be volatile memory, non-volatile memory, or a combination thereof.
[0146] The computer device can also communicate with one or more external devices (such as a keyboard, pointing terminal, monitor, etc.), one or more terminals that enable a user to interact with the computer device, and / or any terminal that enables the computer device to communicate with one or more other computing terminals (such as a network card, modem, etc.). This communication can be performed through an input / output (I / O) interface. Furthermore, the computer device can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) via a network adapter.
[0147] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.
[0148] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0149] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0150] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.
[0151] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0152] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0154] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0155] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.
[0156] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for monitoring learning attention based on Transformer networks, characterized in that, include: Acquire RGB images of the head posture of the monitored object, environmental sound data, and skin conductance signals during the monitoring period, and divide the monitoring period into multiple time-domain segments according to time sequence; The head pose RGB image, ambient sound data, and skin conductance signal corresponding to each time domain segment are input into the trained multimodal information classification model to obtain the head pose angle, ambient sound category, and emotional state category; the multimodal information classification model includes a head pose recognition model, which is used to predict the head pose angle based on the head pose RGB image. The head pose recognition model includes: A feature extraction network is used to extract multi-scale feature vectors from the input head pose RGB image; The first encoding layer is used to map the multi-scale feature vectors to the same dimension through a linear transformation, generate a one-dimensional feature vector, and embed randomly initialized learnable parameters. A multi-layer transformer network, which uses an attention mechanism to operate on the one-dimensional feature vector output by the first encoding layer to obtain the final head pose feature vector; The first classification network performs classification calculations based on the head posture feature vector to obtain the head posture angle of the monitored object. The prediction processes for ambient sound categories, emotional state categories, and head posture angles all share the same multi-layer transformer network. The head posture angle, ambient sound category, and emotional state category are encoded and weighted and summed according to preset weights to obtain the emotional focus score of the monitored object in each time domain segment. The emotional focus scores of each time segment are merged according to preset rules to obtain the focus estimation results of the monitored subject during the monitoring period.
2. The learning focus monitoring method as described in claim 1, characterized in that, The multimodal information classification model includes an environmental sound classification model, used to predict the environmental sound category based on environmental sound data; the environmental sound classification model includes: The second encoding layer is used to encode the environmental sound data and generate a one-dimensional sound feature vector. A multi-layer transformer network, based on an attention mechanism, operates on the one-dimensional sound feature vector output by the second coding layer to obtain the final environmental sound feature vector; The second classification network performs classification calculations based on the environmental sound feature vectors to obtain the environmental sound category of the environment in which the monitored object is currently located.
3. The learning focus monitoring method as described in claim 2, characterized in that, The multimodal information classification model includes a skin conductance emotion classification model, which is used to predict the emotion state category based on skin conductance signals; The skin conductance emotion classification model includes: The third coding layer is used to encode the skin conductance signal and generate a one-dimensional physiological feature vector. A multi-layer transformer network, based on an attention mechanism, operates on the one-dimensional physiological feature vector output by the third encoding layer to obtain the final physiological emotion feature vector; The third classification network performs classification calculations based on the physiological emotion feature vectors to obtain the emotional state category of the monitored object.
4. The learning focus monitoring method as described in claim 3, characterized in that, The head pose recognition model, ambient sound classification model, and skin conductance emotion classification model all share the same multi-layer transformer network.
5. The learning focus monitoring method as described in claim 1, characterized in that, The first classification network comprises three consecutive fully connected layers; The output of the first fully connected layer is connected to the second and third fully connected layers respectively; the output of the second fully connected layer is connected to the third fully connected layer.
6. The learning focus monitoring method according to any one of claims 1-5, characterized in that, The process of fusing the emotional focus scores of each time segment according to preset rules includes: Weights are assigned to the learning content corresponding to each time segment based on its importance, and the emotional focus scores corresponding to each time segment are weighted and summed according to the weights.
7. A learning attention monitoring system based on Transformer networks, characterized in that, include: The data acquisition module is used to acquire the head posture RGB image, environmental sound data and skin conductance signal of the monitored object during the monitoring period, and divide the monitoring period into multiple time domain segments according to the time sequence. The classification module is used to input the corresponding head posture RGB image, ambient sound data and skin conductance signal in each time domain segment into the trained multimodal information classification model to obtain the head posture angle, ambient sound category and emotional state category; the multimodal information classification model includes a head posture recognition model, which is used to predict the head posture angle based on the head posture RGB image. The head pose recognition model includes: A feature extraction network is used to extract multi-scale feature vectors from the input head pose RGB image; The first encoding layer is used to map the multi-scale feature vectors to the same dimension through a linear transformation, generate a one-dimensional feature vector, and embed randomly initialized learnable parameters. A multi-layer transformer network, which uses an attention mechanism to operate on the one-dimensional feature vector output by the first encoding layer to obtain the final head pose feature vector; The first classification network performs classification calculations based on the head posture feature vector to obtain the head posture angle of the monitored object. The prediction processes for ambient sound categories, emotional state categories, and head posture angles all share the same multi-layer transformer network. The focus calculation module is used to encode the head posture angle, ambient sound category and emotional state category and perform weighted summation according to preset weights to obtain the emotional focus score of the monitored object in each time domain segment. The results output module is used to merge the emotional focus scores of each time segment according to preset rules to obtain the focus estimation results of the monitored object during the monitoring period.
8. A computer device, characterized in that, The method includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program that, when executed by the processing unit, causes the processing unit to perform the steps of the learning attention monitoring method according to any one of claims 1 to 6.
9. A storage medium, characterized in that, It stores a software program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the learning attention monitoring method according to any one of claims 1 to 6.