Gamification learning adaptability adjusting method based on emotional physiological binary network

Abstract

The invention provides a gamification learning adaptability adjusting method and device based on an emotional physiological binary network, and belongs to the technical field of informatization education. The method comprises the steps that video data and electroencephalogram data of gamification learning of students are collected through an RGB camera and electroencephalogram equipment; generating facial expression features and electroencephalogram signal features based on the facial image, the electroencephalogram signal and a gamification learning state extractor; according to the facial expression features and the electroencephalogram features, the emotion condition and the physiological condition of the learner are recognized respectively; the self-adaption degree of the student is comprehensively evaluated by taking the emotional condition and the physiological condition as indexes, and a personalized adjustment strategy is provided. According to the method, the emotional state and the physiological state of the student under gamification learning in the remote learning environment are taken as the reference, and the self-adaption degree of the student is accurately described in real time, so that personalized adjustment strategies and feedback suggestions are provided, and teachers and learners are helped to improve the gamification learning effect in the remote learning environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information-based education, and in particular to a game-based learning adaptability adjustment method based on an emotional and physiological dual network. BACKGROUND

[0002] With the rapid development of information-based education technology, the education field has also ushered in an unprecedented revolution. In particular, in the remote learning environment, how to effectively improve the learning effect of students has become the focus of educators and researchers. Game-based learning, as a new teaching method, has shown great potential in stimulating students' interest and enthusiasm in learning due to its interesting, interactive and immersive nature. Specifically, game-based learning is a multi-stage game task with certain scenarios to enable learners to achieve learning goals.

[0003] At present, game-based learning in the remote learning environment still faces many challenges. In the traditional remote learning mode, teachers often have difficulty accurately grasping the real-time emotional state and physiological condition of students in the learning process, making it difficult to make timely and effective adaptive teaching adjustment strategies. In addition, traditional expression recognition usually has only a single image as input, and there are problems such as insufficient data or noise frame interference, resulting in inaccurate prediction results.

[0004] With the continuous progress of artificial intelligence and big data technology, new ideas and methods are provided for the adaptive adjustment of game-based learning. Educators assess the adaptive degree of students' learning by collecting facial expressions and brain electrical signal information of students in a remote environment, and in addition, the application of visual language models can help educators more accurately grasp the emotional state and physiological condition of students in the game-based learning process, thereby improving the accuracy of student adaptive degree assessment. SUMMARY

[0005] The present application provides a game-based learning adaptability adjustment method and device based on an emotional and physiological dual network to solve the defects of at least one of the prior art.

[0006] In a first aspect, the present application provides a game-based learning adaptability adjustment method based on an emotional and physiological dual network, comprising:

[0007] Step 1: Obtain facial expression video data of a target object in a remote learning environment, and extract frame image sequences from the obtained video resources at a preset fixed time interval, and divide the frame into image blocks;

[0008] Step 2: input the image blocks obtained after division into the expression recognition model ITFM of the game-based learning state extractor, and obtain the overall features v of the facial expression video data by using a hierarchical video encoder;

[0009] Step 3: similarity calculation is performed on the overall feature v and the text features obtained using the relevant AU descriptor, to obtain the emotional state of the target object;

[0010] Step 4: collect the electroencephalogram signal of the target object in the remote learning environment, and input the electroencephalogram signal into the electroencephalogram signal model MDSM of the gamified learning state extractor to obtain the physiological state of the target object;

[0011] Step 5: evaluate the learning state of the target object according to the emotional state and physiological state of the target object obtained by the gamified learning state extractor, and generate different intervention measure information according to the evaluation result.

[0012] According to the game-based learning adaptive adjustment method based on the emotional and physiological dual network provided by the application, the step 1 comprises:

[0013] Step 1.1: the RGB camera is arranged on the left and right sides of the target object in the remote learning environment respectively, the RGB camera is started, and multi-angle video data V RGB of the target object is obtained, and the video collection time length is in units of class time;

[0014] Step 1.2: the collected video data V RGB is divided for a fixed time length of 3 seconds, and a group of image sequences SEQ RGB of size 224x224x3 is obtained, SEQ RGB ={IMG RGB1 ,IMG RGB2 ,…,IMG RGB16};

[0015] Step 1.3: each image frame in the sequence is segmented to obtain a plurality of non-overlapping image block sequences X={x1,x2,…,x 16} of size 16x16x3, and the image block sequences are mapped to a specific dimension embedding vector space.

[0016] According to the game-based learning adaptive adjustment method based on the emotional and physiological dual network provided by the application, the step 2 specifically comprises:

[0017] Step 2.1: a 1-D position encoding is used to generate a position encoding vector to mark the position information of each image block, and a learnable class Token is added and input into the Transformer to obtain frame-level features through a multi-head self-attention mechanism;

[0018] Step 2.2: a cross-frame fusion attention module is used to fuse the frame-level features to obtain segment-level features S={s1,s2,…,s k};

[0019] Step 2.3: Embed fragment-level features and their temporal relative positions into e tem The input is processed by a multi-head self-attention mechanism, and then processed by a feedforward neural network and average pooling to obtain video-level features v.

[0020] The gamified learning adaptive regulation method based on an emotion-physiology dual network provided by the present invention is characterized in that step 3 specifically includes:

[0021] Step 3.1: Construct text prompts c = c1, c2, ..., cK using relevant AUS descriptors, and generate corresponding text feature vectors through the cross-modal fusion module CMAM;

[0022] Step 3.2: Calculate the cosine similarity a = {a1, a2, ..., a...} between the fragment-level feature vector s and the text feature vector C. k}, erase noisy frames with similarity below a preset value, leaving the remaining segments To form new local features after denoising;

[0023] Step 3.3: Calculate the cosine similarity between the video-level feature vector v and the text feature vector c to obtain the k-dimensional global prediction result p. glo ;

[0024] Step 3.3.1: Using the video-level feature vector v and the denoised local feature vector... Cosine similarity is used as a weight to calculate weighted local features. Then, the cosine similarity is calculated between the text feature vector c and the feature vector c to obtain the local prediction result p. loc ;

[0025] Step 3.4: Calculate p respectively loc and p glo Cross-entropy loss L loc and L glo and the divergence L between them kld Finally, the consistency loss L is obtained by summing the results according to a certain ratio, and the emotional state of the target object is finally classified.

[0026] According to the gamified learning adaptive regulation method based on the emotional-physiological dual network provided by the present invention, step 4 specifically includes:

[0027] Step 4.1: Divide the EEG signals into 3-second windows, and further break them down into 0.5-second segments;

[0028] Step 4.2: The preprocessed EEG signal is used as the input to the MDSM and divided into several independent patches. By masking 50% of the temporal patch and 80% of the channel patch, it is further divided into masked parts. Non-mask portion

[0029] Step 4.3: Spatial information integration of the Patch from Step 4.2 using the encoder, and generating the corresponding output feature enc h ;

[0030] Step 4.4: Taking the output feature enc h and the temporal position information pos h as input, feeding into the predictor to generate the predicted feature pred t ;

[0031] Step 4.5: Taking the output feature enc h of the masked part and the predicted feature pred h of the unmasked part, as well as the temporal position pos h as input, feeding into the reconstructor to generate the reconstructed Patch res u,t ;

[0032] Step 4.6: Connecting the multi-classification head through the encoder and the adaptive spatial filter for aligning the channels between the electroencephalogram signal and the model, as well as the linear layer for mapping the features to the output logits of the last layer of the network to classify the student's physiological condition.

[0033] According to the game-based learning adaptive adjustment method based on the emotional and physiological dual network provided by the application, the step 5 specifically comprises:

[0034] Step 5.1: According to the output result of the game-based learning state extractor, a weight is assigned to the expression category extracted from the facial expression video data to represent the learning interest and acceptance of the student in the current game-based learning scenario;

[0035] Step 5.2: A weight is assigned to the different physiological conditions obtained in the electroencephalogram signal model of the game-based learning state extractor to represent the concentration and fatigue degree of the target object in the learning process at this stage;

[0036] Step 5.3: The comprehensive evaluation value S = aE + bP is obtained by weighted summation according to the emotional condition evaluation value E and the physiological condition evaluation value P; wherein a and b are weight coefficients for adjusting the proportion of emotional condition and physiological condition in the comprehensive evaluation;

[0037] Step 5.4: Different intervention measures are implemented on the target object in the remote learning environment according to the comprehensive state evaluation value S of the student.

[0038] ​​According to the application, the electroencephalogram signal is collected in real time by using an electrode cap, and the target object is an educated student.

[0039] In a second aspect, the application further provides a game-based learning adaptive adjustment device based on an emotional and physiological dual network, comprising:

[0040] A data acquisition and processing module is configured to acquire facial expression video data of a target object in a remote learning environment, and extract frame image sequences from the acquired video resources at a preset fixed time interval, and divide the frame image sequences into image blocks in units of frames.

[0041] A video feature extraction module is configured to input the divided image blocks into an expression recognition model ITFM of a game-based learning state extractor, and obtain overall features v of the facial expression video data by using a hierarchical video encoder.

[0042] An emotional analysis module is configured to perform similarity calculation on the overall features v and text features obtained by using relevant AUs descriptors, and obtain emotional conditions of the target object.

[0043] A physiological condition analysis module is configured to acquire electroencephalogram signals of the target object in the remote learning environment, and input the electroencephalogram signals into an electroencephalogram signal model MDSM of the game-based learning state extractor to obtain physiological conditions of the target object.

[0044] An adjustment strategy generation module is configured to evaluate learning conditions of the target object according to the emotional conditions and the physiological conditions of the target object obtained by the game-based learning state extractor, and generate different intervention measure information according to the evaluation results.

[0045] In a third aspect, the application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the game-based learning adaptive adjustment method based on an emotional and physiological dual network according to any one of the above aspects.

[0046] In a fourth aspect, the application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the game-based learning adaptive adjustment method based on an emotional and physiological dual network according to any one of the above aspects.

[0047] The game-based learning adaptive adjustment method and device based on an emotional and physiological dual network provided by the application have the following advantages compared with the prior art:

[0048] (1) The game-based learning adaptability adjustment method provided by the application can accurately perceive the self-adaptive learning effect of learners in a game-based learning environment, and provide personalized adjustment strategies and feedback suggestions, thereby helping teachers and learners improve the game-based learning effect in a remote learning environment.

[0049] (2) The expression recognition model ITFM uses a visual text-based method, uses a hierarchical video encoder to model the short-term and long-term relationships of facial expression features in the learning video of the learner, and uses prior relationship to align the facial expression and AUs to construct a text prompt, and further enhances the text prompt through video content.

[0050] (3) The electroencephalogram (EEG) signal analysis model MDSM first divides the EEG signal into a mask part and a non-mask part and performs local space-time embedding, then uses a dual self-supervised learning method of space-time representation alignment and mask reconstruction to strengthen the protection of the model, and finally uses a linear detection method to process the downstream task.

[0051] (4) The game-based adaptive learning adjustment method can accurately evaluate the learning state of learners in a remote learning environment and take different intervention measures. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0053] Figure 1 is one of the flowcharts of the game-based learning adaptability adjustment method based on the emotional and physiological dual network provided by the application;

[0054] Figure 2 is the second flowchart of the game-based learning adaptability adjustment method based on the emotional and physiological dual network provided by the application;

[0055] Figure 3 is a data acquisition schematic diagram in a remote learning environment provided by the application;

[0056] Figure 4 is a network schematic diagram of the learner expression recognition model of the embodiment of the application;

[0057] Figure 5 is a network schematic diagram of the learner electroencephalogram signal analysis model of the embodiment of the application;

[0058] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0061] The following is combined with Figures 1-6 This invention describes the gamified learning adaptive regulation method and apparatus based on an emotion-physiology dual network provided in the embodiments of the present invention.

[0062] Figure 1 This is one of the flowcharts illustrating the gamified learning adaptive regulation method based on an emotion-physiology dual network provided by this invention. Figure 2 This is the second flowchart of the gamified learning adaptive regulation method based on the emotional-physiological dual network provided by this invention. The following is a combination of... Figure 1 and Figure 2 The technical solution of the present invention will be described as follows:

[0063] Step 1: Acquire facial expression video data of the target object in the remote learning environment, and extract frame image sequences from the acquired video resources according to a preset fixed time interval, and divide them into image blocks in units of frames.

[0064] The target group can be students receiving education; such as Figure 3 As shown, learners are engaging in gamified learning in a remote environment. In this scenario, an RGB camera and an EEG device are used to collect the learners' facial expression features and EEG physiological characteristics, respectively. The RGB and EEG information collected in this remote learning environment provides important data sources for facial expression recognition networks and EEG signal analysis networks based on visual language models.

[0065] Optionally, step 1 includes:

[0066] Step 1.1: Place the RGB cameras on the left and right sides of the student in the remote learning environment, respectively, and turn on the RGB cameras to obtain multi-angle video data V of the student in the gamified learning scenario. RGB The video recording duration is measured in class period units;

[0067] Step 1.2: Process the collected student learning video data V at a fixed duration of 3 seconds. RGB The image was divided into segments, resulting in a sequence SEQ of size 224×224×3. RGB ={IMG RGB1 IMG RGB2 ,…,IMG RGB16};

[0068] Step 1.3: Feed the segmented image sequence into the ITFM (Image-Text Fusion Module) of the gamified learning state extractor. First, segment each image frame in the sequence to obtain several non-overlapping image block sequences X = {x1, x2, ..., x...} of size 16×16×3. 16}, and map it to an embedding vector space of a specific dimension.

[0069] Step 2: Input the segmented image blocks into the expression recognition model ITFM of the gamified learning state extractor, and obtain the overall features v of the facial expression video data by using a hierarchical video encoder.

[0070] Optionally, the present invention adds corresponding positional encoding and an additional learnable vector to the image blocks obtained after segmentation and inputs them into the Transformer to learn the image features of each frame in the student gamified learning scenario, i.e., frame-level features. These features are then sent to the Cross-frame Attention Module (CMAM) to extract local features of some learning videos, referred to here as segment-level features. Finally, these features are sent to the Multi-snippet Integration Module (MSIM) to further fuse the local segment features to obtain video-level features.

[0071] like Figure 4 As shown in this embodiment, the visual text expression recognition model ITFM includes a cross-frame fusion attention module (CFAM) and a multi-segment ensemble module (MSIM). The specific training process is as follows:

[0072] Step 2.1: Use 1-D positional encoding to generate positional encoding vectors using sine and cosine functions:

[0073]

[0074] where pos denotes position, i denotes dimension, d model denotes the dimension of embedding vectors. The value of the position encoding vector is calculated to mark the position information between each image block, in addition, a global feature vector is additionally added to the above input and merged into the Transformer;

[0075] Step 2.1.1: The above vectors are sequentially input into the encoder, and the relationship between the image blocks is obtained by adopting the multi-head self-attention mechanism, and the specific operation is as follows:

[0076] MultiHead(Q,K,V)=Concat(head1,…,head hnum )W O

[0077]

[0078] where hnum is the number of heads, are the projection matrices of the u-th head query, key and value respectively, W O is the output projection matrix. And through the feedforward neural network to enhance its features, while combining residual connection and normalization to ensure network stability:

[0079] FFN(x)=W2δ(W1x+b1)W2+b2

[0080]

[0081] where x is the input vector, W1, W2 are weight matrices, b1, b2 are bias terms, δ(·) represents the ReLU activation function, μ and σ are the mean and standard deviation of the input vector respectively, ε is a small constant to prevent division by zero error, γ and β are learnable parameters to restore the expression ability of data. Finally, the frame-level feature

[0082] Step 2.2: In order to obtain the relevance of the local segments in the student learning video, use l-layer Transformer to form a cross-frame fusion attention module (CFAM), which is used to fuse the frame-level features z to obtain segment-level features S, including the following steps:

[0083] Step 2.2.1: At the n-th local segment of the frame The input of CFAM is represented as:

[0084]

[0085] where, represent a learnable class Token, representing the relative time index in the local segment, e spa representing the spatial position encoding.

[0086] Step 2.2.2: In the 1-th layer of CFAM, the Interaction Token is obtained by linear transformation is used to capture the spatio-temporal relationship within the frame. The specific formula is represented as:

[0087]

[0088] where LN(·) represents layer normalization, and MHSA(·) represents multi-headed self-attention.

[0089] Step 2.2.3: The above interaction Token is learned together with its corresponding visual Token and class Token, and and interaction Token are fused to represent the spatio-temporal relationship within the frame. The specific formula can be represented as:

[0090]

[0091] where ([·,·]) represents the feature concatenation of frame Token and interaction Token.

[0092] Step 2.2.4: Finally, the class Token corresponding to each frame output in the above frame interaction Transformer is extracted and composed into a frame-level feature sequence Then it is respectively input into the cross-frame fusion module to obtain the segment-level feature S = {s1, s2, …, s k}. The specific fusion process can be represented as:

[0093]

[0094] Step 2.3: To obtain the video-level feature representation of the student throughout the learning process, the above result is input into MHSA (Multi-Headed Self Attention) together with the time relative position embedding e tem to capture the dependency of the data, and further processed through the feedforward network, and finally obtain the video-level feature v through AvgPool. The specific processing process is as follows:

[0095] v = AvgPool(FFN(MHSA(s + e tem )0),

[0096] where AvgPool(·) denotes average pooling and FFN(·) denotes feed-forward neural network.

[0097] Step 3: Calculate the similarity between the overall feature v and the text feature obtained using the relevant AU descriptor to obtain the emotional state of the target object.

[0098] Optionally, step 3 includes the following steps:

[0099] Step 3.1: Construct a text prompt c = [c1, c2, …, c K ] using the relevant AU descriptor and input it into the CMAM to obtain the corresponding text feature vector. The specific processing method is as follows:

[0100]

[0101] where γ represents a learnable parameter initialized to 0.1, z t represents the final output of the t-th frame obtained from the CFAM. In addition, in the MHSA, the text prompt c is used as the query, the key and the value.

[0102] Step 3.2: To reduce the interference of noise frames, a noise reduction module is introduced. Calculate the cosine similarity a = {a1, a2, …, a k} between the feature vector S of the student learning video segment and the text feature vector C. Use the normalized similarity score a * to identify noise frames in the collected video and erase them. Discard the frames with similarity lower than the preset value, and the remaining segments constitute the new local features after denoising.

[0103] Step 3.3: Since directly erasing noise frames may cause inconsistencies between local segment features and overall video features, a consistency regularization loss function is introduced. Calculate the cosine similarity between the video-level feature vector v and the text feature vector c to obtain the k-dimensional global prediction result p glo .

[0104] Step 3.3.1: Calculate the cosine similarity between the video-level feature vector v and the denoised local feature vector as the weight to obtain the weighted local feature Then calculate the cosine similarity between the weighted local feature and the text feature vector c to obtain the local prediction result p loc .

[0105] Step 3.4: Calculate the cross-entropy loss L loc and L glo between p loc and p glo and the divergence Lkld Finally, the consistency loss L is added in a certain proportion to obtain the consistency loss L, and the student learning state is finally classified.

[0106] Step 4: Collect the electroencephalogram signal of the target object in the remote learning environment, and input the electroencephalogram signal into the electroencephalogram signal model MDSM (Masked Dynamic Spatial Module) of the gamified learning state extractor to obtain the physiological condition of the target object.

[0107] As shown in Figure 5 , in this embodiment, the electroencephalogram signal analysis model MDSM first divides the input electroencephalogram signal x ∈ R M×T (where M represents the number of channels, and T represents the time point) into several independent small blocks p m,h , and embeds each patch as a token by local spatiotemporal embedding, denoted as token m,h , and then processes them into mask part and non-mask part Then, a dual self-supervised learning method using spatiotemporal representation and mask-based reconstruction is used to protect the model, and finally a linear probing method is used to process the downstream task.

[0108] Step 4.1: Preprocess the collected student electroencephalogram signal. Set a 3-second time window to divide the continuous electroencephalogram signal into several non-overlapping independent samples α = {α1, α2, …, α N}, and then further divide them into several 0.5-second segments, and regard each segment as a sample frame for subsequent feature extraction.

[0109] Step 4.2: Use the local spatiotemporal embedding method to patch and embed the above preprocessed EEG signal x in the spatiotemporal dimension, and send it to the encoder. First, the EEG signal is divided into equal-sized patches in the spatiotemporal dimension, denoted as p m,h , m ∈ {1, 2, …, M}, n ∈ {1, 2, …, n}:

[0110] p m,h = x m,(h-1)d:hd

[0111] where d represents the time length of the patch, and N = T / d represents the number of time patches.

[0112] Next, combine the channel embedding information to linearly embed the patch, and by constructing a mapping from the channel name to the channel embedding, the channels of the electroencephalogram data are corresponded to the channels input to the model, so that the model can adapt to multiple data sets and improve the adaptive ability of the channels. The specific formula is as follows:

[0113]

[0114] where Embed(p m,h ) denotes the Patch after linear embedding. Then, the Patch features are predicted by each other using a self-supervised learning based method. 50% of the time and 80% of the channel Patches are masked, and further divided into mask parts and non-mask parts

[0115] Step 4.3: The predicted features are aligned with the output of the momentum encoder using a spatio-temporal alignment method to enhance the robustness of the extracted features and ensure the stability of the output global features. First, all tokens from the mask parts m,h are used as the input of the encoder, and the corresponding output features enc h are generated:

[0116]

[0117] Step 4.4: Then, the above mask part output features enc h and the temporal position information pos h obtained by using the rotation position embedding method are used as input to predict the complete encoding features. At the same time, in order to generate the prediction features of the non-mask parts, a learnable query Token is used, and more correlation information between different Tokens is obtained by using self-supervised training. The specific formula is as follows:

[0118]

[0119] Step 4.5: The output features enc h from the mask parts and the student EEG prediction features pred h from the non-mask parts, as well as the temporal position pos h are input into the reconstructor to generate the reconstructed Patches rec u,t .

[0120]

[0121] where the mask-based reconstruction is implemented using a mean square error based reconstruction loss, and the complete pre-training loss is constructed by adding the sum of and The specific formula is as follows:

[0122]

[0123] Step 4.6: In the downstream task, align the channels between the electroencephalogram signal and the model through a 1x1 spatial adaptive filter, and map the features to the linear layer of the output logits of the last layer of the network, connected to the linear classification head to classify the student's physiological condition.

[0124] Step 5: According to the emotional state and physiological condition of the target object obtained by the gamified learning state extractor, the learning state of the target object is evaluated, and different intervention information is generated according to the evaluation result.

[0125] Step 5.1: At the level of student emotional state, a weight is assigned to each emotional state obtained by the extractor to represent the student's learning interest and acceptance in the current gamified learning scenario. For example, when the output result of the ITFM is "happy", it indicates that the student is in an active state in the current gamified learning scenario, and the student shows a high degree of enthusiasm and strong interest in learning in the learning process, which helps to improve the student's learning efficiency, and the weight value is +1; when the output result of the ITFM is "depressed" or "confused" and other emotions, it indicates that the student is in a passive state in the current gamified learning scenario, and the student may lack interest in learning or feel a certain degree of pressure, which is not conducive to the student to continue the current learning, and the weight is negative.

[0126] Step 5.2: At the level of physiological condition, a weight is assigned to the different physiological conditions of the student obtained in the electroencephalogram model, which represents the student's learning concentration and fatigue level in this stage of learning. For example, when the output result of the MDSM is "high concentration", it indicates that the student can maintain attention and have in-depth understanding and thinking of the learning content, and can effectively handle the current learning content, so the weight is set to a positive value; when the model output result is "distraction", it indicates that the student may be mentally exhausted and unable to concentrate in the current gamified learning stage, and a negative weight is assigned.

[0127] Step 5.3: The comprehensive evaluation value S = aE + bP is obtained by weighted summation of the above emotional state evaluation value E and physiological condition evaluation value P. Where a and b are weight coefficients used to adjust the proportion of emotional state and physiological condition in the comprehensive evaluation; optionally, the gamified adaptive adjustment learning state perception index system is shown in Table 1:

[0128]

[0129] Step 5.4: The educator takes different learning intervention measures for the student in the remote learning environment according to the student's comprehensive state evaluation value S, and the specific measures are as follows:

[0130] When T1 < S (T1 is a set positive threshold value), it indicates that the student is in a state of passionate concentration, and the current learning activity can be continued without special intervention from the teacher.

[0131] When T2 < S ≤ T1 (T2 is a set neutral threshold value), it indicates that the student may be in a state of passionate fatigue or forced concentration, and the teacher can appropriately reduce the learning intensity of the student.

[0132] When S ≤ T2, it indicates that the student is in a state of fatigue and boredom, and immediate intervention from the teacher is needed to reduce the learning difficulty or type of the student.

[0133] On the other hand, the present application also provides a game-based learning adaptive adjustment device based on an emotional and physiological dual network, comprising:

[0134] A data acquisition and processing module is configured to acquire facial expression video data of a target object in a remote learning environment, and extract frame image sequences from the acquired video resources at a preset fixed time interval.

[0135] A video feature extraction module is configured to input the divided image blocks into an expression recognition model ITFM of a game-based learning state extractor, and obtain overall features v of the facial expression video data by using a hierarchical video encoder.

[0136] An emotional analysis module is configured to perform similarity calculation on the overall features v and text features obtained using related AUs descriptors to obtain the emotional state of the target object.

[0137] A physiological condition analysis module is configured to acquire electroencephalogram signals of the target object in the remote learning environment, and input the electroencephalogram signals into an electroencephalogram signal model MDSM of the game-based learning state extractor to obtain the physiological condition of the target object.

[0138] An adjustment strategy generation module is configured to evaluate the learning condition of the target object according to the emotional state and the physiological condition of the target object obtained by the game-based learning state extractor, and generate different intervention measure information according to the evaluation result.

[0139] It should be noted that the game-based learning adaptive adjustment device based on the emotional and physiological dual network provided by the embodiments of the present application can execute the game-based learning adaptive adjustment method based on the emotional and physiological dual network described in any of the above embodiments when it is actually run, and the present embodiment will not be repeated.

[0140] Figure 6 is a structural schematic diagram of an electronic device provided by the present application, like Figure 6As shown, the electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logic instruction in the memory 630 to execute the game-based learning adaptive adjustment method based on the emotional physiological dual network.

[0141] In addition, the logic instruction in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0142] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the game-based learning adaptive adjustment method based on the emotional physiological dual network provided by each of the above embodiments.

[0143] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the game-based learning adaptive adjustment method based on the emotional physiological dual network provided by each of the above embodiments.

[0144] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0145] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A gamified learning adaptive regulation method based on an emotion-physiology dual network, characterized in that, include: Step 1: Acquire facial expression video data of the target object in the remote learning environment, and extract the acquired video resources into frame image sequences according to a preset fixed time interval, and divide them into image blocks in units of frames; Step 2: Input the segmented image blocks into the expression recognition model ITFM of the gamified learning state extractor, and obtain the overall features v of the facial expression video data by using a hierarchical video encoder; Step 3: Calculate the similarity between the overall feature v and the text features obtained using the relevant AUS descriptors to obtain the emotional state of the target object; Step 4: Collect the EEG signal of the target object in the remote learning environment, and input the EEG signal into the MDSM EEG signal model of the gamified learning state extractor to obtain the physiological status of the target object; Step 5: Assess the target's learning status based on the emotional and physiological status of the target subject obtained by the gamified learning status extractor, and generate different intervention measures based on the assessment results.

2. The gamified learning adaptive regulation method based on an emotion-physiology dual network according to claim 1, characterized in that, Step 1 includes: Step 1.1: Place the RGB cameras on the left and right sides of the target object in the remote learning environment, respectively, and turn on the RGB cameras to obtain multi-angle video data V of the target object. RGB The video recording duration is measured in class period units; Step 1.2: Process the acquired video data V for a fixed duration of 3 seconds. RGB The image was divided into segments, resulting in a sequence SEQ of size 224×224×3. RGB ={IMG RGB1 IMG RGB2 ,…,IMG RGB16 }; Step 1.3: Segment each image frame in the sequence to obtain a sequence of several non-overlapping image blocks of size 16×16×3, X={x1,x2,…,x…} 16 } and map it to an embedding vector space of a specific dimension.

3. The gamified learning adaptive regulation method based on an emotion-physiology dual network according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1: Generate positional encoding vectors using 1-D positional encoding, mark the positional information of each image patch, add learnable class tokens, and input them into the Transformer to obtain frame-level features through a multi-head self-attention mechanism; Step 2.2: Use the cross-frame fusion attention module to fuse frame-level features to obtain segment-level features S = {s1, s2, ..., s}. k }; Step 2.3: Embed fragment-level features and their temporal relative positions into e tem The input is processed by a multi-head self-attention mechanism, and then processed by a feedforward neural network and average pooling to obtain video-level features v.

4. The gamified learning adaptive regulation method based on an emotion-physiology dual network according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Construct text prompts c = c1, c2, ..., cK using relevant AUS descriptors, and generate corresponding text feature vectors through the cross-modal fusion module CMAM; Step 3.2: Calculate the cosine similarity a = {a1, a2, ..., a...} between the fragment-level feature vector S and the text feature vector C. k }, erase noisy frames with similarity below a preset value, leaving the remaining segments To form new local features after denoising; Step 3.3: Calculate the cosine similarity between the video-level feature vector v and the text feature vector c to obtain the k-dimensional global prediction result p. glo ; Step 3.3.1: Using the video-level feature vector v and the denoised local feature vector... Cosine similarity is used as a weight to calculate weighted local features. Then, the cosine similarity is calculated between the text feature vector c and the feature vector c to obtain the local prediction result p. loc ; Step 3.4: Calculate p respectively loc and p glo Cross-entropy loss L loc and L glo and the divergence L between them kld Finally, the consistency loss L is obtained by summing the results according to a certain ratio, and the emotional state of the target object is finally classified.

5. The gamified learning adaptive regulation method based on an emotion-physiology dual network according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1: Divide the EEG signals into 3-second windows, and further break them down into 0.5-second segments; Step 4.2: The preprocessed EEG signal is used as the input to the MDSM and divided into several independent patches. By masking 50% of the temporal patch and 80% of the channel patch, it is further divided into masked parts. Non-mask portion Step 4.3: Use the encoder to process data from... The patch integrates spatial information and generates corresponding output features enc. h ; Step 4.4: Output features enc h and time and location information pos h As input, it is fed into the predictor to generate a product belonging to... Predictive features t ; Step 4.5: Convert the output features of the masked portion into enc h Predicted features of the non-masked portion h and time location pos h As input, it is fed into the reconstructor to generate the reconstructed Patchrec. u,t ; Step 4.6: Classify students' physiological conditions by connecting a multi-class head through an encoder and an adaptive spatial filter for aligning the channels between the EEG signal and the model, as well as a linear layer that maps features to the output logits of the last layer of the network.

6. The gamified learning adaptive regulation method based on an emotion-physiology dual network according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Based on the output of the gamified learning state extractor, assign a weight to the facial expression category extracted from the facial expression video data to represent the student's learning interest and acceptance level in the current gamified learning scenario; Step 5.2: Assign a weight to the different physiological states obtained from the EEG signal model of the gamified learning state extractor to represent the target object's focus and fatigue level during this learning stage; Step 5.3: The comprehensive assessment value S = αE + βP is obtained by weighted summation of the emotional status assessment value E and the physical status assessment value P; where α and β are weighting coefficients used to adjust the proportion of emotional status and physical status in the comprehensive assessment. Step 5.4: Implement different intervention measures for the target group in the remote learning environment based on the student's comprehensive status assessment value S.

7. The gamified learning adaptive regulation method based on an emotion-physiology dual network according to claim 1, characterized in that, The electroencephalogram (EEG) signals were acquired in real time using an electrode cap, and the target subjects were students receiving education.

8. A gamified learning adaptive regulation device based on an emotion-physiology dual network, characterized in that, include: The data acquisition and processing module is used to acquire facial expression video data of the target object in the remote learning environment, and extract the acquired video resources into frame image sequences according to a preset fixed time interval, and divide them into image blocks in units of frames. The video feature extraction module is used to input the segmented image blocks into the expression recognition model ITFM of the gamified learning state extractor, and obtain the overall features v of the facial expression video data by using a hierarchical video encoder; The sentiment analysis module is used to calculate the similarity between the overall feature v and the text features obtained using relevant AUS descriptors to obtain the sentiment status of the target object. The physiological condition analysis module is used to collect the electroencephalogram (EEG) signals of the target object in the remote learning environment, and input the EEG signals into the MDSM (Mean Diagnostic and Analytical Brain Signal Model) of the gamified learning state extractor to obtain the physiological condition of the target object. The adjustment strategy generation module is used to assess the learning status of the target object based on the emotional and physiological status of the target object obtained by the gamified learning state extractor, and generate different intervention measures based on the assessment results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the gamified learning adaptive regulation method based on the emotional-physiological binary network as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the gamified learning adaptive regulation method based on the emotional-physiological binary network as described in any one of claims 1 to 7.

Images