A method for acquiring learning engagement data based on multimodal fusion network

Abstract

The present invention discloses a method for acquiring learning engagement data based on a multimodal fusion network. The method uses the facial image, head rotation image and EEG information of the learning subject to be monitored for the evaluation of learning engagement at the same time, and adopts a novel multimodal network model CGVNet to improve the accuracy of the learning engagement data monitoring of the learning subject to be monitored. The acquired head rotation angle feature vector, gaze direction feature vector and EEG information are fused and classified, thereby overcoming the limitation of the single monitoring dimension in the traditional learning engagement monitoring method, reducing the problems of missed detection and false detection, which is of great significance to the improvement of learning quality and auxiliary teaching.

Description

Technical Field

[0001] The present invention relates to the field of intelligent human-computer interaction technology, and more specifically, to a method for acquiring learning engagement data based on a multimodal fusion network. Background Art

[0002] With the rapid development of educational informatization, artificial intelligence technology has been increasingly used in the process of education and teaching. In recent years, online distance learning methods have been widely adopted. Since there is no close supervision by teachers in online learning, the autonomous learning ability of the learning subject has declined. However, recently, with the development of offline classroom learning, how to implement remote supervision or self-supervision to evaluate the learning engagement of the learning subject has become an urgent problem to be solved. It is of great significance to use a real-time classroom learning engagement monitoring method to assist learners in classroom learning, thereby improving the learning efficiency of learners in classroom learning.

[0003] However, the monitoring dimensions in traditional learning engagement monitoring methods are relatively single, and the recognition accuracy of the corresponding data of the monitoring dimensions is not very high, which makes it impossible to evaluate the learning engagement of learning subjects from more dimensions, more levels, and higher accuracy. Summary of the invention

[0004] In response to at least one defect or improvement need in the prior art, the present invention provides a method for acquiring learning engagement data based on a multimodal fusion network, which is used to overcome the technical defects of traditional learning engagement monitoring methods, such as the single monitoring dimension and the low recognition accuracy of the corresponding data of the monitoring dimension, which makes it impossible to evaluate the learning engagement of the learning subject from more dimensions, more levels and higher accuracy.

[0005] To achieve the above object, in a first aspect, the present invention provides a method for acquiring learning engagement data based on a multimodal fusion network, comprising the steps of:

[0006] Acquiring a facial image, a head rotation image, and EEG information of a subject to be monitored based on a device system including a wearable device;

[0007] Preprocessing the acquired facial images, head rotation images and EEG information to make the related images correspond to the EEG information of the corresponding timestamps;

[0008] Input the preprocessed facial image, head rotation image and EEG information into the pre-trained multimodal network model to obtain the current estimated gaze direction, estimated head rotation angle and EEG curve of the subject to be monitored;

[0009] Based on the estimated gaze direction, the estimated head rotation angle and the EEG curve, the current learning engagement data of the learning subject to be monitored is obtained.

[0010] Furthermore, the preprocessed facial image, head rotation image and EEG information are input into a pre-trained multimodal network model to obtain the current estimated gaze direction, estimated head rotation angle and EEG curve of the subject to be monitored, specifically including the following steps:

[0011] Inputting the facial image into the gaze direction estimation model in the multimodal network model for feature extraction to obtain the corresponding gaze direction feature vector;

[0012] The head rotation image is input into the head rotation angle estimation model in the multimodal network model for feature extraction to obtain the corresponding head rotation angle feature vector;

[0013] The gaze direction feature vector and the head rotation angle feature vector are combined with the EEG information for fusion and classification to obtain the current estimated gaze direction, estimated head rotation angle and EEG curve of the subject to be monitored.

[0014] Furthermore, the input layer of the head rotation angle estimation model inputs the head rotation image, the feature extraction layer includes three convolutional layers, three pooling layers, two fully connected layers and three drop-out layers, and the output layer outputs the corresponding head rotation angle feature vector; the head rotation angle estimation model uses Nesterov's accelerated gradient descent method for parameter optimization.

[0015] Furthermore, for the facial image input to the gaze direction estimation model, the GAN-based semantic image inpainting method is first used to inpaint the area covered by the wearable device.

[0016] Furthermore, the use of the GAN-based semantic image restoration method to repair the area covered by the wearable device specifically includes:

[0017] Using the least squares loss method, the training goal of GAN is to minimize L GAN (D) and L GAN (G), where

[0018] L GAN (D)=E x～p [(D(x)-1) 2 ]+E x～pn [(D(G(x))) 2 ],

[0019] L GAN (G) = E x～pn [(D(G(z))-1) 2],

[0020] Among them, D i is a discriminator, D(x i ) is represented as the input image x i is the probability of the true sample; G i represents the generator, which is used to output the synthetic image; L GAN (D) measures the realism of the image in D, L GAN (G) measures the realism of the image produced by G, which is considered as perceptual loss. The formula for perceptual loss is:

[0021] L perception (z) = [D (G (z)) - 1] 2 ,

[0022] The context loss is measured based on the difference between the real image x and the generated image G(z) in the non-masked area. The formula of context loss is:

[0023] L context (z|M,x)=|M1⊙x-M1⊙G(z)|,

[0024] Among them, ⊙ is the element-wise product, M1 is the complement of M, and M is the area that should not be repaired;

[0025] Generating the best image for inpainting is equivalent to finding the best The value minimizes the combination of perceptual loss and contextual loss as follows:

[0026]

[0027] Among them, λ is the weighting parameter. After finding z, the repaired image can be generated by the following formula:

[0028] X inpainted =M1⊙x-M1⊙G(z),

[0029] Then, a Poisson mixture is applied to X inpainted , to generate the final inpainted image with seamless boundaries between the inpainted and non-inpainted areas.

[0030] Furthermore, the facial image is input into the gaze direction estimation model in the multimodal network model for feature extraction, specifically including:

[0031] Use the key point detection module to extract the positions of 5 facial key points to generate eye images;

[0032] The gaze direction estimation model is trained based on the annotated gaze labels.

[0033] Furthermore, the learning engagement data includes EEG curve deviation data; the EEG curve deviation data is obtained by calculating the degree of deviation of the EEG curve from the EEG baseline curve; the EEG baseline curve is a curve formed by EEG information generated by a learning subject whose learning engagement reaches the standard; the EEG curve deviation data can be used to analyze the learning engagement of the learning subject to be monitored.

[0034] Furthermore, when the acquired facial image, head rotation image and EEG information are preprocessed, it also includes:

[0035] Eliminate data that do not meet the quality requirements, and retain data that meet the quality requirements for preprocessing;

[0036] The preprocessed data is subjected to data enhancement operations including one or more methods of distortion, adding noise, and reducing resolution.

[0037] In a second aspect, the present invention provides an electronic device comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit is enabled to perform the steps of any one of the above-mentioned methods.

[0038] In a third aspect, the present invention provides a storage medium storing a computer program executable by an access authentication device, wherein when the computer program runs on the access authentication device, the access authentication device is enabled to perform the steps of any one of the above methods.

[0039] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0040] (1) The present invention discloses a method for acquiring learning engagement data based on a multimodal fusion network, which uses the facial image, head rotation image and EEG information of the subject to be monitored for the evaluation of learning engagement at the same time, and adopts a novel multimodal network model CGVNet to improve the accuracy of the learning engagement data monitoring of the subject to be monitored. The acquired head rotation angle feature vector, gaze direction feature vector and EEG information are fused and classified, thereby overcoming the limitation of the single monitoring dimension in the traditional learning engagement monitoring method, reducing the problems of missed detection and false detection, which is of great significance to the improvement of learning quality and auxiliary teaching.

[0041] (2) The present invention applies the GAN-based semantic image restoration method to the area covered by the wearable device, and bridges the gap between the training image and the test image by removing the abruptness of the wearable device, thereby improving the accuracy of the network and making the trained network model more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a method for acquiring learning engagement data based on a multimodal fusion network provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a scenario in which a device system including a wearable device provided in an embodiment of the present invention collects data such as a facial image, a head rotation image, and EEG information of a subject to be monitored;

[0045] Figure 3 A schematic diagram of the overall network structure of a multimodal network model provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a learning engagement evaluation mechanism provided by an embodiment of the present invention;

[0047] Figure 5 A block diagram of an electronic device suitable for implementing the method described above provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] The terms "including" or "having" and any variations thereof in the specification, claims or drawings of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0050] After long-term research, it is found that learning engagement can be reflected in the learner's head rotation, gaze direction, and EEG information. When the engagement is concentrated, the head should be facing the learning device, the gaze should be focused, and the EEG information should be relatively active. However, in the process of classroom learning, the learner may listen quietly or lower his head to read. It is difficult to accurately judge the current state of learning engagement of the learner by only relying on unilateral information capture, resulting in a lower accuracy of the monitoring results.

[0051] The monitoring of learning engagement can start from the gaze direction, head rotation angle and EEG information of the learning subject during the learning process. In a broad sense, gaze estimation refers to the research on eyeballs, eye movements, gaze, etc. Generally speaking, gaze estimation methods can be divided into two categories: model-based methods and appearance-based methods. The basic idea of ​​the model-based method is to estimate the gaze direction based on features such as corneal reflection of the eye, combined with 3D eyeball prior knowledge; while the appearance-based method directly extracts the visual features of the eye, trains a regression model, and learns a model that maps the appearance to the gaze direction, thereby performing gaze estimation. Through multiple experimental comparative analyses, the model-based method has a higher accuracy, but the requirements for image quality and resolution are also higher. In order to achieve this goal, it is usually required to use specialized hardware, and the user's mobility such as head rotation is greatly restricted; while the appearance-based method performs better for low-resolution and high-noise images, but its model training requires a large amount of data. With the development of deep learning and the disclosure of a large number of data sets, appearance-based methods have received more and more attention. The present invention is based on the current status of the development of this technology, creatively combines the model-based method and the appearance-based method and applies them to the estimation of gaze direction.

[0052] With the development of wearable device technology and information processing algorithms, physiological information has been widely used to monitor the physiological state of learning subjects. The physiological information collected by wearable devices can more accurately reflect the state of the tested learning subjects, thereby improving the overall performance of the learning engagement monitoring system.

[0053] refer to Figure 1 and Figure 4 In one embodiment, a method for acquiring learning engagement data based on a multimodal fusion network mainly includes steps S1-S4:

[0054] S1. Acquire a facial image, a head rotation image, and EEG information of a subject to be monitored based on a device system including a wearable device.

[0055] S2. Preprocessing the acquired facial image, head rotation image and EEG information to make the related images correspond to the EEG information of the corresponding timestamps.

[0056] refer to Figure 2 The facial image, head rotation image and EEG information of the learning subject to be monitored are obtained through the RGB-D camera, motion capture camera and head-mounted gaze tracker (Eyetracking glass) in the classroom.

[0057] The head-mounted gaze tracker includes a physiological information acquisition module, a physiological information transmission module and a physiological information analysis module. The physiological information acquisition module is used to collect the EEG information of the learning subject; the physiological information transmission module synchronizes the timestamp of the physiological information collected by the physiological information acquisition module with other features; for physiological information, it is first necessary to evaluate the quality of the collected physiological information, remove data that does not meet the quality requirements, retain data that meet the quality requirements for processing, pre-process the data, and perform data enhancement by distorting, adding noise, reducing resolution, etc.; the physiological information is pre-processed and feature extracted by the physiological information analysis module, and the feature vector is input into the fusion model to obtain the EEG curve of the learning subject, and the degree of deviation of the EEG curve from the EEG baseline curve is calculated to obtain the degree of deviation; the EEG baseline curve is a curve formed by the EEG signal generated by the learning subject whose learning engagement reaches the standard; the learning engagement of the learning subject to be monitored can be analyzed and evaluated based on the deviation degree data of the EEG curve.

[0058] S3, input the pre-processed facial image, head rotation image and EEG information into the pre-trained multimodal network model to obtain the current estimated gaze direction, estimated head rotation angle and EEG curve of the subject to be monitored. Step S3 specifically includes the following steps S31-S33.

[0059] S31. Input the facial image into a gaze direction estimation model in a multimodal network model for feature extraction to obtain a corresponding gaze direction feature vector.

[0060] S32: Input the head rotation image into the head rotation angle estimation model in the multimodal network model for feature extraction to obtain the corresponding head rotation angle feature vector.

[0061] S33, the gaze direction feature vector and the head rotation angle feature vector are combined with the EEG information for fusion and classification, and the current estimated gaze direction, estimated head rotation angle and EEG curve of the monitored learning subject are obtained.

[0062] like Figure 3As shown in FIG. 1 , the multimodal network model CGVNet includes a head rotation angle estimation model and a gaze direction estimation model. The multimodal network model fuses and classifies the head rotation angle feature vector output by the head rotation angle estimation model and the gaze direction feature vector output by the gaze direction estimation model to obtain the current head rotation estimation angle and gaze estimation direction of the monitored learning subject.

[0063] In this embodiment, the head rotation image is input into the ConvNet head rotation angle estimation model for processing. The network structure (ConvNet) of the head rotation angle estimation model is as follows: Figure 3 As shown. The pre-trained convolutional network (ConvNet) model is used for head pose estimation. The input of the network is a 96×96 grayscale head rotation image, normalized to between 0 and 1. The feature extraction stage includes three convolutional layers, three pooling layers, two fully connected layers, and three drop-out layers. The output layer is a feature vector of head rotation angles, and the multimodal network model CGVNet ultimately outputs the pitch angle, yaw angle, and roll angle representing the head posture. The angle is normalized between -1 and 1. This embodiment uses the Nesterov accelerated gradient descent (NAG) method for parameter optimization, setting the momentum to 0.9 and the learning rate to 0.005.

[0064] Data enhancement plays a vital role in the training of neural networks. In the model training stage, this embodiment targets the problem of head posture estimation (head rotation angle estimation) and adopts the following data enhancement method: sample enhancement is performed on the head rotation image by rotation, translation, scale transformation, etc., so as to enhance the robustness of the model. At the same time, these operations also provide a large number of forged samples for model training.

[0065] Gaze direction estimation is achieved for facial images input to the gaze direction estimation model by using a motion capture system to measure head posture and a head-mounted gaze tracker to measure eye gaze, thereby solving the ground truth problem. Because the present invention is interested in estimating the line of sight (gaze direction) of the learning subject without using a head-mounted gaze tracker, it is important that the test image is not affected by changes in the appearance of the learning subject. Therefore, the facial image input to the gaze direction estimation model VGG is first subjected to GAN-based semantic image restoration to repair the area covered by the head-mounted gaze tracker to eliminate the abruptness caused by the head-mounted gaze tracker. This is mainly to bridge the difference between the training and test images, and then the image is input into the network model for feature extraction. The image with the head-mounted gaze tracker removed is used to train a new gaze direction estimation model, and the restoration improves the accuracy of gaze estimation. There are two conditions to be met: the restored result should look very real (perceptual loss L perception), the inpainted pixels should be well aligned with the surrounding pixels (context loss L context ).

[0066] The gaze direction feature extraction step after preprocessing in the gaze direction estimation model also includes: using a key point detection module to extract the positions of five facial key points for generating eye images. The gaze direction estimation model is trained according to the annotated gaze labels.

[0067] This embodiment trains an independent repair network for the learning subject, and lets D i represents a discriminator that classifies images x of a learning subject from a dataset not wearing a head-mounted gaze tracker i ∈R d (d = 224 × 224 × 3) as input and output a scalar that represents the probability that the input is a true sample. Let G i Denote the potential random variable z that will be sampled from the uniform noise distribution pn=U(1,1) i ∈R z (z=100) as input and output the synthesized image G i (z i )∈R d Ideally, when x i The real dataset p from the learning subject i When D i (x i )=1, when x i From G i When D i (x i )=0.

[0068] This example uses the least squares loss. The training goal of GAN is to minimize L GAN (D) and L GAN (G) where:

[0069] L GAN (D)=E x～p [(D(x)-1) 2 ]+E x～pn [(D(G(x))) 2 ],

[0070] L GAN (G) = E x～pn [(D(G(z))-1) 2 ],

[0071] Among them, L GAN (G) measures the realism of the image produced by G, which is considered as perception loss:

[0072] L perception (z) = [D (G (z)) - 1] 2 ,

[0073] The context loss is measured as the difference between the real image x and the generated image G(z) in the non-masked area:

[0074] L context (z|M,x)=|M1⊙x-M1⊙G(z)|,

[0075] Where ⊙ is the element-wise product and M1 is the complement of M (i.e. defines the area that should not be inpainted).

[0076] The latent random variable z controls the image produced by G(z). Therefore, generating the best image for inpainting is equivalent to finding the best value, which minimizes the combination of perceptual loss and contextual loss:

[0077]

[0078] where λ is a weighting parameter. After finding z, the inpainted image can be generated as follows:

[0079] X inpainted =M1⊙x-M1⊙G(z),

[0080] Then apply Poisson mixing to X inpainted Inpainting (i.e., digital image repair) is performed to generate a final repaired image with a seamless boundary between the repaired area and the non-repaired area.

[0081] Regarding the network architecture for gaze direction estimation (gaze direction estimation model), this embodiment performs hyperparameter adjustment to generate high-quality high-resolution images. The generator is set to Z-dense(25088)-(256)5d2s-(128)5d2s-(64)5d2s-(32)5d2s-(3)5d2s-X, where "(128)52s / (128)5d2s" represents a convolution / deconvolution layer with 128 output feature maps, a kernel size of 5, and a stride of 2. All internal activations use ReLU, while the output layer uses a tanh activation function. The architecture of the discriminator is: X-(16)5c2s-(32)5c2s-(64)5c2s-(128)5c2s-(256)5c2s-(512)5c2s-dense(1). This embodiment uses LeakyReLU with α=0.2 as all internal activations, while the output layer uses sigmoid activation.

[0082] To train G and D, this example uses the Adam optimizer with a learning rate of 0.00005, β1 = 0.9, β2 = 0.99, and a batch size of 128 for 100 epochs. This example uses Xavier weight initialization for all layers to find Will All values ​​in are restricted to the range of [-1, 1] and trained for 1000 iterations with the weight parameter λ set to 0.1.

[0083] In this embodiment, the facial images are fed into the VGG network (gaze direction estimation model) for feature extraction. Each VGG network is followed by a fully connected (FC) layer of size 512 after the last Max-pooling layer, followed by batch normalization and ReLU activation, and then these layers are concatenated to obtain a FC layer of size 1024, which is followed by another FC layer of size 512. The output of the last layer is the feature vector of the gaze direction.

[0084] As a loss function, this embodiment uses the sum of individual L2 losses between the predicted gaze direction vector and the true gaze direction vector, taken from a pre-trained model. The weights of the VGG model are initialized using a pre-trained model on ImageNet. Because weight sharing is considered to lead to performance degradation, it is not used in this embodiment. The weights of the FC layer are initialized using Xavier, using the Adam optimizer, with a learning rate of 0.001, β1=0.95, β2=0.9, and a batch size of 256.

[0085] S4. Based on the estimated gaze direction, the estimated head rotation angle and the EEG curve, the current learning engagement data of the learning subject to be monitored is obtained.

[0086] The monitoring result data of the current learning engagement of the monitored learning subject is generated based on the estimated head rotation angle, estimated gaze direction and EEG curve. Based on the monitoring result data, the learning engagement and learning effect feedback are provided to the classroom learning subject. On the one hand, it can help them understand their own learning status and optimize the learning process. On the other hand, it can evaluate the teaching quality of teachers and help teachers better improve the course content.

[0087] Figure 5 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present invention is schematically shown. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0088] like Figure 5As shown, the electronic device 1000 described in this embodiment includes: a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0089] In RAM 1003, various programs and data required for the operation of system 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other through bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0090] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The system 1000 may further include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0091] According to the method flow of the embodiment of the present disclosure, it can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, equipment, device, module or unit described above can be implemented by a computer program module.

[0092] The embodiments of the present invention further provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0093] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the ROM 1002 and / or RAM 1003 described above.

[0094] It should be noted that the functional modules in the various embodiments of the present invention can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product.

[0095] The flowchart or block diagram in the accompanying drawings illustrates the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope of the present disclosure.

[0097] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A method for acquiring learning engagement data based on a multimodal fusion network, characterized in that: Includes steps: Acquiring a facial image, a head rotation image, and EEG information of a subject to be monitored based on a device system including a wearable device; Preprocessing the acquired facial images, head rotation images and EEG information to make the related images correspond to the EEG information of the corresponding timestamps; Inputting the facial image into the gaze direction estimation model in the multimodal network model for feature extraction to obtain the corresponding gaze direction feature vector; The head rotation image is input into the head rotation angle estimation model in the multimodal network model for feature extraction to obtain the corresponding head rotation angle feature vector; The gaze direction feature vector and the head rotation angle feature vector are combined with the EEG information for fusion and classification, and the current estimated gaze direction, estimated head rotation angle and EEG curve of the subject to be monitored are obtained; wherein, for the face image input into the gaze direction estimation model, the area covered by the wearable device is firstly repaired by using the GAN-based semantic image repair method, and the repairing of the area covered by the wearable device by using the GAN-based semantic image repair method specifically includes: Using the least squares loss method, the training goal of GAN is to minimize ,in, ， ， Among them, D i represents the discriminator, D(x i ) is the input image x i is the probability of the true sample; G i represents a generator, which is used to output a synthetic image; What is measured is the realism of the image in 3D. What is measured is the realism of the image produced by G, which is considered as perceptual loss. The formula for perceptual loss is: ， The context loss is measured based on the difference between the non-occluded real image x and the generated image G(z). The formula of context loss is: ， Among them, ⊙ is the element-wise product, M1 is the complement of M, and M is the area that should not be repaired; Generating the best image for inpainting is equivalent to finding the best The value minimizes the combination of perceptual loss and contextual loss as follows: ， Among them, λ is the weighting parameter. After finding z, the repaired image can be generated by the following formula: ， Then, the Poisson mixture is applied to , to generate the final repaired image with seamless boundaries between the repaired and non-repaired areas; Based on the estimated gaze direction, the estimated head rotation angle and the EEG curve, the current learning engagement data of the learning subject to be monitored is obtained.

2. The method for obtaining learning engagement data according to claim 1, characterized in that: The input layer of the head rotation angle estimation model inputs the head rotation image, the feature extraction layer includes three convolution layers, three pooling layers, two fully connected layers and three drop-out layers, and the output layer outputs the corresponding head rotation angle feature vector; the head rotation angle estimation model uses Nesterov's accelerated gradient descent method for parameter optimization.

3. The method for obtaining learning engagement data according to claim 1, characterized in that: Inputting the facial image into the gaze direction estimation model in the multimodal network model for feature extraction specifically includes: Use the key point detection module to extract the positions of 5 facial key points to generate eye images; The gaze direction estimation model is trained based on the annotated gaze labels.

4. The method for obtaining learning engagement data according to claim 1, characterized in that: The learning engagement data includes the EEG curve deviation degree data; the EEG curve deviation degree data is obtained by calculating the degree to which the EEG curve deviates from the EEG baseline curve; the EEG baseline curve is a curve formed by EEG information generated by a learning subject whose learning engagement reaches the standard; The EEG curve deviation data can be used to analyze the learning engagement of the monitored learning subject.

5. The method for obtaining learning engagement data according to claim 1, characterized in that: When preprocessing the acquired facial image, head rotation image and EEG information, it also includes: Eliminate data that do not meet the quality requirements, and retain data that meet the quality requirements for preprocessing; The preprocessed data is subjected to data enhancement operations including one or more methods of distortion, adding noise, and reducing resolution.

6. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit executes the steps of the method according to any one of claims 1 to 5.

7. A storage medium, characterized in that: It stores a computer program executable by an access authentication device. When the computer program runs on the access authentication device, the access authentication device executes the steps of the method according to any one of claims 1 to 5.

Images