Method and system for learning process focus determination based on multi-modal alignment

Abstract

This invention discloses a method and system for judging attention levels during the learning process based on multimodal alignment, belonging to the field of multimodal pattern recognition. The method includes the following steps: acquiring multimodal data: acquiring raw multimodal data of students' educational behaviors and preprocessing the raw multimodal data; feature extraction: constructing a multimodal data representation encoding model corresponding to each modality of data for corresponding representation extraction; model training: after representation fusion, training the attention level judgment model using the fused multimodal representations and the student's attention level label; attention level prediction: inputting the fused multimodal representation to be detected into the trained attention level judgment model, and outputting the predicted student's attention level. This invention fully utilizes the rich information of multimodal data and the diverse characteristics of learners, improving the accuracy and interpretability of representation learning, and enabling better prediction of learners' attention levels during the learning process.

Description

Technical Field

[0001] This invention belongs to the field of multimodal representation learning, and in particular relates to a method and system for judging the level of attention in the learning process based on multimodal alignment. Background Technology

[0002] In education, evaluating and improving students' learning focus has always been a crucial issue. Deep learning, with its powerful data processing and pattern recognition capabilities, is considered to have enormous potential in assisting in the evaluation of educational effectiveness and the analysis of teaching methods. However, in practical applications, several challenges and limitations remain. First, educational data is typically multimodal, encompassing different types such as text, images, audio, and video, which complicates data integration and processing. Second, the amount of data in educational scenarios is often limited, while deep learning models usually require large amounts of data for training to achieve good performance. This necessitates new methods to overcome the problem of insufficient data. Data scarcity also significantly limits the application of traditional solutions in education. Therefore, how to achieve training with only a small number of samples while obtaining robust multimodal representations has become an urgent problem to be solved.

[0003] Multimodal data representation learning refers to the process of extracting information from different modalities such as audio, video, and text, and transforming this information into efficient mathematical representations. In education, this method can be used to analyze students' behavior, emotional responses, and how they interact with teaching content. For example, analyzing students' voice data can reveal their engagement and emotional state; video data can help monitor students' body language and attention span; and text analysis can assess students' comprehension and expression abilities.

[0004] In the field of multimodal data representation learning, traditional methods often focus only on easily accessible but low-information-density data. How to utilize deep data, including brain imaging, audio, and eye-tracking data, to provide a comprehensive understanding of an individual student's learning process remains a challenge. Simultaneously, data alignment techniques ensure consistency and complementarity between different types of data. Mapping data from diverse sources to the same reference frame unifies and integrates information. This is crucial for improving the accuracy and effectiveness of learning representations and ultimately, for predicting focus levels during the learning process. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and to provide a method and system for judging the level of attention in the learning process based on multimodal alignment.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for judging the level of attention in a learning process based on multimodal alignment, which includes the following steps:

[0008] S1: Obtain raw multimodal data of students in educational behavior and the corresponding students' attention levels, and preprocess the raw multimodal data to obtain preprocessed multimodal data, which includes preprocessed brain imaging data, speech data and eye movement data.

[0009] S2: Based on the needs of extracting multimodal representations, construct a first multimodal data representation coding model, a second multimodal data representation coding model, and a third multimodal data representation coding model specific to the education system domain, respectively, to extract brain imaging data representation, audio data representation, and eye-tracking data representation from the preprocessed multimodal data.

[0010] S3: The brain imaging data representation, audio data representation and eye movement data representation are fused to obtain the fused multimodal representation. A training dataset is constructed from the fused multimodal representation and the student's attention level label. The attention level judgment model is trained on the training dataset until the total loss converges to obtain the trained attention level judgment model, which is used to judge the attention level of each learner in the learning process.

[0011] The attention level judgment model consists of a bidirectional long short-term memory network and a fully connected layer cascaded together. The bidirectional long short-term memory network is used to extract temporal features, and the fully connected layer is used to map the temporal features extracted by the bidirectional long short-term memory network into attention level labels.

[0012] S4: Acquire the raw multimodal data to be tested during educational activities; preprocess the raw multimodal data to be tested to obtain preprocessed multimodal data to be tested; extract the data representations of each modality from the preprocessed multimodal data to be tested using the first multimodal data representation encoding model, the second multimodal data representation encoding model, and the third multimodal data representation encoding model to obtain the brain imaging data representation, audio data representation, and eye movement data representation to be tested; fuse the brain imaging data representation, audio data representation, and eye movement data representation to be tested to obtain the fused multimodal representation to be tested; input the fused multimodal representation to be tested into the trained attention level judgment model to output the predicted attention level of the student.

[0013] Based on the above scheme, each step can be implemented in the following preferred manner.

[0014] As a preferred embodiment of the first aspect mentioned above, the specific process of step S1 is as follows:

[0015] S11: Obtain raw multimodal data of students in educational behavior, including brain imaging data, speech data, and eye-tracking data;

[0016] S12: Clean the original multimodal data. If there are missing data in the cleaned original multimodal data, interpolate the missing data to obtain the pre-processed multimodal data.

[0017] S13: Align the initially processed multimodal data to ensure temporal consistency and obtain preprocessed multimodal data;

[0018] S14: Save the preprocessed multimodal data and manage it uniformly using JSON format.

[0019] As a preferred embodiment of the first aspect mentioned above, the specific process of step S2 is as follows:

[0020] S21: For a preprocessed brain imaging data containing S frames, each frame of preprocessed brain imaging data is used to extract features using the first multimodal data representation coding model pre-trained on medical images to obtain a single frame of brain imaging data representation.

[0021] S22: For the preprocessed speech data, it is first divided into S short audio segments. Then, Mel spectrogram is used to convert each short audio segment into a spectrogram. Each spectrogram is then input into the pre-trained second multimodal data representation coding model to map each spectrogram into a dense vector. The dense vector is used as the audio data representation.

[0022] S23: For the preprocessed eye-tracking data, each frame of preprocessed eye-tracking data includes an RGB image from the subjective perspective and an attention heatmap. The attention heatmap is input into a third multimodal data representation encoding model pre-trained on medical data to obtain a heatmap representation. The RGB image is input into the third multimodal data representation encoding model pre-trained on medical data to obtain an RGB image representation. The heatmap representation and the RGB image representation are concatenated to obtain the eye-tracking data representation.

[0023] As a preferred embodiment of the first aspect above, in step S3, the total loss used when training the focus level judgment model is the cross-entropy loss function, with the following specific form:

[0024]

[0025] in, Z represents the cross-entropy loss function. i This represents the i-th focus level label; This represents the focus level label predicted by the focus level assessment model, where i represents the multimodal data index.

[0026] As a preferred embodiment of the first aspect, in step S2, the first multimodal data representation coding model is the CLIP image encoder, the second multimodal data representation coding model is the audio encoder in ImageBind, and the third multimodal data representation coding model is the dual-stream VGG network model.

[0027] Secondly, the present invention provides a system for judging the level of attention during a learning process based on multimodal alignment, comprising:

[0028] The data processing module is used to acquire raw multimodal data of students in educational behavior and the corresponding students' attention levels, and to preprocess the raw multimodal data to obtain preprocessed multimodal data, which includes preprocessed brain imaging data, speech data and eye movement data.

[0029] The representation extraction module is used to construct a first multimodal data representation coding model, a second multimodal data representation coding model, and a third multimodal data representation coding model specific to the education system domain, according to the needs of extracting multimodal representations. These models are used to extract brain imaging data representations, audio data representations, and eye-tracking data representations from the preprocessed multimodal data.

[0030] The model acquisition module is used to fuse brain imaging data representation, audio data representation and eye movement data representation to obtain a fused multimodal representation. A training dataset is constructed from the fused multimodal representation and the student's attention level label. The attention level judgment model is trained on the training dataset until the total loss converges to obtain a trained attention level judgment model, which is used to judge the attention level of each learner in the learning process.

[0031] The attention level judgment model consists of a bidirectional long short-term memory network and a fully connected layer cascaded together. The bidirectional long short-term memory network is used to extract temporal features, and the fully connected layer is used to map the temporal features extracted by the bidirectional long short-term memory network into attention level labels.

[0032] The result acquisition module is used to acquire the raw multimodal data to be tested in educational activities. It preprocesses the raw multimodal data to obtain preprocessed multimodal data to be tested. Then, it extracts the representations of each modality from the preprocessed multimodal data using a first multimodal data representation encoding model, a second multimodal data representation encoding model, and a third multimodal data representation encoding model, resulting in brain imaging data representation, audio data representation, and eye-tracking data representation. These three representations are then fused to obtain a fused multimodal representation to be tested. Finally, the fused multimodal representation is input into a trained attention level judgment model, which outputs the predicted student's attention level.

[0033] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for judging the level of attention in a learning process based on multimodal alignment as described in any of the solutions in the first aspect above.

[0034] Fourthly, the present invention provides a computer electronic device, including a memory and a processor;

[0035] The memory is used to store computer programs;

[0036] The processor is configured to, when executing the computer program, implement the method for judging the level of attention in the learning process based on multimodal alignment as described in any of the embodiments of the first aspect above.

[0037] Fifthly, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the method for judging the level of attention in a learning process based on multimodal alignment as described in any of the solutions in the first aspect above.

[0038] Compared with traditional single-modal learning methods, this invention has the following advantages:

[0039] This invention fully utilizes the rich information in multimodal data and the diverse characteristics of learners. By leveraging a pre-trained multimodal data representation encoding model, it improves the accuracy and interpretability of representation learning, enabling better prediction of learners' attention levels during the learning process. This invention utilizes the world knowledge inherent in existing pre-trained multimodal data representation encoding models to achieve a robust and highly interpretable attention level judgment model with a relatively small amount of data. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention;

[0041] Figure 2This is a schematic diagram of the modules of the system of the present invention. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0043] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0044] This invention is used to extract and align representations from multimodal data obtained in the field of education, such as brain imaging data, audio data, and eye-tracking data. The specific implementation process of this invention is described in detail below.

[0045] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for judging the level of attention in the learning process based on multimodal alignment is provided. This method utilizes the world knowledge contained in existing pre-trained multimodal large models to obtain a robust and interpretable level of attention judgment model with a small amount of data.

[0046] like Figure 1 As shown, the above method for judging the level of attention in the learning process based on multimodal alignment includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0047] S1: Obtain raw multimodal data of students in educational behavior and the corresponding students' attention levels, and preprocess the raw multimodal data to obtain preprocessed multimodal data, which includes preprocessed brain imaging data, speech data and eye movement data.

[0048] It should be noted that in step S1 of this invention, raw multimodal data of students, including brain imaging data, speech data, and eye movement data, can be acquired during educational activities. Simultaneously, more raw multimodal data in the context of the activity can be obtained from professional medical databases and public databases. This raw multimodal data is then manually aligned; for each student, brain imaging data, speech data, and eye movement data are matched to ensure temporal consistency. Furthermore, the raw multimodal data of each student is matched, associating brain imaging data, speech data, and eye movement data with each student for subsequent analysis and application.

[0049] As an example, various publicly available research datasets, such as those for autism and attention deficit disorder, can be obtained as supplementary data. It should be noted that these datasets are publicly available and have been authorized and consented to by the participants, thus posing no potential privacy or fairness issues.

[0050] As a specific implementation of this invention, the specific steps in S1 include:

[0051] S11: Obtain raw multimodal data of students' educational behaviors, including brain imaging data, speech data, and eye-tracking data.

[0052] S12: Clean the original multimodal data. If there are missing data in the cleaned original multimodal data, interpolate the missing data to obtain the preliminarily processed multimodal data.

[0053] It should be noted that in step S12 of this invention, duplicate data, useless information, and noisy data are removed during the cleaning process. For example, for dialogue data in speech data, if there is other people's voices, the speech data needs to be sliced ​​to ensure that the speech data in the slices belong to the same person; for eye-tracking data, the gaze heatmap and natural scene map under the subjective perspective are aligned.

[0054] S13: Align the initially processed multimodal data to ensure temporal consistency and obtain preprocessed multimodal data.

[0055] S14: Save the preprocessed multimodal data and manage it uniformly using JSON format.

[0056] It should be noted that in step S14 of this invention, the preprocessed multimodal data of each student is numbered and organized into a dictionary and added to a JSON list for subsequent data analysis and application.

[0057] S2: Construct a first multimodal data representation coding model E specific to the education system domain, based on the needs of extracting multimodal representations. BThe second multimodal data representation coding model E A And the third multimodal data representation coding model E T It is used to extract brain imaging data representation, audio data representation, and eye-tracking data representation from preprocessed multimodal data.

[0058] It should be noted that in step S2 of this invention, brain imaging data and eye-tracking data are each divided into frames and input into their respective corresponding encoding models. For speech data, it is divided into short audio segments with the same number of frames as the brain imaging data, and each short audio segment is input into its corresponding encoding model.

[0059] As a specific implementation of an embodiment of the present invention, the specific implementation steps of step S2 above include:

[0060] S21: For a preprocessed brain imaging data containing S frames Each frame of preprocessed brain imaging data The first multimodal data representation encoding model E, pre-trained on medical images, is used. B Feature extraction is performed to obtain a single-frame brain imaging data representation e B =E B (I B ); where H and W represent the height and width of a frame of brain imaging data, respectively.

[0061] It should be noted that in step S21 of this invention, the first multimodal data representation coding model E B The appropriate model can be selected based on actual needs. In this embodiment, the first multimodal data representation coding model E... B Select CLIP as the image encoder.

[0062] The following section details the principles of the CLIP model. The CLIP model is a multimodal model that utilizes a large amount of image and text data for pre-training to learn the relationship between vision and language. CLIP has strong transfer learning capabilities; even in educational settings where multimodal data is scarce, the CLIP model can learn some common features from large-scale data in other domains, reducing the overhead of data acquisition and annotation.

[0063] S22: For the preprocessed speech data, it is first divided into S short audio segments, and then each short audio segment is converted into a spectrogram using Mel spectrogram. The resulting spectrograms are then I... F The data is sequentially input into the pre-trained second multimodal data representation encoding model E. A In this process, each spectrogram is mapped to a dense vector, and the dense vector is used as the audio data representation. A =E A(I F ).

[0064] It should be noted that in step S22 of this invention, the second multimodal data representation coding model E A The appropriate model can be selected based on actual needs. In this embodiment, the second multimodal data representation coding model E... A Select the audio encoder in ImageBind.

[0065] S23: For the preprocessed eye-tracking data, each frame of preprocessed eye-tracking data includes an RGB image I from the subjective viewpoint. TN And Attention Heatmap I TH Attention Heatmap I TH The input is fed into a third-modal data representation encoding model E, which has been pre-trained on medical data. T In the process, a thermal diagram characterizing e is obtained. TH =E T (I TH ); the RGB image I TN The input is fed into a third-modal data representation encoding model E, which has been pre-trained on medical data. T In the process, the RGB image representation e is obtained. TN =E T (I TN After stitching together the heatmap representation and the RGB image representation, the eye-tracking data representation e is obtained. T =[e TN ,e TH ].

[0066] It should be noted that in step S23 of this invention, the third multimodal data representation coding model E T The appropriate model can be selected based on actual needs. In this embodiment, the third multimodal data representation coding model E... T The dual-stream VGG network model was selected.

[0067] In this embodiment, a dual-stream VGG network model was pre-trained on ImageNet and medical eye-tracking data, thus exhibiting excellent zero-shot learning capabilities. It can learn robust eye-tracking data representations with only a small amount of educational multimodal data, which greatly reduces the reliance on large amounts of labeled data.

[0068] S3: The brain imaging data representation, audio data representation and eye movement data representation are fused to obtain the fused multimodal representation. A training dataset is constructed from the fused multimodal representation and the student's attention level label. The attention level judgment model is trained on the training dataset until the total loss converges to obtain the trained attention level judgment model, which is used to judge the attention level of each learner in the learning process.

[0069] The aforementioned attention level judgment model consists of a bidirectional long short-term memory network (Bi-LSTM) and a fully connected layer cascaded together. The Bi-LSTM is used to extract temporal features, and the fully connected layer is used to map the temporal features extracted by the Bi-LSTM to attention level labels.

[0070] It should be noted that in step S3 of this invention, the i-th brain imaging data is characterized. Audio data representation and eye-tracking data representation The fusion is performed, that is, the representation of the i-th brain imaging data. Audio data representation and eye-tracking data representation By splicing the data, we can obtain the i-th fused multimodal representation. Ultimately, all fused multimodal representations can be expressed as M = [e 1 ,e 2 ,...,e S ], where e 1 ,e 2 ,...,e S Let S represent the 1st, 2nd, ..., Sth fused multimodal representations, respectively.

[0071] A focus level judgment model is constructed by cascading a bidirectional long short-term memory network and a fully connected layer. All the fused multimodal representations M and the obtained student focus level labels Z are input into the constructed focus level judgment model to train the focus level judgment model. The bidirectional long short-term memory network Bi-LSTM extracts temporal features from the fused multimodal representations. An additional fully connected layer is added after the last layer of the bidirectional long short-term memory network Bi-LSTM to map the temporal features output by Bi-LSTM to focus level labels, thus obtaining the trained focus level judgment model, which is used to predict the student's focus level during the learning process using multimodal data.

[0072] It should be noted that in step S3 of this invention, the total loss used when training the focus level judgment model is the cross-entropy loss function, with the following specific form:

[0073]

[0074] in, Z represents the cross-entropy loss function. i This represents the i-th focus level label; This represents the focus level label predicted by the focus level assessment model, where i represents the multimodal data index.

[0075] S4: Acquire the raw multimodal data to be tested during educational activities; preprocess the raw multimodal data to be tested to obtain preprocessed multimodal data to be tested; extract the data representations of each modality from the preprocessed multimodal data to be tested using the first multimodal data representation encoding model, the second multimodal data representation encoding model, and the third multimodal data representation encoding model to obtain the brain imaging data representation, audio data representation, and eye movement data representation to be tested; fuse the brain imaging data representation, audio data representation, and eye movement data representation to be tested to obtain the fused multimodal representation to be tested; input the fused multimodal representation to be tested into the trained attention level judgment model to output the predicted attention level of the student.

[0076] It should also be noted that the method for judging the level of attention in the learning process based on multimodal alignment in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a system for judging the level of attention in the learning process based on multimodal alignment, corresponding to the method for judging the level of attention in the learning process based on multimodal alignment provided in the above embodiments. For example... Figure 2 As shown, this system includes four basic modules, namely:

[0077] The data processing module is used to acquire raw multimodal data of students in educational behavior and the corresponding students' attention levels, and to preprocess the raw multimodal data to obtain preprocessed multimodal data, which includes preprocessed brain imaging data, speech data and eye movement data.

[0078] The representation extraction module is used to construct a first multimodal data representation coding model, a second multimodal data representation coding model, and a third multimodal data representation coding model specific to the education system domain, according to the needs of extracting multimodal representations. These models are used to extract brain imaging data representations, audio data representations, and eye-tracking data representations from the preprocessed multimodal data.

[0079] The model acquisition module is used to fuse brain imaging data representation, audio data representation and eye movement data representation to obtain a fused multimodal representation. A training dataset is constructed from the fused multimodal representation and the student's attention level label. The attention level judgment model is trained on the training dataset until the total loss converges to obtain a trained attention level judgment model, which is used to judge the attention level of each learner in the learning process.

[0080] The attention level judgment model consists of a bidirectional long short-term memory network and a fully connected layer cascaded together. The bidirectional long short-term memory network is used to extract temporal features, and the fully connected layer is used to map the temporal features extracted by the bidirectional long short-term memory network into attention level labels.

[0081] The result acquisition module is used to acquire the raw multimodal data to be tested in educational activities. It preprocesses the raw multimodal data to obtain preprocessed multimodal data to be tested. Then, it extracts the representations of each modality from the preprocessed multimodal data using a first multimodal data representation encoding model, a second multimodal data representation encoding model, and a third multimodal data representation encoding model, resulting in brain imaging data representation, audio data representation, and eye-tracking data representation. These three representations are then fused to obtain a fused multimodal representation to be tested. Finally, the fused multimodal representation is input into a trained attention level judgment model, which outputs the predicted student's attention level.

[0082] It is understood that the methods for judging the level of attention in the learning process based on multimodal alignment described in S1 to S4 above can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the method for judging the level of attention in the learning process based on multimodal alignment provided in the above embodiments. This product includes a computer program / instruction, which, when executed by a processor, can implement the method for judging the level of attention in the learning process based on multimodal alignment as described in the above embodiments.

[0083] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method for judging the attention level of the learning process based on multimodal alignment provided in the above embodiments, which includes a memory and a processor;

[0084] The memory is used to store computer programs;

[0085] The processor is configured to implement the method for judging the level of attention in the learning process based on multimodal alignment in the above embodiments when executing the computer program.

[0086] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0087] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method for judging the attention level of the learning process based on multimodal alignment provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can implement the method for judging the attention level of the learning process based on multimodal alignment in the above embodiments.

[0088] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0089] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0090] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0091] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for judging the level of attention in a learning process based on multimodal alignment, characterized in that, Includes the following steps: S1: Obtain raw multimodal data of students in educational behavior and the corresponding students' attention levels, and preprocess the raw multimodal data to obtain preprocessed multimodal data, which includes preprocessed brain imaging data, speech data and eye movement data. S2: Construct a first multimodal data representation coding model, a second multimodal data representation coding model, and a third multimodal data representation coding model according to the needs of extracting multimodal representations, to be used to extract brain imaging data representations, audio data representations, and eye-tracking data representations from the preprocessed multimodal data. S3: The brain imaging data representation, audio data representation and eye movement data representation are fused to obtain the fused multimodal representation. A training dataset is constructed from the fused multimodal representation and the student's attention level label. The attention level judgment model is trained on the training dataset until the total loss converges to obtain the trained attention level judgment model, which is used to judge the attention level of each learner in the learning process. The attention level judgment model consists of a bidirectional long short-term memory network and a fully connected layer cascaded together. The bidirectional long short-term memory network is used to extract temporal features, and the fully connected layer is used to map the temporal features extracted by the bidirectional long short-term memory network into attention level labels. S4: Acquire the raw multimodal data to be tested in educational activities, preprocess the raw multimodal data to be tested to obtain preprocessed multimodal data to be tested; extract the data representations of each modality in the preprocessed multimodal data to be tested from the first multimodal data representation coding model, the second multimodal data representation coding model and the third multimodal data representation coding model to obtain the brain imaging data representation, audio data representation and eye movement data representation to be tested; The brain imaging data representation, audio data representation, and eye movement data representation to be detected are fused to obtain the fused multimodal representation to be detected; the fused multimodal representation to be detected is input into the trained attention level judgment model, and the predicted attention level of the student is output. The specific process of step S2 is as follows: S21: For a segment containing The preprocessed brain imaging data of each frame is used to extract features using a first multimodal data representation encoding model pre-trained on medical images to obtain a single-frame brain imaging data representation. S22: For the preprocessed speech data, first divide it into... The audio segments are divided into short audio fragments, and then each short audio fragment is converted into a spectrogram using Mel spectrogram. Each spectrogram is then input into a pre-trained second multimodal data representation encoding model to map each spectrogram into a dense vector, and the dense vector is used as the audio data representation. S23: For the preprocessed eye-tracking data, each frame of preprocessed eye-tracking data includes an RGB image from the subjective perspective and an attention heatmap. The attention heatmap is input into a third multimodal data representation encoding model pre-trained on medical data to obtain a heatmap representation. The RGB image is input into the third multimodal data representation encoding model pre-trained on medical data to obtain an RGB image representation. The heatmap representation and the RGB image representation are concatenated to obtain the eye-tracking data representation.

2. The method for judging the level of attention in the learning process based on multimodal alignment as described in claim 1, characterized in that, The specific process of step S1 is as follows: S11: Obtain raw multimodal data of students in educational behavior, including brain imaging data, speech data, and eye-tracking data; S12: Clean the original multimodal data. If there are missing data in the cleaned original multimodal data, interpolate the missing data to obtain the pre-processed multimodal data. S13: Align the initially processed multimodal data to ensure temporal consistency and obtain preprocessed multimodal data; S14: Save the preprocessed multimodal data and manage it uniformly using JSON format.

3. The method for judging the level of attention in the learning process based on multimodal alignment as described in claim 1, characterized in that, In step S3, the total loss used when training the focus level judgment model is the cross-entropy loss function, with the following specific form: ； in, This represents the cross-entropy loss function. Indicates the first A focus level label; This represents the focus level label predicted by the focus level assessment model. This indicates a multimodal data index.

4. The method for judging the level of attention in the learning process based on multimodal alignment as described in claim 1, characterized in that, In step S2, the first multimodal data representation coding model is the CLIP image encoder, the second multimodal data representation coding model is the audio encoder in ImageBind, and the third multimodal data representation coding model is the dual-stream VGG network model.

5. A system for judging the level of attention during a learning process based on multimodal alignment, characterized in that, include: The data processing module is used to acquire raw multimodal data of students in educational behavior and the corresponding students' attention levels, and to preprocess the raw multimodal data to obtain preprocessed multimodal data, which includes preprocessed brain imaging data, speech data and eye movement data. The representation extraction module is used to construct a first multimodal data representation encoding model, a second multimodal data representation encoding model, and a third multimodal data representation encoding model according to the needs of extracting multimodal representations, and to extract brain imaging data representations, audio data representations, and eye-tracking data representations from the preprocessed multimodal data respectively. The model acquisition module is used to fuse brain imaging data representation, audio data representation and eye movement data representation to obtain a fused multimodal representation. A training dataset is constructed from the fused multimodal representation and the student's attention level label. The attention level judgment model is trained on the training dataset until the total loss converges to obtain a trained attention level judgment model, which is used to judge the attention level of each learner in the learning process. The attention level judgment model consists of a bidirectional long short-term memory network and a fully connected layer cascaded together. The bidirectional long short-term memory network is used to extract temporal features, and the fully connected layer is used to map the temporal features extracted by the bidirectional long short-term memory network into attention level labels. The result acquisition module is used to acquire the raw multimodal data to be tested in educational activities. It preprocesses the raw multimodal data to obtain preprocessed multimodal data to be tested. Then, it extracts the data representations of each modality from the preprocessed multimodal data using a first multimodal data representation encoding model, a second multimodal data representation encoding model, and a third multimodal data representation encoding model, resulting in brain imaging data representation, audio data representation, and eye-tracking data representation. These three data representations are then fused to obtain a fused multimodal representation to be tested. Finally, the fused multimodal representation is input into a trained attention level judgment model, which outputs a predicted level of student attention. The specific process of the characterization extraction module is as follows: S21: For a segment containing The preprocessed brain imaging data of each frame is used to extract features using a first multimodal data representation encoding model pre-trained on medical images to obtain a single-frame brain imaging data representation. S22: For the preprocessed speech data, first divide it into... The audio segments are divided into short audio fragments, and then each short audio fragment is converted into a spectrogram using Mel spectrogram. Each spectrogram is then input into a pre-trained second multimodal data representation encoding model to map each spectrogram into a dense vector, and the dense vector is used as the audio data representation. S23: For the preprocessed eye-tracking data, each frame of preprocessed eye-tracking data includes an RGB image from the subjective perspective and an attention heatmap. The attention heatmap is input into a third multimodal data representation encoding model pre-trained on medical data to obtain a heatmap representation. The RGB image is input into the third multimodal data representation encoding model pre-trained on medical data to obtain an RGB image representation. The heatmap representation and the RGB image representation are concatenated to obtain the eye-tracking data representation.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method for judging the level of attention in the learning process based on multimodal alignment as described in any one of claims 1 to 4.

7. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the method for judging the level of attention in the learning process based on multimodal alignment as described in any one of claims 1 to 4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the method for judging the level of attention in the learning process based on multimodal alignment as described in any one of claims 1 to 4.

Images