An intelligent learning suggestion recommendation method and system based on multi-modal emotion recognition

Abstract

The application discloses an intelligent learning suggestion recommendation method and system based on multi-modal emotion recognition, belongs to the technical field of intelligent learning monitoring, and through collection of multi-modal emotion feature data, brain wave fluctuation characteristics are supplemented by facial expression features and voice features, interference of pure brain wave fluctuation characteristics on emotion recognition is avoided, accuracy of emotion recognition is greatly improved, meanwhile, a plurality of emotion recognition reference sources are formed by using historical brain wave fluctuation data, brain wave fluctuation characteristic data of a target learner is compared with each emotion recognition reference source, the most similar target reference source is obtained, characteristic enhancement of the target reference source on the brain wave fluctuation characteristic data of the target learner is carried out, different target multi-modal emotion recognition models are adopted for specific emotion recognition for different emotion types, more accurate emotion state categories are obtained, and generalization ability of emotion recognition for various different emotions is greatly improved.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent learning monitoring, specifically relating to an intelligent learning suggestion recommendation method and system based on multimodal emotion recognition. Background Technology

[0002] With the deep integration of artificial intelligence and educational technology, intelligent learning monitoring has become a core direction for improving learners' learning quality. Among these efforts, the real-time and accurate identification of learners' emotional states, and the provision of dynamic learning suggestions accordingly, is crucial for building a next-generation education system with emotional intelligence. Emotions, as a core factor influencing cognitive load, attention retention, and learning motivation, require accurate identification for truly adaptive learning. Currently, emotion recognition primarily relies on the analysis of multimodal physiological and behavioral signals, especially electroencephalogram (EEG) signals, which have received widespread attention for directly reflecting higher cognitive and emotional activities of the brain. However, existing emotion recognition technologies are not yet fully developed in processing and applying EEG signals.

[0003] First, EEG signals are highly specific and influenced by physiological structure, cognitive habits, etc., resulting in huge differences between various emotion types. Furthermore, when an emotion recognition model trained using EEG signals that are more influenced by a certain type of emotion as training data is applied to the recognition of other types of emotions, its performance will drop sharply, that is, there is a significant emotion source bias characteristic.

[0004] Secondly, most existing methods for emotion recognition based on EEG signals perform isolated emotion recognition using EEG signals, ignoring the actual emotions reflected in factors such as user facial expressions and voice. During the learning process, EEG signals may generate abnormal or extreme transient signals for complex learning tasks, thus interfering with emotion recognition based solely on EEG signals. These transient EEG signals do not have a substantial impact on the user's actual facial expressions and voice fluctuations, and existing technologies do not accurately supplement the recognition of user facial expressions and voice.

[0005] As mentioned above, how to provide an intelligent learning suggestion recommendation method and system based on multimodal emotion recognition that can improve the generalization ability of emotion recognition and enhance the accuracy of emotion recognition for learners has become an urgent research topic in this field. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent learning suggestion recommendation method and system based on multimodal emotion recognition, so as to solve the above-mentioned problems existing in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides an intelligent learning suggestion recommendation method based on multimodal emotion recognition, comprising: The original multimodal emotion data of the target learner is obtained, and feature extraction and data preprocessing are performed on the original multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes target learner EEG fluctuation feature data, target learner facial expression feature data and target learner speech feature data; Historical EEG fluctuation data is acquired, and multiple emotion recognition reference sources are formed based on the historical EEG fluctuation data. The target learner's EEG fluctuation feature data in the multimodal emotion feature data is compared with each of the emotion recognition reference sources. The emotion recognition reference source with the highest similarity is selected as the target reference source. The target reference source is then used to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data. The target learner's facial expression feature data and speech feature data in the multimodal emotion feature data are respectively subjected to feature enhancement processing to obtain target learner's facial expression enhancement feature data and target learner's speech enhancement feature data. The target learner's EEG fluctuation enhancement feature data, target learner's facial expression enhancement feature data and target learner's speech enhancement feature data are then fused to obtain the target learner's multimodal emotion recognition features. A pre-trained target multimodal emotion recognition model is obtained. The multimodal emotion recognition features of the target learner are input into the target multimodal emotion recognition model to output the real-time emotion state category of the target learner. Based on the real-time emotion state category of the target learner, corresponding recommended learning suggestions are generated and fed back to the learning terminal device to remind the target learner to adjust their learning state. The target multimodal emotion recognition model is pre-trained using the target reference source.

[0008] In one possible design, the raw multimodal emotion data of the target learner is acquired, and feature extraction and data preprocessing are performed on the raw multimodal emotion data to obtain multimodal emotion feature data, including: Within a preset time window, multi-channel EEG acquisition devices are used to collect multi-channel EEG fluctuation data of the target learner from the EEG regions of the target learner, and the EEG fluctuation data is used as the original EEG fluctuation data of the target learner. The EEG regions include at least the frontal region and the central region. Within a preset time window, the learning terminal device captures the target learner's facial video stream through its camera and the target learner's speech stream through its microphone array. The target learner's facial video stream is used as the original target learner's facial expression data, and the target learner's speech stream is used as the original target learner's speech data. The original target learner's facial expression data includes multiple frames of the target learner's facial expression images. The original target learner's EEG fluctuation data, facial expression data, and speech data are integrated, and time alignment processing is performed on the original target learner's EEG fluctuation data, facial expression data, and speech data to form the target learner's original multimodal emotion data. Obtain a preset sliding time window, and according to the sliding time window, perform time-sliding segmentation on the original target learner EEG fluctuation data in the original multimodal emotion data to form multiple time window samples; Multiple preset feature frequency bands are obtained. Differential entropy features are extracted from each time window sample according to each feature frequency band to obtain the differential entropy features of each feature frequency band corresponding to each time window. In each time window, the differential entropy features of each feature frequency band are concatenated to form the time window EEG fluctuation feature data to form the time window EEG fluctuation feature data corresponding to each time window. The EEG fluctuation feature data corresponding to each time window are spliced ​​together to generate the EEG fluctuation feature data of the pre-target learner. A pre-trained facial expression feature recognition model is obtained. The original target learner facial expression data in the original multimodal emotion data is used as input. The facial expression images of the target learner are input into the facial expression feature recognition model frame by frame. The facial expression feature recognition model outputs the facial expression feature recognition result corresponding to each frame of the target learner's facial expression image. The facial expression feature recognition result corresponding to each frame of the target learner's facial expression image is then concatenated to form the pre-target learner facial expression feature data. Audio waveform features are extracted from the original target learner speech data in the original multimodal emotion data to form pre-target learner speech feature data; The pre-target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data are preprocessed to form target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data. These three data are then integrated to form multimodal emotion feature data.

[0009] In one possible design, the pre-target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data are preprocessed to form target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data, including: Anomaly detection is performed on the pre-target learner's EEG fluctuation feature data, pre-target learner's facial expression feature data, and pre-target learner's speech feature data to determine whether data anomalies and / or data missing occur. If the EEG fluctuation feature data of the pre-target learner shows abnormalities and / or missing data, the EEG fluctuation feature data, facial expression feature data, and speech feature data of the pre-target learner are discarded, and the original multimodal emotion data is collected and feature extracted again. The EEG fluctuation feature data of the pre-target learner obtained again is subjected to anomaly detection until the EEG fluctuation feature data of the pre-target learner does not show abnormalities and / or missing data. If the pre-target learner's EEG fluctuation feature data does not show any data abnormalities and / or missing data, but the pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data show data abnormalities and / or missing data, then the pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data are discarded. Based on the pre-target learner's EEG fluctuation feature data, missing data supplementation conditions are generated. The discarded pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data are then supplemented using the missing data supplementation conditions to obtain supplemented target learner's facial expression feature data and / or supplemented target learner's speech feature data. The pre-target learner's EEG fluctuation feature data is used as the target learner's EEG fluctuation feature data, the supplemented target learner's facial expression feature data is used as the target learner's facial expression feature data, and the supplemented target learner's speech feature data is used as the target learner's speech feature data. If the pre-target learner's EEG fluctuation feature data, pre-target learner's facial expression feature data, and pre-target learner's speech feature data do not show any data anomalies and / or missing data, then the pre-target learner's EEG fluctuation feature data will be used as the target learner's EEG fluctuation feature data, the pre-target learner's facial expression feature data will be used as the target learner's facial expression feature data, and the pre-target learner's speech feature data will be used as the target learner's speech feature data.

[0010] In one possible design, missing condition supplementation is generated based on the pre-target learner's EEG fluctuation characteristics data, including: When the facial expression feature data of the pre-target learner shows data anomalies and / or data missing, and the speech feature data of the pre-target learner does not show data anomalies and / or data missing, the EEG fluctuation feature data and the speech feature data of the pre-target learner are concatenated into a usable modality feature vector, and random noise is added to the usable modality feature vector to obtain missing compensation conditions. When the pre-target learner's speech feature data shows data anomalies and / or data missing, and the pre-target learner's facial expression feature data does not show data anomalies and / or data missing, the pre-target learner's EEG fluctuation feature data and the pre-target learner's facial expression feature data are concatenated into a usable modality feature vector, and random noise is added to the usable modality feature vector to obtain missing compensation conditions; When both the facial expression feature data and the speech feature data of the pre-target learner show data anomalies and / or data missing, an available modality feature vector is generated based on the EEG fluctuation feature data of the pre-target learner, and random noise is added to the available modality feature vector to obtain missing compensation conditions; Accordingly, supplementing the discarded facial expression feature data and / or speech feature data of the pre-target learner using the missing data supplementation conditions includes: A pre-trained conditional noise back-inference model is obtained. The missing supplementation conditions are used as input to the conditional noise back-inference model to perform multiple rounds of back-inference on the random noise in the missing supplementation conditions, so as to obtain the missing modality supplementation feature vector. The missing modality supplementation feature vector is used to supplement the facial expression feature data and / or the speech feature data of the pre-target learner, so as to obtain supplemented target learner facial expression feature data and / or supplemented target learner speech feature data.

[0011] In one possible design, historical EEG fluctuation data is acquired, and multiple emotion recognition reference sources are formed based on the historical EEG fluctuation data. The target learner's EEG fluctuation feature data in the multimodal emotion feature data is compared with each of the emotion recognition reference sources, and the emotion recognition reference source with the highest similarity is selected as the target reference source, including: Acquire historical EEG fluctuation data, wherein the historical EEG fluctuation data includes multiple segments of EEG fluctuation data of the target learner and corresponding emotion tags; According to the emotion label corresponding to each segment of EEG fluctuation data, the EEG fluctuation data segments are classified to form multiple segments of EEG fluctuation emotion data, so that each segment of EEG fluctuation emotion data can be used as an emotion recognition reference source, wherein each of the emotion recognition reference sources is used to represent a type of emotion. For the target learner's EEG fluctuation feature data, a spatial difference matrix is ​​generated according to the channel positions of the multi-channel EEG acquisition device, wherein the spatial difference matrix is ​​used to represent the Euclidean distance between the channel positions of the multi-channel EEG acquisition device; For the target learner's EEG fluctuation feature data, for each of the feature frequency bands, the corresponding center frequency is extracted, the frequency difference between the center frequencies of each feature frequency band is calculated, and a frequency band difference matrix is ​​generated based on the frequency difference between the center frequencies of each feature frequency band, wherein the frequency band difference matrix is ​​used to represent the oscillation feature differences between each of the feature frequency bands. Obtain a preset fusion weight, perform tensor product expansion on the spatial difference matrix and the frequency band difference matrix, and perform matrix fusion on the tensor product-expanded spatial difference matrix and the tensor product-expanded frequency band difference matrix according to the fusion weight to obtain the overall difference matrix. The target learner's EEG fluctuation feature data is flattened into a one-dimensional vector, and the flattened target learner's EEG fluctuation feature data is normalized to obtain the corresponding current EEG feature distribution vector. The EEG fluctuation emotion data in each of the emotion recognition reference sources are flattened into a one-dimensional vector, and the flattened EEG fluctuation emotion data are normalized to obtain the corresponding target reference EEG feature distribution vector. Between the current EEG feature distribution vector and each of the target reference EEG feature distribution vectors, the overall difference matrix is ​​used as a transmission distance constraint. The regularized transmission distance between the current EEG feature distribution vector and each of the target reference EEG feature distribution vectors is solved respectively. The optimal regularized transmission distance is selected from each regularized transmission distance, and the emotion recognition reference source corresponding to the optimal regularized transmission distance is used as the target reference source.

[0012] In one possible design, the target reference source is used to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate enhanced EEG fluctuation feature data of the target learner, including: Obtain the preset enhancement mixing weights, and then add the EEG fluctuation emotion data in the target reference source to the target learner's EEG fluctuation feature data according to the enhancement mixing weights to obtain the target learner's enhanced EEG fluctuation feature data. Obtain feature enhancement ratio According to the aforementioned feature enhancement ratio Randomly select from the target learner's EEG fluctuation feature data the feature enhancement ratio. The corresponding multiple feature data form the pre-target learner's brain wave fluctuation enhancement feature data, and the target learner's brain wave fluctuation enhancement hybrid feature data is used to fill the pre-target learner's brain wave fluctuation enhancement feature data to obtain the target learner's brain wave fluctuation enhancement hybrid feature data.

[0013] In one possible design, feature enhancement processing is performed on the target learner's facial expression feature data and speech feature data in the multimodal emotion feature data to obtain target learner's enhanced facial expression feature data and enhanced speech feature data. Then, feature fusion is performed on the target learner's enhanced EEG fluctuation feature data, enhanced facial expression feature data, and enhanced speech feature data to obtain the target learner's multimodal emotion recognition features, including: Obtain a pre-trained long-short-term attention model for facial expression features, and use the long-short-term attention model for facial expression features to perform attention calculation on the target learner's facial expression feature data in the multimodal emotion feature data to obtain the corresponding target learner's facial expression enhancement feature data. Obtain a pre-trained long-short-term attention model for speech features, and use the speech feature long-short-term attention model to perform attention calculation on the target learner speech feature data in the multimodal emotion feature data to obtain the corresponding target learner speech enhancement feature data; The target learner's enhanced EEG fluctuation hybrid feature data, enhanced facial expression feature data, and enhanced speech feature data are sequentially concatenated to generate multimodal emotion recognition features for the target learner.

[0014] In one possible design, a pre-trained target multimodal emotion recognition model is obtained, and the target learner's multimodal emotion recognition features are input into the target multimodal emotion recognition model to output the target learner's real-time emotion state category, including: A pre-trained target multimodal emotion recognition model is obtained, and the target learner's multimodal emotion recognition features are used as input to the target multimodal emotion recognition model. The target multimodal emotion recognition model outputs the target learner's emotion probability distribution map, wherein the emotion probability distribution map includes at least one emotion recognition result and the corresponding emotion recognition result confidence. Obtain a preset confidence threshold for emotion recognition results, extract the confidence of each emotion recognition result corresponding to the emotion recognition result from the emotion probability distribution map of the target learner, and use the confidence threshold for emotion recognition results to determine the confidence of each emotion recognition result corresponding to the emotion recognition result in the emotion probability distribution map, and obtain the determination result; If the determination result is that there is at least one emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold, then among all the emotion recognition results that exceed the emotion recognition result confidence threshold, the emotion recognition result with the highest emotion recognition result confidence level is selected as the real-time emotion state category of the target learner. If the determination result is that there is no emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold, then the target learner's EEG fluctuation feature data is again enhanced using the target reference source, and the target learner's enhanced EEG fluctuation feature data is updated. Based on the updated target learner's enhanced EEG fluctuation feature data, the corresponding target learner multimodal emotion recognition feature is regenerated. The regenerated target learner multimodal emotion recognition feature is input into the target multimodal emotion recognition model, and the emotion probability distribution map output by the target multimodal emotion recognition model is thresholded using the emotion recognition result confidence threshold until the determination result is that there is at least one emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold.

[0015] In one possible design, based on the real-time emotional state category of the target learner, corresponding recommended learning suggestions are generated, and these suggestions are fed back to the learning terminal device to remind the target learner to adjust their learning state, including: Obtain a preset learning suggestion recommendation table, use the real-time emotional state category of the target learner as the query condition, and retrieve the corresponding learning suggestions from the learning suggestion recommendation table as recommended learning suggestions; Based on the recommended learning suggestions, a corresponding display control signal is generated and fed back to the learning terminal device so that the recommended learning suggestions can be visualized through the display interface of the learning terminal device, thereby completing the adjustment reminder for the learning status of the target learner.

[0016] Secondly, the present invention provides an intelligent learning suggestion recommendation system based on multimodal emotion recognition, comprising: The data acquisition unit is used to acquire the target learner's raw multimodal emotion data, perform feature extraction and data preprocessing on the raw multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes the target learner's EEG fluctuation feature data, the target learner's facial expression feature data and the target learner's speech feature data; The EEG data enhancement unit is used to acquire historical EEG fluctuation data, form multiple emotion recognition reference sources based on the historical EEG fluctuation data, compare the target learner's EEG fluctuation feature data in the multimodal emotion feature data with each of the emotion recognition reference sources, select the emotion recognition reference source with the highest similarity as the target reference source, and use the target reference source to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data. The non-EEG data augmentation unit is used to perform feature augmentation processing on the target learner's facial expression feature data and the target learner's speech feature data in the multimodal emotion feature data, respectively, to obtain target learner's facial expression augmentation feature data and target learner's speech augmentation feature data; The emotion recognition feature fusion unit is used to fuse the target learner's enhanced EEG fluctuation feature data, enhanced facial expression feature data, and enhanced speech feature data to obtain the target learner's multimodal emotion recognition features. An emotion recognition and suggestion unit is used to acquire a pre-trained target multimodal emotion recognition model, input the target learner's multimodal emotion recognition features into the target multimodal emotion recognition model to output the target learner's real-time emotion state category, and generate corresponding recommended learning suggestions based on the target learner's real-time emotion state category, so as to feed the recommended learning suggestions back to the learning terminal device, thereby reminding the target learner to adjust their learning state. The target multimodal emotion recognition model is pre-trained using the target reference source.

[0017] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a transceiver connected in sequence and communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect or any possible design of the first aspect.

[0018] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect or any possible design of the first aspect.

[0019] Fifthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect or any possible design of the first aspect.

[0020] Beneficial Effects: This invention provides an intelligent learning suggestion recommendation method and system based on multimodal emotion recognition, comprising: First, acquiring the original multimodal emotion data of the target learner, performing feature extraction and data preprocessing on the original multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes the target learner's EEG fluctuation feature data, the target learner's facial expression feature data, and the target learner's speech feature data; Second, acquiring historical EEG fluctuation data, forming multiple emotion recognition reference sources based on the historical EEG fluctuation data, comparing the target learner's EEG fluctuation feature data in the multimodal emotion feature data with each of the emotion recognition reference sources, selecting the emotion recognition reference source with the highest similarity as the target reference source, and using the target reference source to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data; Then, processing the multimodal emotion feature data... The target learner's facial expression feature data and speech feature data are respectively subjected to feature enhancement processing to obtain target learner facial expression enhancement feature data and target learner speech enhancement feature data. Then, the target learner's EEG fluctuation enhancement feature data, target learner facial expression enhancement feature data, and target learner speech enhancement feature data are fused to obtain target learner multimodal emotion recognition features. Finally, a pre-trained target multimodal emotion recognition model is obtained, and the target learner multimodal emotion recognition features are input into the target multimodal emotion recognition model to output the target learner's real-time emotion state category. Based on the target learner's real-time emotion state category, corresponding recommended learning suggestions are generated and fed back to the learning terminal device to provide learning state adjustment reminders to the target learner. The target multimodal emotion recognition model is pre-trained using the target reference source. By collecting multimodal emotion feature data, facial expression features and voice features are used to supplement EEG fluctuation features, avoiding interference from simple EEG fluctuation features on emotion recognition and significantly improving the accuracy of emotion recognition. At the same time, multiple emotion recognition reference sources are formed using historical EEG fluctuation data. The target learner's EEG fluctuation feature data is compared with each emotion recognition reference source to obtain the most similar target reference source. This target reference source is then used to enhance the target learner's EEG fluctuation feature data, ensuring that different target multimodal emotion recognition models are used for specific emotion recognition for different emotion types, resulting in more accurate emotion state categories and significantly improving the generalization ability of emotion recognition for various emotions. In addition, by generating recommended learning suggestions, timely reminders are sent to learners to avoid the learning efficiency being affected by emotions. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the intelligent learning suggestion recommendation method based on multimodal emotion recognition provided in an embodiment of the present invention. Figure 2 A schematic diagram of the functional structure of an intelligent learning suggestion recommendation system based on multimodal emotion recognition provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0023] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0024] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0025] Example: like Figure 1 As shown, the first aspect of this embodiment provides an intelligent learning suggestion recommendation method based on multimodal emotion recognition, which may include, but is not limited to, the following steps: S1. Obtain the target learner's original multimodal emotion data, perform feature extraction and data preprocessing on the original multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes the target learner's EEG fluctuation feature data, the target learner's facial expression feature data and the target learner's speech feature data; S2. Acquire historical EEG fluctuation data, form multiple emotion recognition reference sources based on the historical EEG fluctuation data, compare the target learner's EEG fluctuation feature data in the multimodal emotion feature data with each of the emotion recognition reference sources, select the emotion recognition reference source with the highest similarity as the target reference source, and use the target reference source to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data; S3. Perform feature enhancement processing on the target learner's facial expression feature data and the target learner's speech feature data in the multimodal emotion feature data to obtain target learner's facial expression enhancement feature data and target learner's speech enhancement feature data, and perform feature fusion on the target learner's EEG fluctuation enhancement feature data, the target learner's facial expression enhancement feature data and the target learner's speech enhancement feature data to obtain target learner's multimodal emotion recognition features; S4. Obtain a pre-trained target multimodal emotion recognition model, input the target learner's multimodal emotion recognition features into the target multimodal emotion recognition model to output the target learner's real-time emotion state category, and generate corresponding recommended learning suggestions based on the target learner's real-time emotion state category, so as to feed the recommended learning suggestions back to the learning terminal device, thereby reminding the target learner to adjust their learning state. The target multimodal emotion recognition model is obtained by pre-training using the target reference source.

[0026] In one possible implementation, step S1 involves acquiring the target learner's original multimodal emotion data, performing feature extraction and data preprocessing on the original multimodal emotion data to obtain multimodal emotion feature data. This step can be broken down into, but is not limited to, the following steps S11-S1, specifically including: S11. Within a preset time window, multi-channel EEG acquisition equipment is used to collect multi-channel EEG fluctuation data of the target learner from the EEG region of the target learner, and the EEG fluctuation data is used as the original EEG fluctuation data of the target learner, wherein the EEG region includes at least the frontal region and the central region. S12. Within a preset time window, the facial video stream of the target learner is captured through the camera of the learning terminal device, and the speech stream of the target learner is captured through the microphone array of the learning terminal device. The facial video stream of the target learner is used as the original facial expression data of the target learner, and the speech stream of the target learner is used as the original speech data of the target learner. The original facial expression data of the target learner includes multiple frames of facial expression images of the target learner. S13. Integrate the original target learner's EEG fluctuation data, the original target learner's facial expression data, and the original target learner's speech data, and perform time alignment processing on the original target learner's EEG fluctuation data, the original target learner's facial expression data, and the original target learner's speech data to form the target learner's original multimodal emotion data; S14. Obtain a preset sliding time window, and according to the sliding time window, perform time-sliding segmentation on the original target learner EEG fluctuation data in the original multimodal emotion data to form multiple time window samples; S15. Obtain multiple preset feature frequency bands, and extract differential entropy features for each time window sample according to each feature frequency band to obtain the differential entropy features of each feature frequency band corresponding to each time window. In each time window, the differential entropy features of each feature frequency band are concatenated to form time window EEG fluctuation feature data to form time window EEG fluctuation feature data corresponding to each time window. S16. The EEG fluctuation feature data corresponding to each time window are spliced ​​together to generate EEG fluctuation feature data of the pre-target learner; S17. Obtain a pre-trained facial expression feature recognition model, using the original target learner facial expression data in the original multimodal emotion data as input, inputting the target learner's facial expression image frame by frame into the facial expression feature recognition model, outputting the facial expression feature recognition result corresponding to each frame of the target learner's facial expression image according to the facial expression feature recognition model, and concatenating the facial expression feature recognition result corresponding to each frame of the target learner's facial expression image into pre-target learner facial expression feature data; S18. Extract audio waveform features from the original target learner speech data in the original multimodal emotion data to form pre-target learner speech feature data; S19. Perform data preprocessing on the pre-target learner's EEG fluctuation feature data, pre-target learner's facial expression feature data, and pre-target learner's speech feature data to form target learner's EEG fluctuation feature data, target learner's facial expression feature data, and target learner's speech feature data. Integrate the target learner's EEG fluctuation feature data, target learner's facial expression feature data, and target learner's speech feature data to form multimodal emotion feature data.

[0027] It should be noted that the intelligent learning suggestion method provided in this embodiment can perform a preliminary screening of the facial expression images of the target learner before performing facial expression feature recognition. Specifically, it selects the few frames of facial expression images of the candidate target learner (each frame of facial expression image in the facial video stream) collected within the time window that are closest to the center time of each time window sample as the final facial expression image of the target learner, so as to improve the feature extraction efficiency of the facial expression feature recognition model.

[0028] Furthermore, the facial expression feature recognition model described in this embodiment is a deep convolutional neural network model trained based on historical facial expression data. It is used to identify key feature points of facial expressions to generate a sequence of key feature points of facial expressions for each frame of the target learner's facial expression image, which is the facial expression feature recognition result corresponding to each frame of the target learner's facial expression image.

[0029] In one possible implementation, step S19 involves preprocessing the pre-target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data to form target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data. This can be, but is not limited to, decomposed into the following steps S191-S194, specifically including: S191. Perform anomaly detection on the pre-target learner's EEG fluctuation feature data, the pre-target learner's facial expression feature data, and the pre-target learner's speech feature data respectively to determine whether there are data anomalies and / or data missing. S192. If the pre-target learner's EEG fluctuation feature data shows abnormalities and / or missing data, then discard the pre-target learner's EEG fluctuation feature data, the pre-target learner's facial expression feature data, and the pre-target learner's speech feature data, and re-collect and extract the original multimodal emotion data, and perform anomaly detection on the re-obtained pre-target learner's EEG fluctuation feature data until the pre-target learner's EEG fluctuation feature data no longer shows abnormalities and / or missing data; S193. If the pre-target learner's EEG fluctuation feature data does not show any data abnormalities and / or data omissions, but the pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data show data abnormalities and / or data omissions, then the pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data are discarded. Based on the pre-target learner's EEG fluctuation feature data, missing data supplementation conditions are generated. The discarded pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data are then supplemented using the missing data supplementation conditions to obtain supplemented target learner's facial expression feature data and / or supplemented target learner's speech feature data. The pre-target learner's EEG fluctuation feature data is used as the target learner's EEG fluctuation feature data, the supplemented target learner's facial expression feature data is used as the target learner's facial expression feature data, and the supplemented target learner's speech feature data is used as the target learner's speech feature data. S194. If the pre-target learner's EEG fluctuation feature data, pre-target learner's facial expression feature data, and pre-target learner's speech feature data do not show any data anomalies and / or data missing, then the pre-target learner's EEG fluctuation feature data shall be used as the target learner's EEG fluctuation feature data, the pre-target learner's facial expression feature data shall be used as the target learner's facial expression feature data, and the pre-target learner's speech feature data shall be used as the target learner's speech feature data.

[0030] In one possible implementation, step S193, generating missing information based on the pre-target learner's EEG fluctuation characteristic data, can be, but is not limited to, decomposed into the following steps S193a1-S193a3. Specifically, step S193a includes: S193a1. When the facial expression feature data of the pre-target learner shows data abnormalities and / or data missing, and the speech feature data of the pre-target learner does not show data abnormalities and / or data missing, the EEG fluctuation feature data and the speech feature data of the pre-target learner are concatenated into a usable modal feature vector, and random noise is added to the usable modal feature vector to obtain missing supplementation conditions. S193a2. When the pre-target learner's speech feature data shows data abnormalities and / or data missing, and the pre-target learner's facial expression feature data does not show data abnormalities and / or data missing, the pre-target learner's EEG fluctuation feature data and the pre-target learner's facial expression feature data are concatenated into a usable modality feature vector, and random noise is added to the usable modality feature vector to obtain missing compensation conditions; S193a3. When both the facial expression feature data and the speech feature data of the pre-target learner show data anomalies and / or data missing, an available modality feature vector is generated based on the EEG fluctuation feature data of the pre-target learner, and random noise is added to the available modality feature vector to obtain missing supplementation conditions; Accordingly, in step S193, the process of supplementing the discarded facial expression feature data and / or speech feature data of the pre-target learner using the missing data supplementation conditions can be broken down into, but is not limited to, the following steps S193b1-S193b2, specifically including: S193b1. Obtain a pre-trained conditional noise back-inference model, and input the missing supplementation conditions as input to the conditional noise back-inference model to perform multiple rounds of back-inference on the random noise in the missing supplementation conditions through the conditional noise back-inference model to obtain the missing modality supplementation feature vector: S193b2. Using the missing modality supplementation feature vector, feature data supplementation is performed on the pre-target learner facial expression feature data and / or the pre-target learner speech feature data to obtain supplemented target learner facial expression feature data and / or supplemented target learner speech feature data.

[0031] It should be noted that the intelligent learning suggestion recommendation method based on multimodal emotion recognition provided in this embodiment uses a conditional noise back-inference model and a controllable iterative denoising process of stochastic differential equations to iteratively infer the missing modality supplementary feature vector corresponding to the missing modality from the introduced random noise. This makes the multimodal features in this embodiment more stable, and the final generated features are more realistic in detail, maintaining good consistency with the missing modality in emotional content. Through ablation experiments on the conditional noise back-inference model, it was found that even in the extreme case of severely missing speech feature data (missing rate exceeding 0.7), the emotion recognition results in this embodiment still maintain a recognition accuracy of over 97%. This means that in practical applications, even if the overall performance of the terminal learning device drops significantly, the intelligent learning suggestion recommendation method in this embodiment can still provide extremely strong working stability, ensuring that the intelligent learning suggestion recommendation method in this embodiment has strong resilience in the face of incomplete data.

[0032] In one possible implementation, step S2 involves acquiring historical EEG fluctuation data, forming multiple emotion recognition reference sources based on the historical EEG fluctuation data, comparing the target learner's EEG fluctuation feature data in the multimodal emotion feature data with each of the emotion recognition reference sources, and selecting the emotion recognition reference source with the highest similarity as the target reference source. This can be broken down into, but is not limited to, the following steps S21-S28, specifically including: S21. Obtain historical EEG fluctuation data, wherein the historical EEG fluctuation data includes multiple segments of EEG fluctuation data of the target learner and corresponding emotion tags; S22. According to the emotion label corresponding to each segment of EEG fluctuation data, classify the EEG fluctuation data into multiple segments of EEG fluctuation emotion data, so that each segment of EEG fluctuation emotion data can be used as an emotion recognition reference source, wherein each of the emotion recognition reference sources is used to represent a type of emotion. S23. For the target learner's EEG fluctuation feature data, a spatial difference matrix is ​​generated according to the channel positions of the multi-channel EEG acquisition device, wherein the spatial difference matrix is ​​used to represent the Euclidean distance between the channel positions of the multi-channel EEG acquisition device; S24. For the target learner's EEG fluctuation feature data, for each of the feature frequency bands, extract the corresponding center frequency, calculate the frequency difference between the center frequencies of each feature frequency band, and generate a frequency band difference matrix based on the frequency difference between the center frequencies of each feature frequency band, wherein the frequency band difference matrix is ​​used to represent the oscillation feature differences between each of the feature frequency bands. S25. Obtain preset fusion weights, perform tensor product expansion on the spatial difference matrix and the frequency band difference matrix, and perform matrix fusion on the tensor product-expanded spatial difference matrix and the tensor product-expanded frequency band difference matrix according to the fusion weights to obtain the overall difference matrix. S26. Flatten the target learner's EEG fluctuation feature data into a one-dimensional vector, and normalize the flattened target learner's EEG fluctuation feature data to obtain the corresponding current EEG feature distribution vector. S27. Flatten the EEG fluctuation emotion data in each of the emotion recognition reference sources into a one-dimensional vector, and normalize the flattened EEG fluctuation emotion data to obtain the corresponding target reference EEG feature distribution vector. S28. Between the current EEG feature distribution vector and each of the target reference EEG feature distribution vectors, the overall difference matrix is ​​used as a transmission distance constraint to solve the regularized transmission distance between the current EEG feature distribution vector and each of the target reference EEG feature distribution vectors, and the optimal regularized transmission distance is selected from each regularized transmission distance so as to use the emotion recognition reference source corresponding to the optimal regularized transmission distance as the target reference source.

[0033] It should be noted that the intelligent learning suggestion recommendation method based on multimodal emotion recognition described in this embodiment uses an emotion recognition reference source to represent a type of emotion. This emotion category includes at least positive emotions, negative emotions, and insignificant emotions. Determining this target reference source can significantly improve the stability and accuracy of subsequent emotion recognition, quickly locating the learner's emotion within the range of positive, negative, or insignificant emotions, avoiding interference from extreme data on the overall recognition, and improving the efficiency of learner emotion recognition. This enhances the practicality of the intelligent learning suggestion recommendation method in this embodiment.

[0034] Furthermore, the intelligent learning suggestion recommendation method based on multimodal emotion recognition provided in this embodiment eliminates emotion recognition reference sources that differ too much from the current emotion category of the target learner by determining the target reference source. This avoids the influence of messy emotion categories on subsequent emotion recognition and ensures that channel-level feature mixing and random replacement can be performed among samples of the same emotion category during subsequent feature enhancement. This not only significantly increases the diversity of samples but also effectively weakens the excessive dependence of subsequent models on the distribution features in specific emotion recognition reference sources. As a result, the intelligent learning suggestion recommendation method in this embodiment has a strong emotion recognition generalization ability and improves the accuracy and robustness of emotion recognition.

[0035] In one possible implementation, step S2, using the target reference source, performs feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data. This can be, but is not limited to, decomposed into the following steps S29-S210, specifically including: S29. Obtain the preset enhancement mixing weights, and add the EEG fluctuation emotion data in the target reference source to the target learner's EEG fluctuation feature data according to the enhancement mixing weights to obtain the target learner's enhanced EEG fluctuation feature data. S210. Obtain the feature enhancement ratio According to the aforementioned feature enhancement ratio Randomly select from the target learner's EEG fluctuation feature data the feature enhancement ratio. The corresponding multiple feature data form the pre-target learner's brain wave fluctuation enhancement feature data, and the target learner's brain wave fluctuation enhancement hybrid feature data is used to fill the pre-target learner's brain wave fluctuation enhancement feature data to obtain the target learner's brain wave fluctuation enhancement hybrid feature data.

[0036] It should be noted that when adding the target learner's EEG fluctuation feature data in a mixed manner, the EEG fluctuation emotion data should first be paired one by one with the target learner's EEG fluctuation feature data according to the EEG acquisition channel to form a data pair corresponding to each EEG acquisition channel. Then, according to the enhanced mixing weight, the data pair corresponding to each EEG acquisition channel is weighted and summed to obtain the target learner's enhanced mixed EEG fluctuation feature data corresponding to each EEG acquisition channel.

[0037] In one possible implementation, step S3 involves performing feature enhancement processing on the target learner's facial expression feature data and speech feature data in the multimodal emotion feature data to obtain target learner's enhanced facial expression feature data and enhanced speech feature data. Then, feature fusion is performed on the target learner's enhanced EEG fluctuation feature data, facial expression feature data, and speech feature data to obtain the target learner's multimodal emotion recognition features. This can be, but is not limited to, the following steps S31-S33, specifically including: S31. Obtain a pre-trained long-short-term attention model for facial expression features, and use the long-short-term attention model for facial expression features to perform attention calculation on the target learner's facial expression feature data in the multimodal emotion feature data to obtain the corresponding target learner's facial expression enhancement feature data. S32. Obtain a pre-trained long-short-term attention model for speech features, and use the speech feature long-short-term attention model to perform attention calculation on the target learner speech feature data in the multimodal emotion feature data to obtain the corresponding target learner speech enhancement feature data; S33. The target learner's enhanced EEG fluctuation hybrid feature data, enhanced facial expression feature data, and enhanced speech feature data are sequentially concatenated to generate multimodal emotion recognition features for the target learner.

[0038] It should be noted that, in this embodiment, the concatenation of the target learner's EEG fluctuation enhanced hybrid feature data, the target learner's facial expression enhanced feature data, and the target learner's speech enhanced feature data can be achieved, in one possible but not limited implementation, by concatenating and projecting the target learner's facial expression enhanced feature data and the target learner's speech enhanced feature data through a preset multilayer perceptron (MLP) to fuse them into a unified non-EEG multimodal emotion recognition feature for the target learner. Then, the target learner's EEG fluctuation enhanced hybrid feature data and the target learner's non-EEG multimodal emotion recognition feature are concatenated according to dimensions to obtain the final target learner multimodal emotion recognition feature.

[0039] In one possible implementation, step S4 involves obtaining a pre-trained target multimodal emotion recognition model and inputting the target learner's multimodal emotion recognition features into the target multimodal emotion recognition model to output the target learner's real-time emotion state category. This step can be broken down into, but is not limited to, the following steps S41-S44, specifically including: S41. Obtain a pre-trained target multimodal emotion recognition model, input the target learner's multimodal emotion recognition features as input, and output the target learner's emotion probability distribution map through the target multimodal emotion recognition model, wherein the emotion probability distribution map includes at least one emotion recognition result and the corresponding emotion recognition result confidence. S42. Obtain a preset confidence threshold for emotion recognition results, extract the confidence of each emotion recognition result corresponding to the emotion recognition result from the emotion probability distribution map of the target learner, and use the confidence threshold for emotion recognition results to determine the confidence of each emotion recognition result corresponding to the emotion recognition result in the emotion probability distribution map, and obtain the determination result; S43. If the determination result is that there is at least one emotion recognition result whose confidence level exceeds the emotion recognition result confidence level threshold, then among all the emotion recognition results that exceed the emotion recognition result confidence level threshold, the emotion recognition result corresponding to the highest emotion recognition result confidence level is selected as the real-time emotion state category of the target learner. S44. If the determination result is that there is no emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold, then the target learner's EEG fluctuation feature data is again enhanced using the target reference source, and the target learner's EEG fluctuation enhancement feature data is updated to generate the corresponding target learner multimodal emotion recognition feature again based on the updated target learner EEG fluctuation enhancement feature data. The regenerated target learner multimodal emotion recognition feature is input into the target multimodal emotion recognition model, and the emotion probability distribution map output by the target multimodal emotion recognition model is thresholded using the emotion recognition result confidence threshold until the determination result is that there is at least one emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold.

[0040] It should be noted that the target multimodal emotion recognition model described in this embodiment is pre-trained based on the target reference source. This pre-training should be completed before the intelligent learning suggestion recommendation method provided in this embodiment begins. A corresponding multimodal emotion recognition model can be trained for each emotion recognition reference source to avoid interference from different types of emotions in the accurate identification of emotional states. In actual monitoring, these multimodal emotion recognition models can be selected according to the different target reference sources identified by different target learners in different time windows. The output of these multimodal emotion recognition models is the accurate emotion state category. For example, when the target reference source corresponds to the emotion category... When the emotion category is positive, the target multimodal emotion recognition model can output emotion categories such as comfort, happiness, and ecstasy. When the emotion category corresponding to the target reference source is negative, the target multimodal emotion recognition model can output emotion categories such as confusion, fatigue, disappointment, sadness, and lethargy. This precise identification method greatly improves the accuracy of emotion recognition, moving beyond a general range of emotions to a precise state, ensuring that the proposed learning suggestions are more in line with actual needs. For example, when the target learner's emotion category is comfort, a suggestion to extend the learning time is made; when the target learner's emotion category is fatigue, a suggestion to reduce the learning time is made.

[0041] In one possible implementation, step S4, based on the real-time emotional state category of the target learner, generates corresponding recommended learning suggestions, which are then fed back to the learning terminal device to remind the target learner to adjust their learning state. This can be broken down into, but is not limited to, the following steps S45-S46, specifically including: S45. Obtain a preset learning suggestion recommendation table, use the real-time emotional state category of the target learner as the query condition, and retrieve the corresponding learning suggestions from the learning suggestion recommendation table as recommended learning suggestions; S46. Generate a corresponding display control signal based on the recommended learning suggestions, and feed the display control signal back to the learning terminal device so that the recommended learning suggestions can be visualized through the display interface of the learning terminal device, thereby completing the reminder to adjust the learning status of the target learner.

[0042] like Figure 2 As shown, the second aspect of this embodiment provides a hardware system for implementing the intelligent learning suggestion recommendation method based on multimodal emotion recognition described in the first aspect of the embodiment, including: The data acquisition unit is used to acquire the target learner's raw multimodal emotion data, perform feature extraction and data preprocessing on the raw multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes the target learner's EEG fluctuation feature data, the target learner's facial expression feature data and the target learner's speech feature data; The EEG data enhancement unit is used to acquire historical EEG fluctuation data, form multiple emotion recognition reference sources based on the historical EEG fluctuation data, compare the target learner's EEG fluctuation feature data in the multimodal emotion feature data with each of the emotion recognition reference sources, select the emotion recognition reference source with the highest similarity as the target reference source, and use the target reference source to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data. The non-EEG data augmentation unit is used to perform feature augmentation processing on the target learner's facial expression feature data and the target learner's speech feature data in the multimodal emotion feature data, respectively, to obtain target learner's facial expression augmentation feature data and target learner's speech augmentation feature data; The emotion recognition feature fusion unit is used to fuse the target learner's enhanced EEG fluctuation feature data, enhanced facial expression feature data, and enhanced speech feature data to obtain the target learner's multimodal emotion recognition features. An emotion recognition and suggestion unit is used to acquire a pre-trained target multimodal emotion recognition model, input the target learner's multimodal emotion recognition features into the target multimodal emotion recognition model to output the target learner's real-time emotion state category, and generate corresponding recommended learning suggestions based on the target learner's real-time emotion state category, so as to feed the recommended learning suggestions back to the learning terminal device, thereby reminding the target learner to adjust their learning state. The target multimodal emotion recognition model is pre-trained using the target reference source.

[0043] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0044] like Figure 3 As shown, the third aspect of this embodiment provides an electronic device, including: a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect of the embodiment.

[0045] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0046] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee transceiver (a low-power LAN protocol based on the IEEE 802.15.4 standard), a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0047] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0048] The fourth aspect of this embodiment provides a storage medium that stores instructions containing the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect of the embodiment.

[0049] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0050] The working process, working details, and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0051] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the intelligent learning suggestion recommendation method based on multimodal emotion recognition as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0052] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent learning suggestion recommendation based on multimodal emotion recognition, characterized in that, include: The original multimodal emotion data of the target learner is obtained, and feature extraction and data preprocessing are performed on the original multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes target learner EEG fluctuation feature data, target learner facial expression feature data and target learner speech feature data; Historical EEG fluctuation data is acquired, and multiple emotion recognition reference sources are formed based on the historical EEG fluctuation data. The target learner's EEG fluctuation feature data in the multimodal emotion feature data is compared with each of the emotion recognition reference sources. The emotion recognition reference source with the highest similarity is selected as the target reference source. The target reference source is then used to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data. The target learner's facial expression feature data and speech feature data in the multimodal emotion feature data are respectively subjected to feature enhancement processing to obtain target learner's facial expression enhancement feature data and target learner's speech enhancement feature data. The target learner's EEG fluctuation enhancement feature data, target learner's facial expression enhancement feature data and target learner's speech enhancement feature data are then fused to obtain the target learner's multimodal emotion recognition features. A pre-trained target multimodal emotion recognition model is obtained. The multimodal emotion recognition features of the target learner are input into the target multimodal emotion recognition model to output the real-time emotion state category of the target learner. Based on the real-time emotion state category of the target learner, corresponding recommended learning suggestions are generated and fed back to the learning terminal device to remind the target learner to adjust their learning state. The target multimodal emotion recognition model is pre-trained using the target reference source.

2. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 1, characterized in that, Obtain the raw multimodal emotion data of the target learner, perform feature extraction and data preprocessing on the raw multimodal emotion data to obtain multimodal emotion feature data, including: Within a preset time window, multi-channel EEG acquisition devices are used to collect multi-channel EEG fluctuation data of the target learner from the EEG regions of the target learner, and the EEG fluctuation data is used as the original EEG fluctuation data of the target learner. The EEG regions include at least the frontal region and the central region. Within a preset time window, the learning terminal device captures the target learner's facial video stream through its camera and the target learner's speech stream through its microphone array. The target learner's facial video stream is used as the original target learner's facial expression data, and the target learner's speech stream is used as the original target learner's speech data. The original target learner's facial expression data includes multiple frames of the target learner's facial expression images. The original target learner's EEG fluctuation data, facial expression data, and speech data are integrated, and time alignment processing is performed on the original target learner's EEG fluctuation data, facial expression data, and speech data to form the target learner's original multimodal emotion data. Obtain a preset sliding time window, and according to the sliding time window, perform time-sliding segmentation on the original target learner EEG fluctuation data in the original multimodal emotion data to form multiple time window samples; Multiple preset feature frequency bands are obtained. Differential entropy features are extracted from each time window sample according to each feature frequency band to obtain the differential entropy features of each feature frequency band corresponding to each time window. In each time window, the differential entropy features of each feature frequency band are concatenated to form the time window EEG fluctuation feature data to form the time window EEG fluctuation feature data corresponding to each time window. The EEG fluctuation feature data corresponding to each time window are spliced ​​together to generate the EEG fluctuation feature data of the pre-target learner. A pre-trained facial expression feature recognition model is obtained. The original target learner facial expression data in the original multimodal emotion data is used as input. The facial expression images of the target learner are input into the facial expression feature recognition model frame by frame. The facial expression feature recognition model outputs the facial expression feature recognition result corresponding to each frame of the target learner's facial expression image. The facial expression feature recognition result corresponding to each frame of the target learner's facial expression image is then concatenated to form the pre-target learner facial expression feature data. Audio waveform features are extracted from the original target learner speech data in the original multimodal emotion data to form pre-target learner speech feature data; The pre-target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data are preprocessed to form target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data. These three data are then integrated to form multimodal emotion feature data.

3. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 2, characterized in that, The pre-target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data are preprocessed to form target learner's EEG fluctuation feature data, facial expression feature data, and speech feature data, including: Anomaly detection is performed on the pre-target learner's EEG fluctuation feature data, pre-target learner's facial expression feature data, and pre-target learner's speech feature data to determine whether data anomalies and / or data missing occur. If the EEG fluctuation feature data of the pre-target learner shows abnormalities and / or missing data, the EEG fluctuation feature data, facial expression feature data, and speech feature data of the pre-target learner are discarded, and the original multimodal emotion data is collected and feature extracted again. The EEG fluctuation feature data of the pre-target learner obtained again is subjected to anomaly detection until the EEG fluctuation feature data of the pre-target learner does not show abnormalities and / or missing data. If the pre-target learner's EEG fluctuation feature data does not show any data abnormalities and / or missing data, but the pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data show data abnormalities and / or missing data, then the pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data are discarded. Based on the pre-target learner's EEG fluctuation feature data, missing data supplementation conditions are generated. The discarded pre-target learner's facial expression feature data and / or the pre-target learner's speech feature data are then supplemented using the missing data supplementation conditions to obtain supplemented target learner's facial expression feature data and / or supplemented target learner's speech feature data. The pre-target learner's EEG fluctuation feature data is used as the target learner's EEG fluctuation feature data, the supplemented target learner's facial expression feature data is used as the target learner's facial expression feature data, and the supplemented target learner's speech feature data is used as the target learner's speech feature data. If the pre-target learner's EEG fluctuation feature data, pre-target learner's facial expression feature data, and pre-target learner's speech feature data do not show any data anomalies and / or missing data, then the pre-target learner's EEG fluctuation feature data will be used as the target learner's EEG fluctuation feature data, the pre-target learner's facial expression feature data will be used as the target learner's facial expression feature data, and the pre-target learner's speech feature data will be used as the target learner's speech feature data.

4. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 3, characterized in that, Based on the pre-target learner's EEG fluctuation characteristics data, missing conditions are generated, including: When the facial expression feature data of the pre-target learner shows data anomalies and / or data missing, and the speech feature data of the pre-target learner does not show data anomalies and / or data missing, the EEG fluctuation feature data and the speech feature data of the pre-target learner are concatenated into a usable modality feature vector, and random noise is added to the usable modality feature vector to obtain missing compensation conditions. When the pre-target learner's speech feature data shows data anomalies and / or data missing, and the pre-target learner's facial expression feature data does not show data anomalies and / or data missing, the pre-target learner's EEG fluctuation feature data and the pre-target learner's facial expression feature data are concatenated into a usable modality feature vector, and random noise is added to the usable modality feature vector to obtain missing compensation conditions; When both the facial expression feature data and the speech feature data of the pre-target learner show data anomalies and / or data missing, an available modality feature vector is generated based on the EEG fluctuation feature data of the pre-target learner, and random noise is added to the available modality feature vector to obtain missing compensation conditions; Accordingly, supplementing the discarded facial expression feature data and / or speech feature data of the pre-target learner using the missing data supplementation conditions includes: A pre-trained conditional noise back-inference model is obtained. The missing supplementation conditions are used as input to the conditional noise back-inference model to perform multiple rounds of back-inference on the random noise in the missing supplementation conditions, so as to obtain the missing modality supplementation feature vector. The missing modality supplementation feature vector is used to supplement the facial expression feature data and / or the speech feature data of the pre-target learner, so as to obtain supplemented target learner facial expression feature data and / or supplemented target learner speech feature data.

5. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 2, characterized in that, Historical EEG fluctuation data is acquired, and multiple emotion recognition reference sources are formed based on the historical EEG fluctuation data. The target learner's EEG fluctuation feature data in the multimodal emotion feature data is compared with each of the emotion recognition reference sources, and the emotion recognition reference source with the highest similarity is selected as the target reference source, including: Acquire historical EEG fluctuation data, wherein the historical EEG fluctuation data includes multiple segments of EEG fluctuation data of the target learner and corresponding emotion tags; According to the emotion label corresponding to each segment of EEG fluctuation data, the EEG fluctuation data segments are classified to form multiple segments of EEG fluctuation emotion data, so that each segment of EEG fluctuation emotion data can be used as an emotion recognition reference source, wherein each of the emotion recognition reference sources is used to represent a type of emotion. For the target learner's EEG fluctuation feature data, a spatial difference matrix is ​​generated according to the channel positions of the multi-channel EEG acquisition device, wherein the spatial difference matrix is ​​used to represent the Euclidean distance between the channel positions of the multi-channel EEG acquisition device; For the target learner's EEG fluctuation feature data, for each of the feature frequency bands, the corresponding center frequency is extracted, the frequency difference between the center frequencies of each feature frequency band is calculated, and a frequency band difference matrix is ​​generated based on the frequency difference between the center frequencies of each feature frequency band, wherein the frequency band difference matrix is ​​used to represent the oscillation feature differences between each of the feature frequency bands. Obtain a preset fusion weight, perform tensor product expansion on the spatial difference matrix and the frequency band difference matrix, and perform matrix fusion on the tensor product-expanded spatial difference matrix and the tensor product-expanded frequency band difference matrix according to the fusion weight to obtain the overall difference matrix. The target learner's EEG fluctuation feature data is flattened into a one-dimensional vector, and the flattened target learner's EEG fluctuation feature data is normalized to obtain the corresponding current EEG feature distribution vector. The EEG fluctuation emotion data in each of the emotion recognition reference sources are flattened into a one-dimensional vector, and the flattened EEG fluctuation emotion data are normalized to obtain the corresponding target reference EEG feature distribution vector. Between the current EEG feature distribution vector and each of the target reference EEG feature distribution vectors, the overall difference matrix is ​​used as a transmission distance constraint. The regularized transmission distance between the current EEG feature distribution vector and each of the target reference EEG feature distribution vectors is solved respectively. The optimal regularized transmission distance is selected from each regularized transmission distance, and the emotion recognition reference source corresponding to the optimal regularized transmission distance is used as the target reference source.

6. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 5, characterized in that, Using the target reference source, feature enhancement processing is performed on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data, including: Obtain the preset enhancement mixing weights, and then add the EEG fluctuation emotion data in the target reference source to the target learner's EEG fluctuation feature data according to the enhancement mixing weights to obtain the target learner's enhanced EEG fluctuation feature data. Obtain feature enhancement ratio According to the aforementioned feature enhancement ratio Randomly select from the target learner's EEG fluctuation feature data the feature enhancement ratio. The corresponding multiple feature data form the pre-target learner's brain wave fluctuation enhancement feature data, and the target learner's brain wave fluctuation enhancement hybrid feature data is used to fill the pre-target learner's brain wave fluctuation enhancement feature data to obtain the target learner's brain wave fluctuation enhancement hybrid feature data.

7. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 1, characterized in that, The target learner's facial expression feature data and speech feature data in the multimodal emotion feature data are respectively subjected to feature enhancement processing to obtain target learner's enhanced facial expression feature data and target learner's enhanced speech feature data. Then, the target learner's enhanced EEG fluctuation feature data, the target learner's enhanced facial expression feature data, and the target learner's enhanced speech feature data are fused to obtain the target learner's multimodal emotion recognition features, including: Obtain a pre-trained long-short-term attention model for facial expression features, and use the long-short-term attention model for facial expression features to perform attention calculation on the target learner's facial expression feature data in the multimodal emotion feature data to obtain the corresponding target learner's facial expression enhancement feature data. Obtain a pre-trained long-short-term attention model for speech features, and use the speech feature long-short-term attention model to perform attention calculation on the target learner speech feature data in the multimodal emotion feature data to obtain the corresponding target learner speech enhancement feature data; The target learner's enhanced EEG fluctuation hybrid feature data, enhanced facial expression feature data, and enhanced speech feature data are sequentially concatenated to generate multimodal emotion recognition features for the target learner.

8. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 1, characterized in that, Obtain a pre-trained target multimodal emotion recognition model, input the target learner's multimodal emotion recognition features into the target multimodal emotion recognition model, and output the target learner's real-time emotion state category, including: A pre-trained target multimodal emotion recognition model is obtained, and the target learner's multimodal emotion recognition features are used as input to the target multimodal emotion recognition model. The target multimodal emotion recognition model outputs the target learner's emotion probability distribution map, wherein the emotion probability distribution map includes at least one emotion recognition result and the corresponding emotion recognition result confidence. Obtain a preset confidence threshold for emotion recognition results, extract the confidence of each emotion recognition result corresponding to the emotion recognition result from the emotion probability distribution map of the target learner, and use the confidence threshold for emotion recognition results to determine the confidence of each emotion recognition result corresponding to the emotion recognition result in the emotion probability distribution map, and obtain the determination result; If the determination result is that there is at least one emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold, then among all the emotion recognition results that exceed the emotion recognition result confidence threshold, the emotion recognition result with the highest emotion recognition result confidence level is selected as the real-time emotion state category of the target learner. If the determination result is that there is no emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold, then the target learner's EEG fluctuation feature data is again enhanced using the target reference source, and the target learner's enhanced EEG fluctuation feature data is updated. Based on the updated target learner's enhanced EEG fluctuation feature data, the corresponding target learner multimodal emotion recognition feature is regenerated. The regenerated target learner multimodal emotion recognition feature is input into the target multimodal emotion recognition model, and the emotion probability distribution map output by the target multimodal emotion recognition model is thresholded using the emotion recognition result confidence threshold until the determination result is that there is at least one emotion recognition result whose confidence level exceeds the emotion recognition result confidence threshold.

9. The intelligent learning suggestion recommendation method based on multimodal emotion recognition according to claim 8, characterized in that, Based on the real-time emotional state category of the target learner, corresponding recommended learning suggestions are generated and fed back to the learning terminal device to remind the target learner to adjust their learning state, including: Obtain a preset learning suggestion recommendation table, use the real-time emotional state category of the target learner as the query condition, and retrieve the corresponding learning suggestions from the learning suggestion recommendation table as recommended learning suggestions; Based on the recommended learning suggestions, a corresponding display control signal is generated and fed back to the learning terminal device so that the recommended learning suggestions can be visualized through the display interface of the learning terminal device, thereby completing the adjustment reminder for the learning status of the target learner.

10. An intelligent learning suggestion recommendation system based on multimodal emotion recognition, characterized in that, The intelligent learning suggestion recommendation method based on multimodal emotion recognition, as described in any one of claims 1 to 9, includes: The data acquisition unit is used to acquire the target learner's raw multimodal emotion data, perform feature extraction and data preprocessing on the raw multimodal emotion data to obtain multimodal emotion feature data, wherein the multimodal emotion feature data includes the target learner's EEG fluctuation feature data, the target learner's facial expression feature data and the target learner's speech feature data; The EEG data enhancement unit is used to acquire historical EEG fluctuation data, form multiple emotion recognition reference sources based on the historical EEG fluctuation data, compare the target learner's EEG fluctuation feature data in the multimodal emotion feature data with each of the emotion recognition reference sources, select the emotion recognition reference source with the highest similarity as the target reference source, and use the target reference source to perform feature enhancement processing on the target learner's EEG fluctuation feature data to generate target learner's enhanced EEG fluctuation feature data. The non-EEG data augmentation unit is used to perform feature augmentation processing on the target learner's facial expression feature data and the target learner's speech feature data in the multimodal emotion feature data, respectively, to obtain target learner's facial expression augmentation feature data and target learner's speech augmentation feature data; The emotion recognition feature fusion unit is used to fuse the target learner's enhanced EEG fluctuation feature data, enhanced facial expression feature data, and enhanced speech feature data to obtain the target learner's multimodal emotion recognition features. An emotion recognition and suggestion unit is used to acquire a pre-trained target multimodal emotion recognition model, input the target learner's multimodal emotion recognition features into the target multimodal emotion recognition model to output the target learner's real-time emotion state category, and generate corresponding recommended learning suggestions based on the target learner's real-time emotion state category, so as to feed the recommended learning suggestions back to the learning terminal device, thereby reminding the target learner to adjust their learning state. The target multimodal emotion recognition model is pre-trained using the target reference source.

Images