Multi-modal hypergraph emotion recognition method for video learning scene

Abstract

The invention discloses a multi-modal hypergraph emotion recognition method for a video learning scene, and the method is characterized in that the method comprises the following steps: 1), data collection; the method comprises the steps of (1) data acquisition, (2) data preprocessing, (3) feature extraction, (4) multi-modal hypergraph construction and fusion, (5) hypergraph learning and emotion recognition and (6) testing and evaluation.By means of the method, the emotion recognition accuracy can be improved, model parameters can be greatly reduced due to the fact that only eye movement features are adopted in the training stage, and the training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to a hypergraph neural network and a multimodal emotion recognition technology, and in particular to an online learning multimodal emotion recognition technology that uses a hypergraph neural network to capture high-order relationships in data, and specifically is a multimodal hypergraph emotion recognition method for video learning scenarios. Background Art

[0002] Multimodal emotion recognition refers to the ability to identify and understand human emotional states using data in a variety of different forms (such as voice, text, images, and physiological signals). This technology uses comprehensive analysis of multiple information sources to more accurately capture and understand the complexity of human emotions. The research purpose of multimodal emotion recognition is to improve the accuracy and comprehensiveness of emotion recognition: by integrating multiple data sources, emotions can be more comprehensively and accurately identified and understood, avoiding the limitations of a single modality. Data from different modalities can further improve the overall accuracy of emotion recognition through information complementarity between modalities. For example, some types of emotional states change more obviously in the visual modality, while other types of emotional states change more clearly in audio. Since in the video learning scenario, learners' emotions are induced by the content of the teaching video, it is necessary to use physiological changes as the main basis for emotion recognition in this scenario. Physiological signals are highly correlated and highly reliable evaluation indicators of human emotional states. Because physiological signals are regulated by the parasympathetic nervous system and are not easily affected by human subjective consciousness, physiological signals are more objective and accurate than emotional data from other modalities (such as facial expressions, voice, etc.). Based on the above analysis, the multimodal relationships in the teaching videos are integrated to guide the deep learning model to better explore the changes in learners' eye movement signals during the video learning process, so as to achieve more effective emotional state recognition.

[0003] Hypergraph neural network refers to the construction of hypergraph structure to capture the high-order relations implicit in the data (i.e., the adjacent or identical connections / paths between vertices) during the training process to improve the effective aggregation of information. Hypergraph is an extension of general graph, in which hyperedges can connect any number of vertices. Therefore, it is suitable for modeling high-order connections / correlations in data and has developed rapidly. The learning process of hypergraph learning can be regarded as a propagation process on the hypergraph structure. Reasoning on the hypergraph aims to minimize the label differences between vertices with strong connections on the hypergraph. In recent years, hypergraph learning has been widely developed and applied in the field of multimodal emotion recognition. For sentiment analysis in video learning scenarios, most existing studies focus on the fusion methods at different levels such as data, features or decisions. However, such methods cannot capture the implicit high-order correlations between different modalities and different types of data, and it is difficult to model the emotion generation process of the human brain in video learning scenarios, ignoring the impact of multiple high-order relations in teaching videos on learners. Therefore, the main motivation of this work is to establish a cross-individual emotion recognition framework that uses hypergraph neural networks to fuse multimodal high-order relationships to improve the performance of physiological signal emotion recognition in video learning scenarios.

[0004] In recent years, due to the impact of technological innovation and the impact of the epidemic, social needs and personal pursuits have also undergone profound changes. Learning by watching teaching videos has become one of the important learning methods for people to adapt to social changes and achieve personal development. In the field of education, emotions can help teachers better evaluate students' learning status in the learning process, understand their attention focus, information processing methods and cognitive load in the learning process, so as to optimize teaching design and improve learning effects. However, the video learning method will cause the corresponding "emotional loss" problem, that is, the information transmission between teachers and students is one-way, and teachers cannot understand students' learning status in real time, nor can they make effective adjustments in the teaching process; and learners may have negative emotions when watching teaching videos, which will have a negative impact on learners' cognitive state and teaching effects. Therefore, using existing technologies to analyze teaching videos and identify learners' emotional states is of great significance for assisting teachers in teaching, improving teaching design, and improving teaching experience.

[0005] At present, the research on sentiment analysis of video learning scenarios mainly focuses on the learner's own emotional induction, ignoring the impact of multiple high-order relational information implicit in the teaching video itself on the learner's emotions. Therefore, it is necessary to study a video learning emotion recognition method that integrates multiple high-order relations to explore their impact on the learner's emotional state and improve the performance of emotional state recognition. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal hypergraph emotion recognition method for video learning scenarios in view of the deficiencies of the prior art. This method can not only improve the accuracy of emotion recognition, but also greatly reduce model parameters and improve training efficiency because only eye movement features are used in the training stage.

[0007] The technical solution for achieving the purpose of the present invention is:

[0008] A multimodal hypergraph emotion recognition method for video learning scenarios includes the following steps:

[0009] 1) Data collection: Four teaching videos were selected as emotion-inducing materials. After the experimental volunteers signed the informed consent form, they were asked to watch the teaching videos and their eye movement signals were collected simultaneously.

[0010] 1-1) Eye movement signal acquisition: When the volunteers watched the teaching video, a tablet eye tracker was used to collect the volunteers' gaze, eye saccades, pupil size, and eye movement trajectory data in real time without contact. The eye movement data was aligned with the teaching video in terms of timestamps, and finally the eye movement samples were divided into 3-second windows;

[0011] 1-2) Video segment extraction: Segment the teaching video according to the subtitle timestamp to obtain multiple segments for subsequent data processing and audio-visual feature extraction;

[0012] 2) Data preprocessing: Data processing of the multimodal physiological signal data and teaching video clips collected in step 1) includes:

[0013] 2-1) Eye movement data processing: During the experiment, there may be missing eye movement data. Missing eye movement data refers to missing data collected when blinking, closing eyes or lowering the head. To address this problem, linear interpolation is used to fill in the missing values, and baseline correction is performed on these filled eye movement data to ensure data quality based on the differences between subjects;

[0014] 2-2) Video segment data processing: The time length of the video segment is processed, and the short segment without video subtitles and sound is merged into the previous video segment. Then the frame number of the video segment is counted to obtain the corresponding frame number for the subsequent calculation of statistical features;

[0015] 3) Feature extraction: Extract features from preprocessed data, including:

[0016] 3-1) Statistical feature extraction of eye movement signals: In terms of eye movement data, we sorted out multiple eye movement time domain information including pupil diameter, fixation time, saccade duration, saccade speed, blink frequency, pupil diameter, and calculated the maximum value, minimum value, average value, edge factor, and standard deviation of these time domain information. The specific features are shown in Table 1, and the minimum-maximum value normalization of the data was performed as the characteristics of the volunteers' eye movement signals.

[0017] Table 1

[0018]

[0019]

[0020] 3-2) Statistical feature extraction of audio-visual information: including:

[0021] 3-2-1) Visual information extraction: Calculate the pixel change rate sequence of each frame in the video, and use the mean, maximum, minimum, median, standard deviation, variance and number of non-zero pixels of the pixel change sequence, a total of 7 statistical features, as the statistical features of the relevant visual images of the teaching video, and perform data normalization;

[0022] 3-2-2) Audio information extraction: For each frame in the video, a set of MFCC filters with different parameters are used to obtain 13 MFCC coefficients from the audio spectrum. The MFCC coefficients usually include 13: the first coefficient reflects the overall energy of the speech signal, the 2nd to 6th coefficients mainly reflect the resonance peak information of the speech, which is related to the timbre and sound quality of the speech, the 7th to 12th coefficients mainly reflect the frequency characteristics of the speech, which is related to the pitch and timbre of the speech, and the 13th coefficient: usually related to the overall shape or envelope of the speech. Then, the maximum value, minimum value, mean, range, median, standard deviation and variance of the MFCC coefficients, the maximum value, minimum value, mean, range, median, standard deviation and variance of the first-order differential of the MFCC coefficients, and the kurtosis factor are extracted, a total of 15 statistical features, 195 features are extracted from the 13 MFCC coefficients, and these features are used as teaching The statistical characteristics of auditory audio in learning videos are obtained, and data is standardized. Specifically, the original speech signal is preprocessed, including pre-emphasis, framing and windowing processes, to improve the effect of subsequent feature extraction. Then, each speech frame is subjected to short-time Fourier transform to obtain the frequency domain amplitude spectrum. Next, the frequency domain amplitude spectrum is mapped to the Mel frequency scale, and filtered through a set of triangular Mel filter groups to highlight the frequency bands that are more relevant to human auditory characteristics. The energy values ​​of the filtered frequency bands are logarithmically compressed to simulate the response characteristics of the human ear to sound. Finally, the compressed frequency band energy values ​​are subjected to discrete cosine transform DCT to obtain a 13-dimensional MFCC feature vector. These 13 MFCC coefficients reflect the total energy, low-frequency energy distribution, intermediate-frequency resonance peak characteristics, high-frequency sound quality characteristics and overall envelope characteristics of the speech signal, which can better describe the acoustic characteristics of the speech signal.

[0023] 3-3) Feature Dimensionality Reduction: In order to evaluate the correlation between these statistical features and emotional states and avoid information redundancy, the principal component analysis method PCA (Principal Component PCA is used to reduce the dimension of eye movement features, visual features and audio features. In PCA, each principal component contributes a certain proportion to the total variance of the original data. By checking the contribution rate of these principal components, it can be decided how many principal components to retain to represent the original data. Usually, the first N principal components with a cumulative contribution rate of 95% are selected to represent the principal component features of the original data, and the feature dimension is reduced to N dimensions, which can retain most of the information of the original data and greatly reduce the data dimension. In addition, since the features of the same modality are highly correlated, the eye movement statistical features, visual statistical features and audio statistical features are reduced in dimension respectively to retain the principal components with a contribution rate greater than 1%. Finally, 19 principal components are retained from the 38 statistical features of the eye movement signal as effective eye movement features of the eye movement signal, 19 principal components are retained from the 195 statistical features of the audio signal as effective audio features, and 3 principal components are retained from the 7 statistical features of the video image as effective video features.

[0024] 3-4) Deep feature extraction: Compared with traditional feature engineering, deep feature extraction can automatically learn more meaningful and expressive features. These abstract features can not only better describe the essential characteristics of the data, but also reveal the potential laws and internal mechanisms behind the data, which can significantly improve the performance of the model in understanding, recognition, classification or prediction tasks. Transformer can extract higher-order and more abstract feature representations from eye movement data, which not only contains basic eye movement information, but also reveals the deeper visual attention patterns and complex associations between vision and cognition in the human cognitive process, which helps to more accurately predict and analyze human visual attention behavior. Therefore, in the process of deep feature extraction, a layer of Transformer encoder is used to extract deep features from eye movement signals, and the extracted deep visual features are aligned with the deep audio features and the deep eye movement features in terms of sample quantity;

[0025] 4) Multimodal hypergraph construction and fusion: including:

[0026] 4-1) Hypergraph construction: The hypergraph structure is constructed in the following two ways:

[0027] 4-1-1) Feature space K-nearest neighbor construction method: By projecting the eigenvalue of each sample into the same feature space and calculating the distance between each sample and other samples, a hyperedge is constructed between the sample and its K similar samples;

[0028] 4-1-2) Sample attribute construction method: manually classify the subject, video clip, and video type to which each sample belongs, and construct hyperedges for the corresponding samples according to the category, and finally obtain multiple hypergraphs including eye movement hypergraph, visual hypergraph, audio hypergraph, and attribute hypergraph;

[0029] 4-2) Hypergraph fusion: Different modalities contain different emotional information. At the same time, data from different modalities can provide complementary emotional information. By fusing this information, the limitations of single-modality emotion recognition can be made up. Therefore, an adaptive fusion strategy is adopted to assign a separate weight to the hypergraph obtained in step 4-1-2) and perform hypergraph fusion. Adaptive adjustment is achieved during the learning process, which helps to integrate high-order relationships within multiple modalities. Finally, a fully connected layer is used to use the vertex features learned through hypergraph learning to classify the four emotions of interest, confusion, boredom, and happiness.

[0030] 5) Hypergraph learning and emotion recognition: The eye movement features are input into the hypergraph neural network, and the vertex→hyperedge→vertex convolution method is used for message transmission and vertex feature update. Finally, a fully connected layer is used to classify the vertex features learned through hypergraph learning into four emotions: interest, confusion, boredom, and happiness.

[0031] 6) Testing and evaluation: The results of steps 1 to 5) were comprehensively tested and evaluated to verify performance and effectiveness. The model was cross-validated, and the accuracy and F1 score were used to measure the recognition effect of the model. The number of parameters (NP) and floating-point operations per second (FLOPS) were used to measure the overall cost of the model:

[0032] Accuracy=(TP+TN) / (TP+TN+FP+FN),

[0033] F1-score=2*(Precision*Recall) / (Precision+Recall),

[0034] Where: Precision = TP / (TP+FP),

[0035] Recall = TP / (TP+FN).

[0036] TP (True Positive) is the number of samples correctly predicted as positive, that is, the model correctly predicts samples that are actually positive as positive; FP (False Positive) is the number of samples incorrectly predicted as positive, that is, the model incorrectly predicts samples that are actually negative as positive; TN (True Negative) is the number of samples correctly predicted as negative, that is, the model correctly predicts samples that are actually negative as negative; FN (False Negative) is the number of samples incorrectly predicted as negative, that is, the model incorrectly predicts samples that are actually positive as negative.

[0037] Inspired by the rapid development and application of hypergraph neural networks, the feature of this technical solution is that it mines the high-order relationships hidden in the video to guide the implicit influence of the teaching video on the learners. When the learners watch the teaching video, an eye tracker is used to collect their eye movement signals, and feature extraction is performed to obtain the eye movement signal features. At the same time, the teaching video is segmented and the audio-visual information is extracted to obtain the visual features and audio features. According to the extracted multiple modes, hypergraphs are constructed separately to form hyperedge groups related to each mode. Then, an adaptive fusion method is used to fuse multiple hyperedge groups to form the final hypergraph for training. Eye movement features are input in the training stage, and the final vertex features are used to identify the four emotional states of interest, confusion, boredom, and happiness. This technical solution not only improves the accuracy of emotion recognition, but also greatly reduces model parameters and improves training efficiency because only eye movement features are used in the training stage.

[0038] The beneficial effects of this technical solution are as follows:

[0039] 1. Utilize high-order relationships within multimodal information: Compared with most existing multimodal emotion recognition methods, this technical solution completes the emotion recognition task by constructing a hypergraph structure and performing deep learning, which helps capture and integrate the high-order relationships implicit in the data;

[0040] 2. Hypergraph construction of multimodal data: This technical solution uses eye movement data from volunteers and audio-visual information in stimulus materials to construct hypergraphs, capture high-order relationships within a single modality, and apply them to emotion recognition, thereby more comprehensively understanding the learner's emotional state;

[0041] 3. Multimodal hypergraph fusion: In view of the differences and complementarities of different modalities, this technical solution gives each hypergraph Figure 1 Adaptive weights are used to integrate multi-modal high-order relationships. This method can better explore the emotional importance of different modalities.

[0042] 4. Simplified modal fusion: In the process of multimodal fusion, this technical solution only fuses the high-order relations of multiple modalities, and fuses the multimodal high-order relations in the stimulus material with the high-order relations of the eye movement signal, rather than performing feature-level modal fusion, thereby reducing the number of model parameters and greatly improving the inference speed;

[0043] 5. The Transformer encoder is introduced to extract deep features. This method can more comprehensively mine the potential information and relationships in the data, provide richer information for emotion recognition, and has better information representation capabilities than traditional feature extraction methods.

[0044] 6. Applicable to online learning scenarios: This technical solution is particularly suitable for online learning environments and can make up for the lack of emotional interaction in online learning, thereby enhancing learners' learning experience and teaching effects;

[0045] 7. Effectively assist online learning: By analyzing learners’ emotional states, this technical solution is expected to help educators better adjust course content and teaching methods to meet students’ needs and improve overall teaching quality;

[0046] 8. Deepen understanding of the learning process: With the help of hypergraph neural network technology, this technical solution can provide deeper insights into teaching video content and learners' feedback, which will help improve teaching practice and deepen educational research.

[0047] In general, this application innovatively proposes a multimodal emotion recognition method using a hypergraph neural network for video learning scenarios, which fully models and utilizes high-order relationships in multimodal data, and will be used to identify the emotional state of learners. Compared with the prior art, this method not only provides a more comprehensive and accurate emotion recognition technology solution, but also the identification of emotional states can provide more useful references for the teaching content and design in teaching videos, and can better evaluate students' real-time cognitive load. The above innovations constitute the technical solution to be protected by this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the method in the embodiment;

[0049] Figure 2 A data flow diagram of the method in the embodiment;

[0050] Figure 3 4 is a network structure diagram of the deep feature extractor of eye movement signals in the embodiment. DETAILED DESCRIPTION

[0052] The content of the present invention is further described below in conjunction with the drawings and embodiments, but the present invention is not limited thereto.

[0053] Example:

[0054] Reference Figure 1 , Figure 2 , a multimodal hypergraph emotion recognition method for video learning scenarios, comprising the following steps:

[0055] 1) Data collection: Learners watch teaching videos on a computer screen, while the Tobii TX300 tablet eye tracker collects learners' eye movement signals, including fixations, eye saccades, pupil size, and eye movement trajectories, and counts the duration and number of each eye movement state. The eye tracker is used to collect learners' eye movement signals, achieving contactless collection, which will not interfere with learners during the collection process and has the potential to be applied in real educational scenarios;

[0056] Before officially watching the teaching video, learners need to watch the calibration points on the screen and calibrate. After the calibration is completed, the teaching video will start playing and the eye movement data will be collected at the same time. The sampling rate of the eye movement signal is 60Hz, that is, 60 samples per second.

[0057] 2) Preprocess the collected eye movement data and audio-visual information of the teaching video: Process the eye movement data: There may be missing data during the eye movement signal collection process, such as blinking, closing eyes or lowering the head. The missing values ​​are filled in by linear interpolation method:

[0058] The linear interpolation method formula is defined as:

[0059]

[0060] For the acquisition of video clips, a subtitle extraction tool is used to generate the start and end timestamps of each subtitle, and the teaching video is segmented according to the timestamps. For the clips that are too short, they are merged with the previous video clip to obtain the video clips for feature extraction, and the corresponding timestamps are recorded. They are aligned with the eye movement data in time, and the attributes such as learner number, teaching video clip, and teaching video type corresponding to the eye movement samples are extracted;

[0061] 3) Feature extraction: In this step, feature extraction is performed on the preprocessed data, including extracting the statistical features of eye movement signals and audio-visual data respectively, and using principal component analysis (PCA) to remove redundancy from the statistical features of the three modalities, and further using Transformer to mine its deep features;

[0062] Standard feature extraction of eye movement signals: In terms of eye movement data, multiple time domain features including pupil diameter, fixation time and saccade duration were extracted with a window size of 3s, that is, the maximum, minimum, mean, standard deviation and variance of these data were calculated every 3 seconds.

[0063] For audio feature extraction of video clips, for each frame, 13 MFCC coefficients are obtained from the audio spectrum through a set of MFCC filters using different parameters. Since the characteristics of the signal are difficult to express in the time domain, the Fourier transform is used to transfer the signal to the frequency domain. The spectrum is smoothed by the MFCC filter to eliminate the influence of harmonics. The MFCC coefficients are obtained using the discrete cosine transform (DCT). Then, the first-order differential of the MFCC coefficients is calculated. For a given time window, the maximum, minimum, mean, range, median, standard deviation and variance of the MFCC coefficients are extracted. The same statistical value of the first-order differential of the MFCC coefficients, the kurtosis factor, a total of 15 statistical features are shown in Table 1. 195 statistical features are extracted from the 13 MFCC coefficients:

[0064] Table 1

[0065]

[0066] For video image feature extraction, the pixel change rate sequence PCRS (Pixel Change Rate Sequence, PCRS for short) is used to capture the high-speed switching and rapid movement of the target in the teaching video. The PCRS feature can effectively represent the switching rate of the teaching video screen. The PCRS calculation formula is as follows:

[0067]

[0068] Z represents the pixel change level of two adjacent frames of images, At = X t -X t-1 Indicates the difference between two adjacent frames of image pixels, expressed as an absolute value, X t represents the pixel grayscale value matrix in frame t, n is the total number of pixels in the grayscale image, X t(i，j) is the element in the i-th row and j-th column of the matrix. Z is the average value of the sum of all the elements in the matrix. The Z value of each two adjacent images within the window time is calculated to obtain the pixel change level Z. seq =[Z 1 , Z 2 ,......Z u-1 ], where u is the number of image frames in the window,

[0069] There are 7 statistical features in total, namely Z seqThe maximum value, minimum value, median, standard deviation, mean, variance and number of non-zero pixels C are calculated as follows:

[0070]

[0071] A t is the difference matrix of two adjacent grayscale images, m is the number of grayscale images in the sampling window, is the number of non-zero elements in the matrix. Therefore, the larger the value of C, the greater the pixel change of the image within the time window. In addition, in order to evaluate the correlation between these features and emotional states, the principal component analysis (PCA) is used to reduce the dimension of the above features and retain the principal components that are more relevant to the emotional state. The first n principal components with a cumulative contribution rate of 90% are retained, and the remaining low-contribution and low-correlation component information is removed to remove redundancy. At the same time, since the features of the same modality are highly correlated, the principal components with a contribution rate greater than 1% are retained. Finally, 19 principal components are retained from the 38 features of the eye movement signal, 19 principal components are retained from the 195 features of the audio signal, and 3 principal components are retained from the 7 features of the video image. In terms of deep feature extraction, although statistical features contain basic characteristics in the data, deep features can further represent the potential patterns and relationships in the data, thereby facilitating subsequent tasks of classification or prediction. In this process, the eye movement signal, visual features, and audio features are passed through a Transformer encoder to obtain relevant deep features. The network structure is as follows: Figure 3 As shown in the figure, it includes position encoding, regularization and multi-head attention mechanism. The multi-head attention layer has 8 attention heads. The multi-head attention mechanism can focus on key information more effectively, which helps to fully explore the intrinsic information contained in the statistical features. The deep feature extractor can obtain deep features for subsequent hypergraph construction and hypergraph learning to identify emotional states.

[0072] 4) Multimodal hypergraph construction and fusion: The eye movement features are combined with the visual features and audio features of the video through the feature space K-nearest neighbor method to construct a hypergraph, and the high-order relationships within the modality are captured. Then, through the sample attribute construction method, the eye movement data belonging to the same subject are regarded as a type of high-order relationship, and the eye movement data belonging to the same video clip are regarded as a type of high-order relationship, and so on, to obtain an attribute hypergraph;

[0073] Multi-hypergraph fusion: Adopting adaptive weight strategy, assigning Figure 1 There are learnable weights w i , and then the corresponding incidence matrix H iAfter multiplying with the corresponding weights, the concatenation is performed to achieve adaptive adjustment during the learning and training process, integrate all hypergraphs, and organically integrate the high-order relationship of eye movement with the high-order relationship of video and audio, so as to comprehensively utilize the high-order relationship of different modalities and different attributes for emotion recognition;

[0074] 5) Hypergraph learning and sentiment classification: Hypergraph information transmission is achieved by using hypergraph spatial convolution (point→edge, edge→point), vertex features are updated, and the updated vertex features are classified using a fully connected layer to achieve classification of four emotions such as interest, confusion, boredom, and happiness to obtain the output of the learner's emotional state;

[0075] 6) Testing and Evaluation: The multimodal emotion recognition model using the hypergraph neural network was comprehensively tested and evaluated. The data of 12 learners out of 59 learners were randomly selected as the test set, and the rest were used for training and verification to verify its performance and effectiveness. The model was cross-validated, and the accuracy and F1-score indicators were used to measure the effect of the model. The test results are shown in Table 2:

[0076] Table 2

[0077] Comparison Method Average accuracy Average F1 score ResNet18 63.43％ 0.54 VGG16 66.95％ 0.65 Transformer 53.63％ 0.47 MTL-CNN 52.62％ 0.45 This patented method 82.64％ 0.76

[0078] The experimental results show that after capturing the high-order relations in the data through hypergraph construction, the accuracy of emotion recognition is improved by about 15% compared with the commonly used methods. This fully confirms the effectiveness of modeling high-order relations for emotion recognition. On this basis, the performance of the model is tested, and the performance of the model is measured by the FLOPS indicator of the model parameter quantity NP floating point calculation. The results are shown in the following table:

[0079] Comparison Method FLOPS NP ResNet18 8.36M 5.02M Transformer 12.73M 12.55M VGG16 56.43M 55.32M Methods in this example 31.17M 0.01M

[0080] The experimental results show that the hypergraph learning method can significantly reduce the number of model parameters, improve training efficiency, and accelerate reasoning, while ensuring a high accuracy rate in emotion recognition. This fully confirms the superiority of the hypergraph neural network used in this example in the task of emotion recognition.

Claims

1. A multimodal hypergraph emotion recognition method for video learning scenarios, characterized in that: The steps include: 1) Data collection: Four teaching videos were selected as emotion-inducing materials. After the experimental volunteers signed the informed consent form, they were asked to watch the teaching videos and their eye movement signals were collected simultaneously, including: 1-1) Eye movement signal acquisition: When the volunteers watched the teaching video, a tablet eye tracker was used to collect the volunteers' gaze, eye saccades, pupil size, and eye movement trajectory data in real time without contact. The eye movement data was aligned with the teaching video in terms of timestamps, and finally the eye movement samples were divided into 3-second windows; 1-2) Video segment extraction: Segment the teaching video according to the subtitle timestamp to obtain multiple segments; 2) Data preprocessing: Data processing of the multimodal physiological signal data and teaching video clips collected in step 1) includes: 2-1) Eye movement data processing: During the experiment, there will be missing eye movement data. Missing eye movement data refers to missing data collected when blinking, closing eyes or lowering the head. Linear interpolation is used to fill in the missing values, and baseline correction is performed on these filled eye movement data; 2-2) Video segment data processing: Process the time length of the video segment, merge the short segment without video subtitles and sound into the previous video segment, and then count the number of frames of the video segment to obtain the corresponding frame number; 3) Feature extraction: Extract features from preprocessed data, including: 3-1) Statistical feature extraction of eye movement signals: In terms of eye movement data, multiple eye movement time domain information including pupil diameter, fixation time, saccade duration, saccade speed, blink frequency, pupil diameter, etc. are sorted out, and the maximum value, minimum value, average value, edge factor, and standard deviation of these time domain information are calculated. The specific features are shown in Table 1, and the minimum-maximum value of the data is normalized as the characteristics of the volunteers' eye movement signals. Table 1: ； 3-2) Statistical feature extraction of audio-visual information: including: 3-2-1) Visual information extraction: Calculate the pixel change rate sequence of each frame in the video, and use the mean, maximum, minimum, median, standard deviation, variance and number of non-zero pixels of the sequence, a total of 7 statistical features, as the statistical features of the visual image of the teaching video, and perform minimum-maximum normalization; 3-2-2) Audio information extraction: For each frame in the video, a set of MFCC filters with different parameters are used to obtain MFCC coefficients from the audio spectrum, and then the maximum, minimum, mean, range, median, standard deviation and variance of the MFCC coefficients, the same statistical value of the first-order differential of the MFCC coefficients, and the statistical characteristics of the kurtosis factor are extracted. These characteristics are used as the statistical characteristics of the auditory audio of the teaching video, and the data is standardized; 3-3) Feature Dimensionality Reduction: The principal component analysis method PCA (Principal Component Analysis for short PCA) is used to reduce the feature dimension of eye movement features, visual features and audio features. In PCA, each principal component contributes a certain proportion to the total variance of the original data. By looking at the contribution rate of these principal components, it can be decided how many principal components to retain to represent the original data. The first N principal components with a cumulative contribution rate of 95% are selected to represent the principal component characteristics of the original data, and the feature dimension is reduced to N dimensions. The eye movement statistical features, visual statistical features and audio statistical features are reduced in dimension to retain the principal components with a contribution rate greater than 1%; 3-4) Deep feature extraction: In the process of deep feature extraction, a layer of Transformer encoder is used to extract deep features from eye movement signals, and the extracted deep visual features, audio deep features, and eye movement deep features are aligned in terms of sample quantity; 4) Multimodal hypergraph construction and fusion: including: 4-1) Hypergraph construction: The hypergraph structure is constructed in the following two ways: 4-1-1) Feature space K-nearest neighbor construction method: By projecting the feature value of each sample into the same feature space and calculating the distance between each sample and other samples, a hyperedge is constructed between the sample and its K similar samples; 4-1-2) Sample attribute construction method: manually classify the subject, video clip, and video type to which each sample belongs, and construct hyperedges for the corresponding samples according to the category, and finally obtain multiple hypergraphs including eye movement hypergraph, visual hypergraph, audio hypergraph, and attribute hypergraph; 4-2) Multimodal hypergraph fusion: Adopting an adaptive fusion strategy, each hypergraph obtained in step 4-1-2) is given a separate weight, and hypergraph fusion is performed, and adaptive adjustment is achieved during the learning process. 5) Hypergraph learning and emotion recognition: The eye movement features are input into the hypergraph neural network, and the vertex→hyperedge→vertex convolution method is used for message transmission and vertex feature update. Finally, a fully connected layer is used to classify the vertex features learned through hypergraph learning into four emotions: interest, confusion, boredom, and happiness. 6) Testing and evaluation: The results of steps 1 to 5) were comprehensively tested and evaluated to verify the performance and effectiveness. The model was cross-validated, and the accuracy and F1 score were used to measure the recognition effect of the model. The model parameter number NP and floating-point calculation number FLOPS were used to measure the overall cost of the model.

Images