A student psychological state detection method based on feature interaction learning network

Abstract

The application discloses a student psychological state detection method based on a feature interaction learning network, which comprises the following steps: obtaining facial expression video data and heart rate data of a test object when the test object watches a plurality of test videos with a single emotional feature; performing feature extraction on the facial expression video data and the heart rate data to obtain facial expression emotional features and heart rate features of the test object; fusing the facial expression emotional features and the heart rate features, and performing emotion recognition based on the fusion result to obtain an emotional category corresponding to each sample video; comparing an emotional feature label corresponding to each test video with the emotional category thereof, and combining comparison results of the test videos and a preset threshold condition to generate an emotion detection result of the test object. The application fuses facial expressions and heart rate signals to perform emotion recognition, improves the accuracy of emotion recognition and classification, and provides a scientific basis and support for college student psychological health state prediction, emotion management and intervention treatment.

Description

Technical Field

[0001] This application relates to the field of intelligent human-computer interaction technology, and more specifically, to a method, system, and computer device for detecting student psychological state based on a feature-based interactive learning network. Background Technology

[0002] With the overall development of my country's economy, people's material living standards have improved rapidly. While pursuing physical health, people are also paying more attention to mental health. For college students, mental health is the foundation for academic achievement, career success, and a happy life. Therefore, paying attention to the psychological development process of college students, understanding their mental health status, and correctly guiding them to overcome psychological obstacles, eliminate psychological shadows, and prevent mental illness should be the primary task of current mental health education for college students.

[0003] To strengthen students' mental health, Chinese universities typically collect data through questionnaires. However, this method is heavily influenced by students' subjective will, resulting in low accuracy. Emotional state can be reflected in responses to various psychological tests. However, some students may become perfunctory or resistant due to the sheer number of questions, introverted personalities, or pessimistic outlooks, making it difficult for relevant school departments to accurately assess students' mental health and provide more targeted and comprehensive intervention and assistance.

[0004] In addition, technologies and methods for predicting student mental health based on big data and artificial intelligence are gradually emerging. However, the more common prediction methods currently available do not achieve feature interaction, short-term emotional changes and details are difficult to capture, and factors such as false expressions displayed by the test students all affect the accuracy and objectivity of the prediction of students' mental health status. Summary of the Invention

[0005] To address at least one deficiency or improvement need in existing technologies, this invention provides a student psychological state detection method based on a feature-interactive learning network. This method simultaneously uses facial expression video data and heart rate data for emotion state perception, and performs emotion recognition and classification by fusing facial expression emotion features and heart rate features, which can greatly improve recognition accuracy.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for detecting student mental states based on a feature-interactive learning network is provided, comprising:

[0007] Acquire facial expression video data and heart rate data of subjects while watching multiple test videos with a single emotional characteristic;

[0008] Feature extraction is performed on the facial expression video data to obtain the facial expression emotion features of the subject.

[0009] Feature extraction is performed on the heart rate data to obtain the heart rate characteristics of the subject;

[0010] The facial expression emotion features are fused with heart rate features, and emotion recognition is performed based on the fusion result to obtain the emotion category corresponding to each sample video.

[0011] The emotion feature label corresponding to each test video is compared with its emotion category. The emotion detection results of the test subjects are generated by combining the comparison results of each test video with the preset threshold conditions.

[0012] Furthermore, in the aforementioned student psychological state detection method, the step of extracting features from facial expression video data to obtain the facial expression and emotional features of the test subject includes:

[0013] The facial expression video data is divided into several sub-videos of the same length according to time sequence, and the segment features corresponding to each sub-video are extracted based on the attention mechanism;

[0014] Perform linear projection on each segment feature to obtain the corresponding query vector, key vector, and value vector;

[0015] Construct the first query matrix, the first key matrix, and the first value matrix based on the query vector, key vector, and value vector of all fragment features;

[0016] Based on the first query matrix, the first key matrix, and the first value matrix, segment location encoding is applied to describe the location of each sub-video in the facial expression video data, resulting in the encoded first emotion feature, which serves as the facial expression emotion feature of the subject.

[0017] Furthermore, in the aforementioned student psychological state detection method, the extraction of segment features corresponding to each sub-video based on the attention mechanism includes:

[0018] For each sub-video, extract the feature vector of each frame;

[0019] Perform linear projection on the feature vector of each frame to obtain the corresponding query vector, key vector, and value vector;

[0020] Construct a second query matrix, a second key matrix, and a second value matrix based on the query vector, key vector, and value vector of all frames;

[0021] Calculate the general feature vector of the corresponding sub-video based on the second query matrix, the second key matrix, and the second value matrix;

[0022] Estimate the distance between the general feature vector and the feature vector of each frame, and reweight the feature vector of each frame based on the distance to obtain the refined feature vector of each frame.

[0023] The refined feature vectors of each frame are weighted and summed to obtain the segment features of the corresponding sub-video.

[0024] Furthermore, in the above-mentioned student psychological state detection method, the step of extracting features from heart rate data to obtain the heart rate characteristics of the subject includes:

[0025] The peak values ​​of heart rate data are extracted, and their mean, variance, root mean square value, standard deviation, and impulse factor are calculated based on the peak values ​​as the heart rate characteristics of the subjects.

[0026] Furthermore, in the aforementioned student psychological state detection method, before feature extraction from heart rate data, the following steps are also included:

[0027] Wavelet transform was used to preprocess the heart rate data to filter out heart rate noise.

[0028] Furthermore, in the aforementioned student psychological state detection method, the single emotional characteristic includes neutrality, happiness, sadness, surprise, fear, anger, and disgust.

[0029] According to a second aspect of the present invention, a student psychological state detection system based on a feature-interactive learning network is also provided, which includes an emotion recognizer and a detection unit;

[0030] The emotion recognizer includes a facial emotion recognition model, a physiological emotion recognition model, and a fusion network;

[0031] The facial emotion recognition model is used to extract features from the facial expression video data of the subject when watching a variety of sample videos with a single emotional feature, so as to obtain the facial expression emotion features of the subject.

[0032] The physiological emotion recognition model is used to extract features from the heart rate data of the subjects when they watch a variety of sample videos with a single emotion feature, so as to obtain the heart rate features of the subjects.

[0033] The fusion network is used to fuse the facial expression emotion features with the heart rate features, and to perform emotion recognition based on the fusion result, thereby obtaining the emotion category corresponding to each sample video.

[0034] The detection unit is used to compare the emotion feature label corresponding to each sample video with its emotion category, and generate the emotion detection result of the subject by combining the comparison results of each sample video and the preset threshold conditions.

[0035] Furthermore, the aforementioned student psychological state detection system also includes an infrared camera and a BVP sensor, which are used to collect facial expression video data and heart rate data of the test subjects, respectively.

[0036] Furthermore, in the aforementioned student psychological state detection system, the facial emotion recognition model adopts the Transformer model; and the physiological emotion recognition model adopts the Long Short-Term Memory artificial neural network model.

[0037] According to a third aspect of the present invention, a computer device is also provided, characterized in that it includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of any of the above-described student psychological state detection methods.

[0038] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0039] (1) This invention uses facial expression video data and heart rate data simultaneously in the perception of emotional state. It adopts a multimodal fusion model and uses an infrared camera and BVP sensor to accurately extract facial expression emotional features and heart rate features. It fuses facial expression emotional features and heart rate features to overcome the limitations of single detection dimension in traditional detection methods, reduce the interference of false expressions and low accuracy, and is of great significance to strengthening the construction of students' mental health in colleges and universities.

[0040] (2) The facial emotion recognition model constructed in this invention uses an expression segment transformer to achieve accurate video-based facial expression recognition; the attention-enhanced segment feature prediction method and the random segment sequence prediction method effectively solve the problems in Transformer-based facial expression recognition, such as focusing only on the visual cues of frames with peak expression changes, while ignoring a large amount of useful spatiotemporal information in other time periods of the video, which can greatly improve the recognition accuracy. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the student psychological state detection method based on a feature-interactive learning network provided in this embodiment.

[0043] Figure 2This is a flowchart illustrating the student psychological state detection method provided in this embodiment;

[0044] Figure 3 This is a schematic diagram illustrating a scenario where students undergo psychological testing, as provided in this embodiment.

[0045] Figure 4 This is a schematic diagram of the facial expression recognition mechanism provided in this embodiment;

[0046] Figure 5 This is a schematic diagram of the heart rate data recognition mechanism provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0048] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0049] Furthermore, to avoid obscuring the understanding of the invention by those skilled in the art, well-known or widely used techniques, elements, structures, and processes may not be described or shown in detail. Although the accompanying drawings illustrate exemplary embodiments of the invention, the drawings are not necessarily drawn to scale, and specific features may be enlarged or omitted to better illustrate and explain the invention.

[0050] Figure 1 This is a flowchart illustrating a student psychological state detection method based on a feature-interactive learning network provided in this embodiment. Figure 2 This is a flowchart of the student psychological state detection method provided in this embodiment. Please refer to it. Figure 1 , Figure 2 The method for assessing the student's psychological state mainly includes the following steps:

[0051] S1 acquires facial expression video data and heart rate data of subjects while watching sample videos with multiple single emotional characteristics;

[0052] In this embodiment, the subjects were students participating in a school mental health assessment. Figure 3 This is a schematic diagram of a scenario where a test subject participates in a mental health assessment, as provided in this embodiment.

[0053] In a specific example, an infrared camera records facial expression images of the student, obtaining facial expression video data; a BVP sensor records the student's heart rate changes, obtaining heart rate data. This infrared camera can be integrated into a computer, or it can be a standalone device mounted on a computer, or placed in other areas where effective facial expression video data of the student can be acquired. Figure 3 As shown, facial expression information can be captured using the built-in infrared camera on the computer screen.

[0054] In one alternative implementation, the single emotional characteristic includes seven categories: neutral, happy, sad, surprised, fearful, angry, and disgusted.

[0055] Specifically, the students watched videos featuring seven different single emotions (anger, sadness, disgust, fear, happiness, surprise, and neutrality), with each type of video played for one minute in this order. The corresponding video clips were taken from the following films:

[0056] Table 1. Materials for Emotional Testing

[0057]

[0058]

[0059] After watching each type of film clip, the students listened to a two-minute piece of light music to adjust their emotional state.

[0060] S2 performs feature extraction on the facial expression video data to obtain the facial expression emotion features of the subject.

[0061] In an optional implementation, feature extraction is performed on the facial expression video data to obtain the facial expression emotion features of the subject, including:

[0062] The facial expression video data is divided into several sub-videos of the same length according to time sequence, and the segment features corresponding to each sub-video are extracted based on the attention mechanism;

[0063] Perform linear projection on each segment feature to obtain the corresponding query vector, key vector, and value vector; construct the first query matrix, the first key matrix, and the first value matrix based on the query vector, key vector, and value vector of all segment features;

[0064] Based on the first query matrix, the first key matrix, and the first value matrix, segment location encoding is applied to describe the location of each sub-video in the facial expression video data, resulting in the encoded first emotion feature, which serves as the facial expression emotion feature of the subject.

[0065] In a specific example, facial expression video data and extracted heart rate data of the test subject are input into a trained emotion recognizer to extract the subject's current emotional features and heart rate characteristics. This emotion recognizer includes a facial emotion recognition model used to extract features from the facial expression video data to obtain the subject's facial expression emotion features. This facial emotion recognition model employs a Transformer model with an encoder and decoder.

[0066] Figure 4 This is a schematic diagram of the facial expression recognition mechanism provided in this embodiment; specifically, facial expression video data is decomposed into a series of segments to enhance the Transformer model's ability to simulate subtle visual changes in each segment. Formally, given an input facial expression video data C, it is decomposed into a series of smaller sub-videos: C = {C1, C2, ..., C...} i …,C n}, where C i Let C represent the i-th sub-video, and n be the total number of sub-videos. Each sub-video C i A segment refers to a sequence of several adjacent frames in a video. All segments have the same length and are sequential in chronological order.

[0067] An encoder is used to encode fragment features, and then a decoder transforms the encoded features into a more robust expression. For the fragment features R... i Perform linear projection to obtain the corresponding query vector q. i Key vector k i Sum vector v i All q i ,k i ,v i All are vectors of size d. Then, the query vectors, key vectors, and value vectors of different segments are combined to obtain a query matrix Q, a key matrix K, and a value matrix V, where Q, K, V ∈ R. n×d Based on the obtained Q, K, and V values, analysis is performed throughout the entire segment. Furthermore, segment position coding, such as sine and cosine position coding, is applied to describe the segment positions in the video.

[0068] The final encoder output is the encoded fragment feature H∈R n×d .

[0069]

[0070] Here, A represents the self-attention mechanism, and Q, K, and V are the query matrix, key matrix, and value matrix, respectively. In this embodiment, three encoder layers are used, with four attention heads in each layer.

[0071] In an optional implementation, segment features corresponding to each sub-video are extracted based on an attention mechanism, including:

[0072] For each sub-video, extract the feature vector of each frame;

[0073] Linear projection is performed on the feature vector of each frame to obtain the corresponding query vector, key vector, and value vector; a second query matrix, a second key matrix, and a second value matrix are constructed based on the query vector, key vector, and value vector of all frames.

[0074] Calculate the general feature vector of the corresponding sub-video based on the second query matrix, the second key matrix, and the second value matrix;

[0075] Estimate the distance between the general feature vector and the feature vector of each frame, and reweight the feature vector of each frame based on the distance to obtain the refined feature vector of each frame.

[0076] The refined feature vectors of each frame are weighted and summed to obtain the segment features of the corresponding sub-video.

[0077] Specifically, a two-level hierarchical attention structure is applied to simulate subtle changes in facial expressions. The first level extracts frame-level attention, while the second level focuses on extracting fragment-level global attention. For the first level, the global average pooling output of the deep convolutional neural network (DCNN) is extracted as the feature vector for each frame, using r... j,i Let I represent the feature vector of the j-th frame in the i-th segment. Using linear projection, the feature vector I is transformed into a query matrix I. Q Key Matrix I K Sum matrix I V Using the above formula (1), we obtain the feature I′∈R. j×d For the second-level structure, features I′ are summarized into a unified general feature vector. Then, the relationship between this general feature vector and the feature vectors of each frame is estimated. The obtained relationship is later used to reweight the feature vectors of each frame for refinement. Finally, the refined features are simplified into a single representation α. j The calculation is as follows:

[0078]

[0079] Here, ‖·‖ refers to the Euclidean norm. The refined eigenvector α is estimated... j This allows us to determine which frame contains more bias information and non-expression noise.

[0080] By reweighting and summarizing the feature vectors of each frame, the segment features of the corresponding sub-video are obtained:

[0081]

[0082] This embodiment also provides a training method for the facial emotion recognition model, which mainly includes:

[0083] Obtain facial expression sample video data with facial expression labels, input this facial expression sample video data into the facial emotion recognition model to be trained, and output the predicted facial expression category; the data processing and prediction process of this facial emotion recognition model is as follows:

[0084] The facial expression sample video data is divided into several sample sub-videos of the same length according to time sequence. The sample segment features corresponding to each sample sub-video are extracted based on the attention mechanism. This process is the same as the extraction of the segment features corresponding to the sub-videos above, and will not be repeated here.

[0085] Linear projection is performed on the features of each sample segment to obtain the corresponding query vector, key vector, and value vector; a third query matrix, a third key matrix, and a third value matrix are constructed based on the query vector, key vector, and value vector of all sample segment features.

[0086] Based on the third query matrix, the third key matrix, and the third value matrix, segment location encoding is applied to describe the location of each sample sub-video in the facial expression sample video data to obtain the encoded third emotion feature. Based on the third emotion feature, the facial expression category is predicted.

[0087] The loss function is calculated based on the predicted facial expression category and facial expression label. The model parameters of the facial emotion recognition model to be trained are adjusted in reverse until the iteration stopping condition is met. The iteration is then stopped, and the trained facial emotion recognition model is obtained.

[0088] In a more preferred embodiment, the training method for the facial emotion recognition model further includes the following steps:

[0089] The sample sub-videos after the facial expression sample video data is segmented are randomly arranged to obtain multiple facial expression sample video reconstructed data with different arrangement orders;

[0090] The fourth emotion feature corresponding to each facial expression sample video reconstruction data is extracted separately, and each of the fourth emotion features is fused with the third emotion feature. The facial expression category is then predicted based on the fused emotion feature.

[0091] To make the Transformer model more comprehensive and avoid ignoring subtle facial expressions in non-peak segments, this embodiment introduces a Random Segment Order Prediction (SSOP) with corresponding loss to assist in training.

[0092] SSOP (Sectional Segmentation of Expressions) is used to train a facial emotion recognition model. Segments are randomly arranged, and the model predicts the outcome. For example, with 7 segments, the randomized order could be S = {3, 2, 7, 1, 4, 6, 5}. In practice, 10 different randomized orders of varying types were generated. Then, segments are sampled from all generated orders and rearranged accordingly. These randomly ordered segments are then fed into the expression segment transformer within the facial emotion recognition model. After extracting the emotional information T, the emotional information T is compared with the emotional features H∈R corresponding to facial expression sample video data with normal temporal sequence. 7×d The features are fused to predict random ordering. Training the Transformer with SSOP involves maximizing the posterior probability (MAP) estimate, where the relevant conditional probability density function is:

[0093]

[0094] Among them, o i c refers to the feature vector of the i-th sample segment; i This represents the i-th sample segment.

[0095] In a specific example, training a facial expression recognition model includes the following steps:

[0096] (1) Construct a training sample set (80%) and a test sample set (20%);

[0097] (2) The facial expression recognition model is trained using the training sample set. The predicted facial expressions of the training samples and the real facial expressions are calculated using the cross-entropy function, and the network is optimized using the stochastic gradient descent method to improve the accuracy of the network.

[0098] (2.1) After weak data augmentation (WA), the facial expression video data is fed into the facial expression recognition model. The output results are compared with the true values, and the loss is calculated using the cross-entropy function as the loss function.

[0099]

[0100] Among them, YC C represents the facial expression label for each video, and C indexes a training video. This represents the probability of a facial expression predicted by EST (Expression Segment Transformer).

[0101] (2.2) Optimize the results using stochastic gradient descent:

[0102]

[0103]

[0104] (2.3) Using a loss function to identify the order of random segments:

[0105]

[0106] in, S represents the permutation type of the predicted random order. C It is a truth pair for all labels, indicating the correct permutation and combination type.

[0107] S3 performs feature extraction on the heart rate data to obtain the heart rate characteristics of the subject;

[0108] In this embodiment, the emotion recognizer includes a physiological emotion recognition model used to extract features from heart rate data to obtain the heart rate characteristics of the test student. Specifically, the peak value of the heart rate data is first extracted, and its mean, variance, root mean square value, standard deviation, and impulse factor are calculated based on the peak value as the heart rate characteristics of the test student.

[0109] In addition, wavelet transform was used to preprocess the heart rate data before feature extraction to filter out heart rate noise. Figure 5 This is a schematic diagram of the heart rate data recognition mechanism provided in the embodiments of this application.

[0110] (1) Preprocessing of heart rate feature data

[0111] (1.1) Since noise and signal are mixed together, a wavelet basis function is first selected, and wavelet transform is used to decompose the noisy heart rate signal at a certain scale to obtain the wavelet coefficients at each scale:

[0112]

[0113] Where a > 0, is the scale factor. τ reflects displacement, and its value can be positive or negative. a and τ are continuous variables. At different scales, the duration of the wavelet widens with increasing values, and the amplitude... It decreases inversely, but the shape of the wave remains unchanged.

[0114] (1.2) After the heart rate signal is decomposed by wavelet transform, the wavelet coefficients with larger amplitudes are useful signals, and the wavelet coefficients with smaller amplitudes are noise. According to the frequency distribution of the heart rate signal and the mixed noise, the wavelet coefficients at each scale are thresholded, and the wavelet coefficients smaller than the threshold are set to zero or processed by the threshold function.

[0115] (1.3) After processing the low-frequency coefficients and high-frequency coefficients after wavelet scaling decomposition, the signal is reconstructed.

[0116] (2) Feature extraction

[0117] In this embodiment, the physiological emotion recognition model uses LSTM (Long Short-Term Memory Artificial Neural Network) to extract features from heart rate data. The heart rate signal is a time-series signal, and each LSTM unit selectively forgets information from distant memories and remembers new information. The LSTM unit outputs the hidden layer state h at each time step. t (t = 1, 2, ..., n). How forgetting, remembering, and outputting occur is controlled by the forget gate, input gate, and output gate, which are determined by the hidden layer's state h at the last moment. t-1 and current input The forgetting gate selects the information to be forgotten based on the following formula, which determines the distant memory C from the previous moment. t How much is retained to the current time? This formula can be written as:

[0118]

[0119] Among them, f t W represents the output of the forget gate. f It is the weight matrix of the forget gate. It is the union of two vectors, b f σ represents the bias of the forget gate, and σ is the Sigmoid function.

[0120] Input gate selects the current input. To memorize. According to the formula below, it determines how much of the current input... Saved to cell state C t The degree of:

[0121]

[0122] Among them, i t W represents the output of the input gate. i It is the weight matrix of the input gate. It is the connection of two vectors, b i σ represents the bias of the forgotten gate, and σ is the Sigmoid function.

[0123] Temporary unit state As shown below, it represents the current cell state.

[0124]

[0125] Among them, W c It is the weight matrix of the tanh gate. It is the connection of two vectors, b c This indicates the bias of the tanh gate, where tanh means restricting the cell state to between -1 and 1.

[0126] Current cell state C t As shown below, it can combine current memories with previous memories to form new unit states.

[0127]

[0128] Where f represents the output of the forget gate, and C t-1 Indicates the cell state at the last moment, i t Represents the output of the input gate. It is a temporary unit state.

[0129] According to the following formula, the cell state controlled by the output gate can be used as the current output of the LSTM. This formula can be written as:

[0130]

[0131] Among them, o t W represents the output of the output gate. o It is the weight matrix of the forget gate. It is the union of two vectors, b o σ represents the bias of the forget gate, and σ is the Sigmoid function.

[0132] The cell state is processed by tanh, resulting in a value between -1 and 1. This value is then multiplied by the output of the output gate to obtain a new hidden layer state h. t .

[0133] h t =o t *tanh(C t (15)

[0134] In a specific example, training a physiological emotion recognition model includes the following steps:

[0135] (1) Construct the training sample set (80%) and the test sample set (20%).

[0136] (2) The physiological emotion recognition model is trained using the training sample set. The model is trained by minimizing the loss function through backpropagation and stochastic gradient descent.

[0137]

[0138] Among them, e h1 E represents the error of hidden layer node c. h2 This represents the error of hidden layer node d;

[0139]

[0140]

[0141] S4 fuses the facial expression emotion features with the heart rate features, and performs emotion recognition based on the fusion result to obtain the emotion category corresponding to each sample video.

[0142] In this embodiment, the emotion recognizer includes a fusion network, which is used to fuse the facial expression emotion features output by the facial emotion recognition model with the heart rate features output by the physiological emotion recognition model, and perform emotion recognition and classification based on the fused features to obtain the emotion category corresponding to each sample video.

[0143] The fusion network consists of an input layer, a competition layer, a hidden layer, and an output layer.

[0144] First, the fusion network automatically performs preliminary recognition of video images by mapping linearly inseparable images in a high-dimensional space. The images in the high-dimensional space are mapped to linearly separable vectors in the upper layer. Then, clustering features are transferred from the competition layer to the hidden layer. Finally, a non-linear mapping from input to output is completed in a supervised learning mode, classifying the emotions of the test students. Heart rate data features and facial expression features are input into neurons in the input layer. Neighboring features in the input space are mapped to neighboring neurons in the competition layer, which is a two-dimensional plane that preserves the topological structure of the input space. Therefore, the two-dimensional features are successfully classified, thus completing the preliminary classification of the input features.

[0145] The output layer outputs the predicted sentiment category. If the output layer does not conform to the expected emotion category Y = {y1, y2, y3, y4}, it will enter the backpropagation stage of the error, thus completing the non-linear mapping from input features to the subject's emotion:

[0146]

[0147] Among them, w ij This represents the weights between the input neuron i and the mapping neuron j.

[0148] W ij (t+1)=W ij (t)+η t h c,j (t)(X t -W ij (t)) (20)

[0149] Where η(t) represents the learning rate, 0 < η(t) < 1, h c,j (t) represents the domain function.

[0150]

[0151] Where dc,j represents the distance from the winning neuron c to any activated neuron j in the neighborhood, and r is the radius of the neighborhood. The update rule is as follows:

[0152]

[0153]

[0154] Here, INT() represents rounding a number to the nearest integer, and T represents the total number of iterations.

[0155] Y k =f(min||Xw) j ||) (24)

[0156] Where f() represents a nonlinear function, 0 <f()<1。

[0157]

[0158] Wherein, the error is the output of INBVNet. The difference between the expected output Y = {y1, y2, y3, y4} and the expected output.

[0159] S5 compares the emotion feature label corresponding to each test video with its emotion category, and generates the emotion detection result of the subject by combining the comparison results of each test video with the preset threshold conditions.

[0160] The emotional results of the students after watching the test video for each emotional stimulus were assigned a value. If the output emotional feature label matches the emotional category of the video, 1 point is awarded; otherwise, 0 points are awarded. The total score is 7 points.

[0161] Thresholds were set and corresponding measures were taken: A score of 0-2 indicates that the subject may have severe mental health disorder and is considered a high-risk student. The student's information and test results will be fed back to the school's psychological counseling department, and staff will provide one-on-one intervention and treatment. A score of 3-5 indicates that the subject has mental health disorder and needs to attend regular mental health lectures held on campus. A score of 6-7 indicates that the subject has normal mental health and does not require special psychological intervention.

[0162] It should be noted that although the operations of the methods of the embodiments of this specification are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0163] This embodiment also provides a student psychological state detection system based on a feature-interactive learning network. The system can be implemented in software and / or hardware and can be integrated into a computer device. Specifically, the system includes an infrared camera, a BVP sensor, an emotion recognizer, and a detection unit. The emotion recognizer includes a facial emotion recognition model, a physiological emotion recognition model, and a fusion network.

[0164] Among them, the infrared camera and BVP sensor are used to collect facial expression video data and heart rate data of the subjects when they watch a variety of sample videos with a single emotional feature.

[0165] The facial emotion recognition model is used to extract features from the facial expression video data of the test subjects to obtain the facial expression emotion features of the test subjects.

[0166] The physiological emotion recognition model is used to extract features from the heart rate data of the test subjects to obtain the heart rate characteristics of the test subjects;

[0167] The fusion network is used to fuse the facial expression emotion features with heart rate features, and to perform emotion recognition based on the fusion result, thereby obtaining the emotion category corresponding to each sample video.

[0168] The detection unit compares the emotion feature label corresponding to each sample video with its emotion category, and generates the emotion detection result of the subject by combining the comparison results of each sample video with the preset threshold conditions.

[0169] Specific limitations regarding the student mental health monitoring system can be found in the above section on the limitations of the student mental health monitoring methods, and will not be repeated here. Each module in the aforementioned student mental health monitoring system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0170] This embodiment also provides a computer device, which includes at least one processor and at least one memory. The memory stores a computer program. When the computer program is executed by the processor, it causes the processor to perform the steps of the student psychological state detection method, which will not be described in detail here. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be volatile memory, non-volatile memory, or a combination thereof.

[0171] The computer device can also communicate with one or more external devices (such as a keyboard, pointing terminal, monitor, etc.), one or more terminals that enable a user to interact with the computer device, and / or any terminal that enables the computer device to communicate with one or more other computing terminals (such as a network card, modem, etc.). This communication can be performed through an input / output (I / O) interface. Furthermore, the computer device can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) via a network adapter.

[0172] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0174] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0177] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0178] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0179] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting student psychological states based on feature-interactive learning networks, characterized in that, include: Acquire facial expression video data and heart rate data of subjects while watching multiple test videos with a single emotional characteristic; Feature extraction is performed on the facial expression video data to obtain the facial expression emotion features of the subject; including: The facial expression video data is divided into several sub-videos of the same length according to time sequence, and the segment features corresponding to each sub-video are extracted based on the attention mechanism; Perform linear projection on each segment feature to obtain the corresponding query vector, key vector, and value vector; Construct the first query matrix, the first key matrix, and the first value matrix based on the query vector, key vector, and value vector of all fragment features; Based on the first query matrix, the first key matrix, and the first value matrix, segment location encoding is applied to describe the location of each sub-video in the facial expression video data to obtain the encoded first emotion feature, which serves as the facial expression emotion feature of the subject. Feature extraction is performed on the heart rate data to obtain the heart rate characteristics of the subject; The facial expression emotion features are fused with heart rate features, and emotion recognition is performed based on the fusion result to obtain the emotion category corresponding to each sample video. The emotion feature label corresponding to each test video is compared with its emotion category. The emotion detection results of the test subjects are generated by combining the comparison results of each test video with the preset threshold conditions. The step of extracting segment features corresponding to each sub-video based on the attention mechanism includes: For each sub-video, extract the feature vector of each frame; Perform linear projection on the feature vector of each frame to obtain the corresponding query vector, key vector, and value vector; Construct a second query matrix, a second key matrix, and a second value matrix based on the query vector, key vector, and value vector of all frames; Calculate the general feature vector of the corresponding sub-video based on the second query matrix, the second key matrix, and the second value matrix; Estimate the distance between the general feature vector and the feature vector of each frame, and reweight the feature vector of each frame based on the distance to obtain the refined feature vector of each frame. The refined feature vectors of each frame are weighted and summed to obtain the segment features of the corresponding sub-video.

2. The student psychological state detection method as described in claim 1, characterized in that, The step of extracting features from heart rate data to obtain the heart rate features of the subject includes: The peak values ​​of heart rate data are extracted, and their mean, variance, root mean square value, standard deviation, and impulse factor are calculated based on the peak values ​​as the heart rate characteristics of the subjects.

3. The student psychological state detection method as described in claim 2, characterized in that, Before feature extraction from heart rate data, the following steps are also included: Wavelet transform was used to preprocess the heart rate data to filter out heart rate noise.

4. The student psychological state detection method as described in claim 1, characterized in that, The single emotional characteristics include neutrality, happiness, sadness, surprise, fear, anger, and disgust.

5. A student psychological state detection system based on a feature-interactive learning network, characterized in that, Includes an emotion recognizer and detection unit; The emotion recognizer includes a facial emotion recognition model, a physiological emotion recognition model, and a fusion network; The facial emotion recognition model is used to extract features from the facial expression video data of the test subjects when they watch a variety of sample videos with a single emotional feature, so as to obtain the facial expression emotion features of the test subjects; the facial emotion recognition model divides the facial expression video data into several sub-videos of the same length according to time sequence, and extracts the segment features corresponding to each sub-video based on the attention mechanism. Perform linear projection on each segment feature to obtain the corresponding query vector, key vector, and value vector; Construct the first query matrix, the first key matrix, and the first value matrix based on the query vector, key vector, and value vector of all fragment features; Based on the first query matrix, the first key matrix, and the first value matrix, segment location encoding is applied to describe the location of each sub-video in the facial expression video data to obtain the encoded first emotion feature, which serves as the facial expression emotion feature of the subject. The step of extracting segment features corresponding to each sub-video based on the attention mechanism includes: For each sub-video, extract the feature vector of each frame; Perform linear projection on the feature vector of each frame to obtain the corresponding query vector, key vector, and value vector; Construct a second query matrix, a second key matrix, and a second value matrix based on the query vector, key vector, and value vector of all frames; Calculate the general feature vector of the corresponding sub-video based on the second query matrix, the second key matrix, and the second value matrix; Estimate the distance between the general feature vector and the feature vector of each frame, and reweight the feature vector of each frame based on the distance to obtain the refined feature vector of each frame. The refined feature vectors of each frame are weighted and summed to obtain the segment features of the corresponding sub-video. The physiological emotion recognition model is used to extract features from the heart rate data of the subjects when they watch a variety of sample videos with a single emotion feature, so as to obtain the heart rate features of the subjects. The fusion network is used to fuse the facial expression emotion features with the heart rate features, and to perform emotion recognition based on the fusion result, thereby obtaining the emotion category corresponding to each sample video. The detection unit is used to compare the emotion feature label corresponding to each sample video with its emotion category, and generate the emotion detection result of the subject by combining the comparison results of each sample video and the preset threshold conditions.

6. The student psychological state detection system as described in claim 5, characterized in that, It also includes an infrared camera and a BVP sensor, which are used to collect video data of the subjects' facial expressions and heart rate, respectively.

7. The student psychological state detection system as described in claim 5, characterized in that, The facial emotion recognition model uses the Transformer model; the physiological emotion recognition model uses the Long Short-Term Memory artificial neural network model.

8. A computer device, characterized in that, It includes at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the computer program is executed by the processing unit, the processing unit performs the steps of the student psychological state detection method according to any one of claims 1 to 4.

Images