A multimodal real-time mental health analysis system based on artificial intelligence methods

Abstract

This invention provides a multimodal real-time mental health analysis system based on artificial intelligence methods, relating to the field of emotion analysis technology. It includes an EEG biofeedback device, a cloud server, and a computer. The EEG biofeedback device communicates with the computer via a USB serial port, while the computer communicates with the cloud server via TCP / IP. The deep learning model and data analysis code are deployed on the cloud server, the audio and image collection module and user interaction module are integrated into the computer, and the EEG collection module is deployed in the EEG biofeedback device. This invention provides a multidimensional, accurate, and real-time multimodal emotion analysis algorithm by fusing data from EEG, images, and audio. This algorithm not only processes and analyzes multimodal data in real time, improving the accuracy of emotion recognition, but also significantly enhances the robustness and accuracy of the algorithm, providing a more comprehensive and precise solution for emotion recognition and analysis.

Description

Technical Field

[0001] This invention relates to the field of emotion analysis technology, and in particular to a multimodal real-time mental health analysis system based on artificial intelligence methods. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and sensing technology, sentiment analysis technology has received widespread attention and application. However, traditional sentiment analysis techniques still have limitations in expressing more multi-dimensional and comprehensive emotional indicators, and cannot provide detailed and accurate emotional data. Their ability to analyze and express emotions is still imperfect.

[0003] To overcome these limitations, researchers have proposed several new sentiment analysis techniques, including:

[0004] Emotional audio analysis technology: This technology analyzes a user's audio signals to determine their emotional state, and it is a widely used method of emotional expression.

[0005] Emotional image analysis technology: Based on the user's facial expressions, it analyzes emotions and identifies emotions by capturing facial features;

[0006] Emotional EEG analysis technology: This technology obtains specific emotional indicators by analyzing users' brainwave data, providing a method for assessing internal emotional states.

[0007] Despite the significant progress these technologies have made in their respective fields, existing sentiment analysis algorithms still have shortcomings in the following aspects:

[0008] (1) Failed to reasonably link numerical indicators with emotional state, and lacked a deep understanding of the interaction between different indicators;

[0009] (2) There are shortcomings in the display and expression of emotional indicators, and it is not possible to intuitively present emotional changes;

[0010] (3) Existing technologies mainly focus on single-modal sentiment analysis and fail to effectively combine multimodal data for synchronous analysis, resulting in insufficient accuracy of the analysis results;

[0011] (4) Existing sentiment analysis technologies suffer from certain delays in data processing and feedback, which cannot meet the needs of real-time application scenarios, such as real-time customer service and online education. This lack of real-time capability may reduce the timeliness and practicality of sentiment analysis. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a multimodal real-time mental health analysis system based on artificial intelligence. By integrating data from three modalities—electroencephalogram (EEG), image, and audio—the system provides a comprehensive, accurate, and real-time multimodal emotion analysis algorithm. This algorithm can not only process and analyze multimodal data in real time and improve the accuracy of emotion recognition, but also significantly enhance the robustness and accuracy of the algorithm.

[0013] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0014] A multimodal real-time mental health analysis system based on artificial intelligence methods includes an EEG biofeedback device, a cloud server, and a computer; the EEG biofeedback device communicates with the computer via a USB serial port, and the computer communicates with the cloud server via TCP / IP;

[0015] In an EEG biofeedback device, an EEG acquisition module and a signal analysis module are deployed. The EEG acquisition module receives raw EEG signals, and the signal analysis module analyzes the raw EEG signals and transmits the signals to a computer.

[0016] A computer program for deploying deep learning models and data analysis is deployed on a cloud server. This program includes an interactive data storage module, an interactive parameter storage module, an audio processing module, an image processing module, an EEG processing module, a multimodal emotion analysis module, a result feedback module, and a result report display module. The functions of each module are as follows: The cloud server receives audio streams, data streams, and EEG signal streams from the audio-video EEG collection module on the computer; the interactive data storage module stores these streams in the cloud database according to timestamps. The cloud server receives parameters input by the user during the interaction process; the interactive parameter storage module stores these parameters in the cloud database according to timestamps. The audio processing module performs emotion analysis based on audio data. The image processing module performs emotion analysis based on image data. The EEG processing module performs emotion analysis based on EEG data. The multimodal emotion analysis module receives emotion indicators from the EEG-image-audio modalities in real time and performs multimodal emotion analysis. The result feedback module transmits the emotion analysis results back to the computer in real time for the user to view. The result report display module generates an emotion analysis report after recording is complete and transmits it back to the computer.

[0017] The computer integrates an audio / video EEG data collection module and a user interaction module. These modules are used to collect audio / video EEG data via computer components and to interact with the algorithm through a computer interface. Specifically, the computer collects audio data (Raw_A) via a microphone, image data (Raw_V) via a camera, and EEG data (Raw_B) via the EEG data collection module. Users can operate the algorithm using a mouse and the computer screen to start and stop recording, modify parameters, and view results.

[0018] Furthermore, the process of the EEG acquisition module acquiring EEG is as follows: receiving raw EEG data, denoted as Rawdata, and converting it into five types of EEG signals: Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave, a total of five unsigned values. Using a horizontal dynamic comparison method, the data is retained and transmitted to the signal analysis module when the packet loss rate is less than 10% and the checksum is passed.

[0019] The signal analysis module performs signal analysis and transmission as follows: it calculates the attention and relaxation levels using the five types of brainwave signals and maps them to a range of 0 to 100. Then, it transmits the seven types of brainwaves (Attention, Meditation, Delta, Theta, Alpha, Beta, and Gamma) to the computer via USB serial communication.

[0020] Furthermore, the specific process by which each module of the cloud server completes its function is as follows:

[0021] The process of storing interactive data in the interactive data storage module is as follows: the computer transmits the audio signals collected by the microphone, the image data collected by the camera, and the brainwave signals collected by the brainwave collection module to the cloud server in the form of real-time data stream via TCP / IP. The interactive data storage module stores the data into the cloud database in the order of timestamps.

[0022] The interaction parameter storage module stores interaction parameters as follows: the parameters entered by the user are stored in the local browser cache through the localStorage method. When the cloud server starts, it will retrieve the parameters entered by the user from the local browser cache. After the call is completed, the interaction data will be deleted to ensure the protection of user privacy.

[0023] The audio processing module handles audio data and performs sentiment analysis as follows: First, the audio processing module converts the Raw_A audio signal received from the cloud server into mono data and resamples it to 16000Hz. Then, it divides the signal into segments of fixed duration. The preprocessed audio data is then used for sentiment recognition through a trained audio prediction model. The specific process is as follows:

[0024] The input audio signal x(t) is first processed by a wav2vec pre-trained model as a feature encoder to obtain a feature vector sequence X = f_encoder(x(t)). The feature vector sequence X is then projected into a higher-dimensional space to match the input dimension of the Transformer model. After Transformer encoding and pooling operations, three emotion indicators are finally obtained: audio_arousal, audio_dominance, and audio_valence, with values ​​ranging from [0,1]. These emotion indicators are further mapped to the range of [-3,3]. The initial analysis period is set to 2, which can be modified according to actual user needs. Finally, the emotion indicator values ​​are stored in a cloud database according to timestamps.

[0025] The image processing module handles image data and performs sentiment analysis as follows: First, the image processing module feeds the image signal Raw_V received from the cloud server into a pre-trained DLIB face detection model to detect whether a face appears in the image and selects the largest face. Next, the selected face is fed into a pre-trained DLIB facial landmark detection model for landmark detection, generating an object containing the detected facial landmarks, denoted as landmarks_object. Subsequently, the landmarks_object is passed to a pre-trained sentiment estimator. The sentiment estimator uses partial least squares regression analysis to construct a two-dimensional sentiment space prediction model and employs Neutral to Apex regression. The morphing method calculates the emotional intensity, resulting in three key facial emotion indicators: face_arousal, face_intensity, and face_valence. The value range of face_intensity is [0,1], while the values ​​of the other two indicators range from [-1,1]. These three facial emotion indicators are further mapped to the range of [-3,3]. The output period is once per frame, and the three facial emotion indicators are stored in a cloud database according to the timestamp.

[0026] The EEG processing module handles EEG data and performs emotion analysis as follows: First, the module segments the Raw_B EEG signal received from the cloud server according to a fixed period. For the data within each period, the mean period of Delta, Theta, Alpha, Beta, and Gamma waves is calculated separately. Based on this, the standard deviation of these wavebands is calculated, and a probability density function of a normal distribution is constructed. The integral value of the data for the current second on the normal distribution is calculated. Then, the integral value is normalized and converted into a standard score. Based on these standard scores and preset weights, seven emotion indicators are calculated: light (happiness), dominance, alertness, clearness, nervousness, attention, and meditation. The calculation of each emotion indicator is based on the comprehensive weight of different waveband signals to ensure the accuracy and reliability of emotion analysis.

[0027] The multimodal sentiment analysis module analyzes multimodal emotions as follows: First, it retrieves the values ​​of six indicators—audio_arousal, audio_dominance, audio_valence, face_arousal, face_intensity, and face_valence—from the cloud database and maps them to the PAD sentiment table to obtain the predicted emotions for the current face and audio. The multimodal sentiment analysis module then visualizes audio_arousal, audio_dominance, and audio_valence as three-dimensional images and face_arousal and face_valence as two-dimensional images, storing them along with the analysis results in the cloud database.

[0028] The process of the result feedback module sending back the emotion results is as follows: After the cloud server receives the data and finishes the analysis, it retrieves the real-time analysis results from the database to form a dictionary, packages it in JSON format, and sends it back to the computer via TCP / IP. Users can view the currently predicted facial audio and EEG data, PAD emotion type, audio 3D stereoscopic image and facial 2D planar image.

[0029] The process of generating the emotion analysis report in the results report display module is as follows: After the user finishes recording, the cloud server assigns different weights to the PAD emotion index obtained from the audio facial analysis and the emotion index obtained from the EEG analysis, resulting in an analysis report of positive emotions, negative emotions, and physiological emotions.

[0030] Furthermore, the specific calculation method for the analysis report of positive emotions, negative emotions, and physiological emotions is as follows:

[0031] Positive emotion indicators include self-confidence, self-regulation ability, vitality, and mind-body balance;

[0032] The self-confidence mentioned is a probability index that the emotion is in the moderate positive range of the emotional space; it is calculated by weighting the probability that all three dimensions of emotion in the image and audio signals are greater than 0.

[0033] The self-regulation ability refers to the speed at which the intensity changes from high to low. It is calculated as follows: calculate the sum of the absolute changes between all adjacent elements, divide it by the number of changes between elements, and obtain the average rate of change. The self-regulation ability is evaluated based on the average rate of change. Intensity is used for images and dominance is used for audio.

[0034] The vitality is obtained by weighting the arousals of the two modalities from [-3,3] to [0,100], and the entire process is calculated using the average sentiment value.

[0035] The mind-body balance is assessed through a comprehensive evaluation of tension, anxiety, and stress; the higher the anxiety, the lower the mind-body balance.

[0036] Negative emotion indicators include aggression, tension, facial stress, and suspicion;

[0037] The aggression was calculated using two dimensions: hostility and uncontrollability, using only audio data; the final result was obtained by weighting the percentages of the two emotions as a distance metric.

[0038] The tension index is comprehensively evaluated by combining the results of EEG, images, and audio; a distance metric percentage is used, and the audio, video, and EEG data are weighted to obtain the final percentage; where tension in the audio is replaced by anxiety, and tension values ​​are used directly from the EEG data.

[0039] The facial expression stress is evaluated through image results, mapping intensity directly from [-3,3] to [0,100], and calculated using the average emotional value throughout the process;

[0040] The suspiciousness is calculated by comprehensively converting three dimensions: vigilance, tension, and alertness. First, the distance between image, audio, and EEG data is calculated as a percentage in a custom PAD emotional space. Then, the vigilance, tension, and alertness of the three modalities are weighted to obtain the final suspiciousness. Among them, the tension in the audio is obtained by anxiety conversion, and the alertness in the EEG is obtained by tension conversion.

[0041] Physiological and emotional indicators include attention, relaxation level, fatigue index, anxiety index, stress index, depression index, alertness, dominance, clarity, happiness, depression, and energy.

[0042] The levels of attention and relaxation were obtained by EEG biofeedback analysis using Rawdata.

[0043] The fatigue index is assessed using the arousal of images, audio, and brainwaves. The arousals of the three are directly mapped to [0, 100], and then weighted proportionally to obtain the final percentage of non-fatigue. The fatigue index percentage is then obtained by subtracting the percentage of non-fatigue from 100%.

[0044] The anxiety index is calculated using a distance metric; the distance between the current emotion and the anxiety is calculated based on the location of the anxiety in the PAD emotional space.

[0045] The stress index is calculated based on arousal, valence, and dominance, and is assessed according to the PAD emotional space.

[0046] The depression index is calculated based on arousal, valence, and dominance, and is assessed according to the PAD emotional space.

[0047] The alertness is obtained by adding the proportion of beta waves (weighted at 0.9) and the proportion of gamma waves (weighted at 0.1).

[0048] The control index is comprehensively evaluated by combining audio and EEG results; a distance metric percentage is used, and the audio and EEG data are weighted to obtain the final percentage;

[0049] The alertness index is comprehensively evaluated by combining audio and EEG results; a distance metric percentage is used, and the audio and EEG data are weighted to obtain the final percentage, with EEG data having the main weight.

[0050] The happiness is obtained by mapping arousal to [0, 100].

[0051] The depression is represented by a weighted average of dominance and pleasure; high dominance and low pleasure represent depression, calculated by subtracting the weighted percentages of happiness and dominance from 100%.

[0052] The energy is calculated using arousal and intensity; the energy is obtained by multiplying the arousal and intensity at each time step, where arousal comes from the weighted sum of the three modalities and intensity comes from the image modality.

[0053] Furthermore, the mapping process used in the calculation of positive emotions, negative emotions, and physiological emotions is represented by the following formula:

[0054] X i =m·Li +c

[0055] Among them, L i X represents the original probability value. i Let m be the value after mapping, m be the multiplier of the mapping, and c be the constant term of the mapping.

[0056] The weighting process is represented by the following formula:

[0057]

[0058] Among them, X i For the i-th average value, w i Here, represents the weight corresponding to the i-th average value, and n represents the total number of values ​​involved in the calculation.

[0059] The mapping process of the PAD emotional space is as follows:

[0060] Calculate the maximum distance D from each extreme point in the PAD emotional space to the emotional coordinate T. max :

[0061] D max =max{D1,D2,…,D n}

[0062] Calculate the distance D between the current emotion and the emotion coordinate T. 当前情绪 :

[0063]

[0064] Where, T = [T x T y T z [x, y, z] represents the coordinates of a certain emotional position in the emotional space, and E = [x, y, z] represents the PAD emotional space coordinates of the current emotion;

[0065] Calculate the percentage of emotion using the following formula:

[0066]

[0067] The energy index is calculated as follows:

[0068] For each emotion pair (a i i i The energy is calculated as the product of its absolute values, as shown in the following formula:

[0069] energy i =|a i |×|i i |

[0070] If the energy list is empty, return 0;

[0071] The maximum energy value `max_energy` in the energy list is calculated to normalize the energy values ​​to the range [0, 100], as follows:

[0072]

[0073] Summing the normalized energy lists and dividing by the length of the energy lists yields the average energy value, as shown in the following formula:

[0074]

[0075] Here, len(normalized_energy) represents the length of the energy list.

[0076] Furthermore, the specific process of training the CNN14 model in the audio processing and audio prediction model within the audio processing module is as follows:

[0077] Step 1: Generate a log-Mel spectrum;

[0078] Step 1.1: Frame division;

[0079] The audio signal x(t) is divided into several time frames. Each frame is weighted using a window function w(k). The purpose of the window function is to reduce spectral leakage and smooth edges, as shown in the following equation:

[0080] x f [n]=x[n·H+k]·w[k],k=0,1,…,L-1

[0081] Where H is the step size, 10 milliseconds; L is the window size, 32 milliseconds; k represents the sampling point index within the window function w[k], from 0 to L-1; x f [n] represents the information of the nth frame after processing by the window function, and f is the frame index used to distinguish different frames;

[0082] Step 1.2: Perform a short-time Fourier transform;

[0083] Perform a Fourier transform on each frame according to the following formula:

[0084]

[0085] Where Π represents pi, approximately equal to 3.14159, X f [m,k] is the k-th frequency component of the m-th frame;

[0086] Step 1.3: Convert to Mel frequency scale;

[0087] Convert the spectrum to Mel spectrum using a Mel filter:

[0088]

[0089] Where f represents the index of the Mel frequency band, and H[f,k] is the filter coefficient of the Mel filter bank, representing the response of the f-th Mel frequency band to the k-th frequency component;

[0090] Step 1.4: Take the logarithm;

[0091] Taking the logarithm of the Mel spectrum yields the log-Mel spectrum:

[0092] Log-Mel(m,f) = log(M[m,f])

[0093] Step 2: Feed the log-Melogram obtained in Step 1 as a feature map into the CNN14 model. The CNN14 model consists of 6 convolutional blocks, each with two layers. Each layer has a max pooling operation, where the convolutional kernel size is 3*3 with a stride of 1, and the max pooling layer has a stride of 2. Details are as follows:

[0094] Step 2.1: Convolutional layer;

[0095] The output of each convolutional layer is represented as follows:

[0096] C l =σ(W l *X l +b l )

[0097] Among them, W l It is the convolution kernel, * indicates the convolution operation, b l σ is the bias, X is the activation function, and X is the activation function. l It is the input feature log-Mel spectrogram;

[0098] Step 2.2: Max pooling layer;

[0099] The output of each pooling layer is represented as follows:

[0100] P l =MaxPool(C l )

[0101] Step 2.3: Dropout layer;

[0102] Apply Dropout after each convolutional block:

[0103] D l =Dropout(P l ,p)

[0104] Where p is the probability of Dropout;

[0105] Step 2.4: Average pooling and max pooling;

[0106] The output of the last convolutional layer is subjected to both average pooling and max pooling:

[0107]

[0108]

[0109] Where N represents the total number of elements in the feature map, C i This represents the value of the i-th element in the feature map;

[0110] Step 2.5: Fully connected layer;

[0111] The pooled features are then input into the fully connected layer:

[0112] F1=σ(W1·[MeanPool;MaxPool]+b1)

[0113] F2 = σ(W2·F1 + b2)

[0114] Where W1 and W2 represent the weight matrices of the fully connected layer, b1 and b2 represent the corresponding bias terms, and σ represents the activation function;

[0115] After convolutional layer transformation, the output time series feature representation is given, with a dimension of T×512, where T is the time step of the feature vector. The output feature is then transformed into a high-dimensional feature representation with a dimension of 1024, as shown in the following formula:

[0116] H = f project (F2)=FW p +b p

[0117] Among them, f project Denotes the projection function, W p and b p These represent the weight matrix and bias term of the projection layer, respectively;

[0118] Step 3: Transformer encoding layer;

[0119] The Transformer encoder consists of 24 Transformer layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network, with a dimension of 1024.

[0120] Step 3.1: Multi-head self-attention mechanism; The input to the first layer is H, and for the input H of the l-th layer... (L-1) , as follows:

[0121] Step 3.1.1: Linear Transformation:

[0122] Q = H (l-1) W Q K = H (l-1) W K V=H (l-1) W V

[0123] Among them, W Q 、W K 、W v All dimensions are 1024×1024; W Q 、W K and W v These are the weight matrices for the query, key, and value, respectively. Q, K, and V are the query, key, and value matrices obtained through linear transformations.

[0124] Step 3.1.2: Calculate the self-attention score:

[0125]

[0126] in, h is the number of heads;

[0127] Step 3.1.3: Multi-headed attention:

[0128] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0129] The output of each attention head is head. i =Attention(Q) i ,K i V i );W O It is a weight matrix used for linear transformation after splicing, with a dimension of 1024×1024;

[0130] Step 3.2: Feedforward Neural Network;

[0131] Step 3.2.1: First-level linear transformation and activation function:

[0132] FFN1(x)=ReLU(xW1+b1)

[0133] Where W1 has a dimension of 1024×4096, and b1 is the bias;

[0134] Step 3.2.2: Second-level linear transformation:

[0135] FFN2(x)=ReLU(xW1+b1)W2+b2

[0136] Where W2 has a dimension of 4096×1024, and b2 is the bias;

[0137] Step 3.3: Perform layer normalization and residual connections after each sub-layer:

[0138] Step 3.3.1: Self-attention sublayer:

[0139] H (l) =LayerNorm(H (l-1) +MultiHead(Q,K,V))

[0140] Step 3.3.2: Feedforward Neural Network Sublayer:

[0141] O = LayerNorm(H (l) +FFN(H (l) ))

[0142] After 24 Transformer layers, the final output O is obtained;

[0143] Step 4: Pooling operation:

[0144] Pooling is applied to the output O of the last layer of the Transformer encoder to obtain a fixed-size output vector:

[0145] p = f pool (O)

[0146] Step 5: Output:

[0147] The pooled feature vector p is passed through one or more fully connected layers, and then through an output layer to predict the sentiment dimension score:

[0148] y = f output (P)

[0149] Where y is the predicted three-dimensional vector, including scores for arousal, dominance, and sentiment value:

[0150]

[0151] y1=f arousal (P)

[0152] y2=f dominance (P)

[0153] y3=f valence (P)

[0154] Among them, arousal, dominance, and valence represent arousal, dominance, and affectivity, respectively, and are three commonly used dimensions to describe emotional states.

[0155] Furthermore, the image processing performed by the image processing module, along with the sentiment estimator based on geometric features, finite element analysis, and partial least squares regression, is as follows:

[0156] Step 1: Obtain image key points;

[0157] Step 1.1: Image preprocessing;

[0158] After the camera captures an image, the "cvtColor" function in the OpenCV library is used to convert it into a grayscale image;

[0159] Step 1.2: Key point detection;

[0160] Face detection is performed using the DLIB frontal face detector get_frontal_face_detector() to extract the outer rectangle of the face; the DLIB library-trained 68-point face feature detector is used to extract the facial contour features and obtain the coordinates of 49 filtered facial coordinate points.

[0161] Step 2: Facial positivity;

[0162] Step 2.1: Scaling, Translation, and Rotation:

[0163] First, scale, translate, and rotate the coordinate points so that all coordinate points are on the same frontal face:

[0164]

[0165] in: It is the position point or feature vector of the k-th sample in the original coordinate system, where o represents the original coordinate system; Indicates all The mean vector is the mean of all sample points in the original coordinate system, used for translation processing; R k This is the rotation matrix used to adjust the orientation of coordinate points; k: represents the index of the sample; o: the identifier in the original coordinate system, used to indicate the point in the original system; M: represents the coordinate system after scaling, translation, and rotation.

[0166] Step 2.2: Construct a transpose matrix A containing the coordinates of all non-frontal expressions and a ground truth matrix Y containing the coordinates of the corresponding frontal expressions, and fill the normalized coordinates into the matrices;

[0167] Step 2.3: Objective function:

[0168] argmin X |Y-AX|2+λ|X|2

[0169] Where λ is the regularization parameter; X represents the parameter matrix to be optimized;

[0170] Step 2.4: Predicted values ​​for positive mapping X:

[0171]

[0172] Where I represents the identity matrix, and its dimensions are... Same; T indicates transpose;

[0173] Step 3: Partial Least Squares Regression;

[0174] Step 3.1: Input and Output:

[0175] X1: Initial feature, with a size of N×1176, is a 1176-dimensional feature calculated from the pairwise distances of 49 facial landmarks;

[0176] Y1: Response feature, with a size of N×3, including three dimensions of output;

[0177] Step 3.2: Model Representation:

[0178]

[0179]

[0180] Where T and U are latent variable matrices: of size N×29, with 29 latent variables; P and Q are loading matrices; E and F are residual matrices;

[0181] Step 3.3: Objective function;

[0182] The goal is to maximize the covariance between the response features and the predicted features:

[0183]

[0184] Where t and u are column vectors of T and U, respectively; X1 represents the initial feature matrix, used for predicting feature data; Y1 represents the response feature matrix, containing the values ​​of the target variable; w and v are two direction vectors, used to project data into the feature space and response space, respectively; X1w represents the product of matrix X1 and vector w, the result of which is the projection along the w direction in the feature space; Y1v represents the product of matrix Y1 and vector v, the result of which is the projection along the v direction in the response space.

[0185] Step 3.4: Solve for the regression coefficient matrix B using the following formula, which is used to map the feature matrix X1 to the predicted response:

[0186]

[0187] Step 3.5: Predict the response matrix:

[0188] The final predicted response matrix is ​​calculated using the following formula:

[0189]

[0190] in, The predicted response feature matrix includes scores for arousal, sentiment, and intensity.

[0191]

[0192] Furthermore, the specific steps for processing the EEG data and scoring the emotion index are as follows:

[0193] Step 1: Calculate the mean:

[0194] The input is the value x of the i-th data point. i The total number of data points, n;

[0195] The output is the mean μ of the data points;

[0196] Step 2: Calculate the standard deviation:

[0197]

[0198] The input is the mean μ of the data points, and the output is the standard deviation σ of the data points.

[0199] Step 3: Construct the probability density function of the normal distribution:

[0200]

[0201] The inputs are the current data point value x, the mean μ of the data point, and the standard deviation σ of the data point;

[0202] The output is the probability density value f(x) of the data point (x) under the normal distribution;

[0203] Step 4: Calculate the integral of the data for the current second on this distribution:

[0204]

[0205] The inputs are the value of the data point in the current second, x_current, and the probability density value f(x) of the data point (x) under the normal distribution.

[0206] The output is the cumulative distribution function value P(x) of the current data point on a normal distribution. current );

[0207] Step 5: Normalize and convert to fractions:

[0208] Score = 100 × P(x) current )

[0209] The input is the cumulative distribution function value P(x) of the current data point on a normal distribution. current );

[0210] The output is the normalized score for the current data point;

[0211] Step 6: The overall process formula is:

[0212]

[0213] The beneficial effects of adopting the above technical solution are as follows: The multimodal real-time mental health analysis system based on artificial intelligence provided by this invention proposes a multimodal emotion analysis algorithm based on EEG, image, and audio signals. It can comprehensively assess the user's emotional state by combining multiple biological signals and behavioral data, improving the accuracy of emotion recognition and the robustness of the algorithm; it overcomes the shortcomings of existing technologies in single-modal emotion analysis, achieving simultaneous analysis of multimodal data and improving the accuracy of emotion analysis results; through deep learning models and data analysis techniques, it processes audio, image, and EEG signals to generate emotion indicators, and provides a more intuitive display of emotion changes through multimodal fusion of emotional expression methods; it also designs... The cloud-edge collaborative framework deploys deep learning models and data analysis on a cloud server, while the audio / video EEG collection module and user interaction module are integrated on the local computer, achieving efficient data transmission and processing, and giving the algorithm higher scalability and robustness. An efficient data transmission and processing mechanism is established between the EEG biofeedback device, the local computer, and the cloud server, enabling real-time data transmission and analysis through USB serial communication and TCP / IP communication, ensuring the timeliness and accuracy of emotion analysis. The multimodal emotion analysis algorithm of this invention provides comprehensive emotion indicators, including positive emotions, negative emotions, and physiological emotions, enabling detailed analysis and evaluation of the user's emotional state, and improving the user's cognition and understanding of emotional states. Attached Figure Description

[0214] Figure 1 This is an architecture diagram of a multimodal real-time mental health analysis system based on artificial intelligence methods provided in an embodiment of the present invention.

[0215] Figure 2 A flowchart of a multimodal real-time mental health analysis algorithm provided in an embodiment of the present invention;

[0216] Figure 3 This is a schematic diagram of the main interface of the multimodal algorithm provided in an embodiment of the present invention;

[0217] Figure 4This is a schematic diagram of the EEG modal interface provided in an embodiment of the present invention;

[0218] Figure 5 This is a diagram of the brainwave emotion analysis results provided in an embodiment of the present invention;

[0219] Figure 6 This is a schematic diagram of an image-audio modal interface provided in an embodiment of the present invention;

[0220] Figure 7 The image analysis result diagram displayed by the image display module provided in this embodiment of the invention;

[0221] Figure 8 An audio analysis result diagram displayed by the audio analysis module provided in this embodiment of the invention;

[0222] Figure 9 The multimodal results display interface and the displayed positive sentiment radar chart are provided for embodiments of the present invention;

[0223] Figure 10 The negative sentiment radar chart provided in this embodiment of the invention;

[0224] Figure 11 Physiological emotion radar map provided for embodiments of the present invention;

[0225] Figure 12 This is a schematic diagram of a user setting parameter pop-up window provided in an embodiment of the present invention.

[0226] In the diagram: 1. Image display module; 2. Line chart module; 3. Image analysis module; 4. Audio analysis module; 5. User interaction button. Detailed Implementation

[0227] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0228] A multimodal real-time mental health analysis system based on artificial intelligence methods, such as Figure 1 As shown, the system includes an EEG biofeedback device, a cloud server, and a computer. The EEG biofeedback device communicates with the computer via USB serial port, while the computer communicates with the cloud server via TCP / IP. The complete workflow for multimodal real-time mental health analysis using this system is as follows: Figure 2 As shown, the system welcome and function selection interface is as follows: Figure 3 As shown.

[0229] The EEG biofeedback device deploys an EEG acquisition module and a signal analysis module. The EEG acquisition module receives raw EEG signals, and the signal analysis module analyzes the raw EEG signals and transmits them to a computer.

[0230] The process of acquiring brainwaves by the brainwave acquisition module is as follows: it receives raw brainwave data, which is denoted as Rawdata, and converts it into five types of brainwave signals: Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave, a total of five unsigned values. It uses a horizontal dynamic comparison method to retain the data and transmit it to the signal analysis module if the packet loss rate is less than 10% and the checksum is passed.

[0231] The signal analysis module performs signal analysis and transmission as follows: Attention and Meditation are calculated from the five types of EEG signals mentioned above and mapped to a range of 0-100. Then, seven types of EEG signals (Attention, Meditation, Delta, Theta, Alpha, Beta, and Gamma) are transmitted to the computer via USB serial communication. Real-time data will be displayed in two visualization formats, such as... Figure 4 As shown. The overall score will be displayed after the recording is complete, as shown below. Figure 5 As shown.

[0232] A computer program for deploying deep learning models and data analysis is deployed on a cloud server, including an interactive data storage module, an interactive parameter storage module, an audio processing module, an image processing module, an EEG processing module, a multimodal emotion analysis module, a results feedback module, and a results report display module. The functions of each module are as follows:

[0233] The cloud server receives audio and video EEG data from the computer's EEG collection module (such as...). Figure 6 (As shown) The returned audio stream, data stream, and EEG signal stream will be stored in the cloud database in timestamp order by the interactive data storage module. Figure 6 The display modules include image display module 1, line chart module 2, image analysis module 3, audio analysis module 4, and user interaction buttons 5. The cloud server accepts parameters input by the user during the interaction process, such as... Figure 12 As shown in the prompt: "Save this data to local storage?", the interaction parameter storage module will store the data in the cloud database according to the timestamp order; the audio processing module will perform emotion analysis based on the audio data; the image processing module will perform emotion analysis based on the image data; the EEG processing module will perform emotion analysis based on the EEG data; the multimodal emotion analysis module will receive emotion indicators from the EEG-image-audio modalities in real time and perform multimodal emotion analysis; the result feedback module will send the emotion analysis results back to the computer in real time for the user to view on the computer, such as... Figure 7 , Figure 8 As shown; the results report display module is used to generate a sentiment analysis report after the recording is completed and send it back to the computer, such as... Figure 9 , Figure 10and Figure 11 As shown.

[0234] The interactive data storage module stores interactive data as follows: The computer transmits the audio signals collected by the microphone (denoted as Raw_A), the image data collected by the camera (denoted as Raw_V), and the brainwave signals collected by the brainwave collection module (denoted as Raw_B) to the cloud server in real-time data streams via TCP / IP. The audio and video brainwave stream data includes timestamp information to ensure data integrity and timing accuracy. Simultaneously, the interactive data storage module stores the data in the database according to the timestamp order.

[0235] The interaction parameter storage module stores interaction parameters as follows: the parameters entered by the user are stored in the local browser cache through the localStorage method. When the cloud server starts, it retrieves the parameters entered by the user from the local browser cache. After the call is completed, the interaction data will be deleted to ensure the protection of user privacy.

[0236] The audio processing module handles audio data and performs emotion analysis as follows: The audio signal (denoted as Raw_A) collected by the microphone is transmitted to the audio processing module for detailed analysis. First, the audio processing module converts the audio signal Raw_A received from the cloud server into mono data, resamples it to 16000Hz, and then divides it into segments of fixed duration. The preprocessed audio data is then used for emotion recognition through a trained audio prediction model. The specific process is as follows:

[0237] In audio emotion recognition primarily based on the Transformer architecture, the input audio signal x(t) is first processed by a wav2vec pre-trained model as a feature encoder to obtain a feature vector sequence X = f_encoder(x(t)). The feature vector sequence X is then projected into a higher-dimensional space to match the input dimension of the Transformer model. After Transformer encoding and pooling operations, three emotion indicators are finally obtained: audio arousal, audio dominance, and audio valence. These emotion indicators range from [0,1]. To facilitate further analysis, these emotion indicators are further mapped to the range [-3,3]. The default analysis cycle is once every 2 seconds, but this can be modified according to user needs. Finally, the emotion indicator values ​​are stored in a cloud database according to their timestamps. The specific training process of the CNN14 model in the audio processing and audio prediction model is as follows:

[0238] Step 1: Generate a log-Mel spectrum.

[0239] Step 1.1: Frame division.

[0240] The audio signal x(t) is divided into several time frames. Each frame is weighted using a window function w(k). The purpose of the window function is to reduce spectral leakage and smooth edges, as shown in the following equation:

[0241] x f [n]=x[n·H+k]·w[k],k=0,1,…,L-1

[0242] Where H is the step size, 10 milliseconds; L is the window size, 32 milliseconds; k represents the sampling point index within the window function w[k], from 0 to L-1; x f [n] represents the information of the nth frame after processing by the window function, and f is the frame index used to distinguish different frames.

[0243] Step 1.2: Perform a short-time Fourier transform.

[0244] Perform a Fourier transform on each frame according to the following formula:

[0245]

[0246] Where Π represents pi, approximately equal to 3.14159, X f [m,k] is the k-th frequency component of the m-th frame.

[0247] Step 1.3: Convert to Mel frequency scale.

[0248] Convert the spectrum to Mel spectrum using a Mel filter:

[0249]

[0250] Where f represents the index of the Mel frequency band, and there are usually 64 Mel frequency bands. H[f,k] are the filter coefficients of the Mel filter bank, representing the response of the f-th Mel frequency band to the k-th frequency component.

[0251] Step 1.4: Take the logarithm.

[0252] Taking the logarithm of the Mel spectrum yields the log-Mel spectrum:

[0253] Log-Mel(m,f) = log(M[m,f])

[0254] Step 2: The log-Melogram obtained in Step 1 is fed into the CNN14 model as a feature map. The CNN14 model consists of 6 convolutional blocks, each with two layers. Each layer has a max pooling operation, where the convolutional kernel size is 3*3 with a stride of 1, and the max pooling layer has a stride of 2. Details are as follows:

[0255] Step 2.1: Convolutional layer.

[0256] The output of each convolutional layer is represented as follows:

[0257] C l =σ(W l *X l +b l )

[0258] Among them, W l It is the convolution kernel, * indicates the convolution operation, b l σ is the bias, X is the activation function, and X is the activation function. l It is the input feature logarithm-Mel spectrogram.

[0259] Step 2.2: Max pooling layer.

[0260] The output of each pooling layer is represented as follows:

[0261] P l =MaxPool(C l )

[0262] Step 2.3: Dropout layer.

[0263] Apply Dropout after each convolutional block:

[0264] D l =Dropout(P l ,p)

[0265] Where p is the probability of Dropout.

[0266] Step 2.4: Average pooling and max pooling;

[0267] The output of the last convolutional layer is subjected to both average pooling and max pooling:

[0268]

[0269]

[0270] Where N represents the total number of elements in the feature map, C i This represents the value of the i-th element in the feature map.

[0271] Step 2.5: Fully connected layer;

[0272] The pooled features are then input into the fully connected layer:

[0273] F1=σ(W1·[MeanPool;MaxPool]+b1)

[0274] F2 = σ(W2·F1 + b2)

[0275] Where W1 and W2 represent the weight matrices of the fully connected layer, b1 and b2 represent the corresponding bias terms, and σ represents the activation function;

[0276] After convolutional layer transformation, the output time series feature representation is given, with a dimension of T×512, where T is the time step of the feature vector. The output feature is then transformed into a high-dimensional feature representation with a dimension of 1024, as shown in the following formula:

[0277] H = f project (F2)=FW p +b p

[0278] Among them, f project Denotes the projection function, W p and b p These represent the weight matrix and bias term of the projection layer, respectively;

[0279] Step 3: Transformer encoding layer;

[0280] The Transformer encoder consists of 24 Transformer layers, each including a multi-head self-attention mechanism and a feedforward neural network, with a dimension of 1024.

[0281] Step 3.1: Multi-Head Self-Attention Mechanism. The input to the first layer is H, and for the input H of the l-th layer... (L-1) , as follows:

[0282] Step 3.1.1: Linear Transformation:

[0283] Q = H (l-1) W Q K = H (l-1) W K V=H (l-1) W V

[0284] Among them, W Q W K W v All dimensions are 1024×1024; W Q W K and W v These are the weight matrices for the query, key, and value, respectively. Q, K, and V are the query, key, and value matrices obtained through linear transformation.

[0285] Step 3.1.2: Calculate the self-attention score:

[0286]

[0287] in, h is the number of heads.

[0288] Step 3.1.3: Multi-headed attention:

[0289] MultiHead(Q,K,V)=Concat(head1,…,head h W O

[0290] The output of each attention head is head. i =Attention(Q) i ,K i V i );W O It is a weight matrix used for linear transformation after splicing, with a dimension of 1024×1024.

[0291] Step 3.2: Feedforward Neural Network.

[0292] Step 3.2.1: First-level linear transformation and activation function:

[0293] FFN1(x)=ReLU(xW1+b1)

[0294] Where W1 has a dimension of 1024×4096, and b1 is the bias.

[0295] Step 3.2.2: Second-level linear transformation:

[0296] FFN2(x)=ReLU(xW1+b1)W2+b2

[0297] Where W2 has a dimension of 4096×1024, and b2 is the bias.

[0298] Step 3.3: Perform layer normalization and residual connection after each sub-layer.

[0299] Step 3.3.1: Self-attention sublayer:

[0300] H (l) =LayerNorm(H (l-1) +MultiHead(Q,K,V))

[0301] Step 3.3.2: Feedforward Neural Network Sublayer:

[0302] O = LayerNorm(H (l) +FFN(H(l) ))

[0303] After 24 Transformer layers, the final output O is obtained.

[0304] Step 4: Pooling operation:

[0305] Pooling is applied to the output O of the last layer of the Transformer encoder to obtain a fixed-size output vector:

[0306] p = f pool (O)

[0307] Step 5: Output:

[0308] The pooled feature vector p is passed through one or more fully connected layers (fine-tuning process), and then through an output layer to predict the sentiment dimension score:

[0309] y = f output (P)

[0310] Where y is the predicted three-dimensional vector, including scores for arousal, dominance, and sentiment value:

[0311]

[0312] y1=f arousal (P)

[0313] y2=f dominance (P)

[0314] y3=f valence (P)

[0315] Among them, arousal, dominance, and valence represent arousal, dominance, and affectivity, respectively, and are three commonly used dimensions to describe emotional states.

[0316] Arousal is defined as the degree or intensity of an emotion, representing the excitement of the emotion. It measures the level of excitement or alertness of an individual in an emotional state; the range is usually from low to high, such as from calm to excited.

[0317] Dominance is defined as: describing an individual's sense of control or dominance in an emotional state. It reflects the degree of control or dominance an individual has in an emotional state; the range is usually from low to high, such as from helplessness to strong sense of control.

[0318] The definition of emotional valence is: describing the positive or negative nature of an emotion, indicating the degree of pleasure or unpleasantness of the emotion; it measures the positive or negative nature of the emotion; the range is usually from negative to positive, such as from sadness to happiness.

[0319] The image processing module handles image data and performs sentiment analysis as follows: The video signal (denoted as Raw_V) captured by the camera is transmitted to the image processing module for detailed analysis. First, the image processing module feeds the image signal Raw_V received from the cloud server into a pre-trained DLIB face detection model to detect whether a face appears in the image and selects the largest face. Next, the selected face is fed into a pre-trained DLIB facial landmark detection model for landmark detection, generating an object containing the detected facial landmarks, denoted as landmarks_object. Subsequently, the landmarks_object is passed to a pre-trained sentiment estimator. The sentiment estimator uses partial least squares (PLS) regression analysis to construct a two-dimensional sentiment space prediction model and employs the Neutral to Apex morphing method to calculate sentiment intensity. Through this process, three key facial sentiment indicators can be derived: face_arousal, face_intensity, and face_valence. The facial intensity (face_intensity) ranges from [0,1], while the other two indicators range from [-1,1]. For further analysis, these three facial emotion indicators are further mapped to the range [-3,3]. By default, the output period is once per frame, and the three facial emotion indicators are stored in a cloud database according to their timestamps. The image processing and emotion estimator based on geometric feature-based finite element analysis and partial least squares regression are detailed below:

[0320] Step 1: Obtain image key points.

[0321] Step 1.1: Image preprocessing;

[0322] After the camera captures an image, the "cvtColor" function in the OpenCV library is used to convert it into a grayscale image.

[0323] Step 1.2: Key point detection;

[0324] Face detection is performed using the DLIB frontal face detector (get_frontal_face_detector()), and the outer bounding box of the face is extracted. Then, a 68-point face feature detector trained using the DLIB library is used to extract facial contour features, obtaining the coordinates of 49 selected facial points.

[0325] Step 2: Facial positivity.

[0326] Step 2.1: Scaling, Translation, and Rotation:

[0327] First, scale, translate, and rotate the coordinate points so that all coordinate points are on the same frontal face:

[0328]

[0329] in: It is the position point or feature vector of the k-th sample in the original coordinate system, where o represents the original coordinate system; Indicates all The mean vector of R is the mean of all sample points in the original coordinate system, used for translation processing; k is the rotation matrix used to adjust the orientation of coordinate points; k: represents the index of the sample; o: the identifier in the original coordinate system, used to indicate the point in the original system; M: represents the coordinate system after scaling, translation and rotation.

[0330] Step 2.2: Construct a transpose matrix A containing the coordinates of all non-frontal expressions and a ground truth matrix Y containing the coordinates of the corresponding frontal expressions, and fill the normalized coordinates into the matrices.

[0331] Step 2.3: Objective function:

[0332] argmin X |Y-AX|2+λ|X|2

[0333] Where λ is the regularization parameter; X represents the parameter matrix to be optimized.

[0334] Step 2.4: Predicted values ​​for positive mapping X:

[0335]

[0336] Where I represents the identity matrix, and its dimensions are... Same; T indicates transpose.

[0337] Step 3: Partial Least Squares Regression.

[0338] Step 3.1: Input and Output:

[0339] Input X1: Initial features, size N×1176, 1176-dimensional features calculated from the pairwise distances of 49 facial landmarks;

[0340] Output Y1: Response feature, size N×3, including three dimensions of output.

[0341] Step 3.2: Model Representation:

[0342]

[0343]

[0344] Where T and U are latent variable matrices: size N×29, with 29 latent variables; P and Q are loading matrices; E and F are residual matrices.

[0345] Step 3.3: Objective function;

[0346] The goal is to maximize the covariance between the response features and the predicted features:

[0347]

[0348] Where t and u are column vectors of T and U, respectively; X1 represents the initial feature matrix, used for predicting feature data; Y1 represents the response feature matrix, containing the values ​​of the target variable; w and v are two direction vectors, used to project data into the feature space and response space, respectively; X1w represents the product of matrix X1 and vector w, the result of which is the projection along the w direction in the feature space; Y1v represents the product of matrix Y1 and vector v, the result of which is the projection along the v direction in the response space.

[0349] Step 3.4: Solve for the regression coefficient matrix B using the following formula, which is used to map the feature matrix X1 to the predicted response Y1:

[0350]

[0351] Step 3.5: Predict the response feature matrix:

[0352] Final predicted response feature matrix Calculated using the following formula:

[0353]

[0354] in, The predicted response feature matrix includes scores for arousal, sentiment, and intensity:

[0355]

[0356] The process of processing EEG data and analyzing emotions by the EEG processing module is as follows: The brainwave signals (denoted as Raw_B) collected by the EEG biofeedback device are transmitted to the EEG processing module for detailed analysis. First, the EEG processing module segments the Raw_B EEG signals received from the cloud server according to a fixed period. For the data within each period, the period mean of Delta, Theta, Alpha, Beta, and Gamma waves is calculated separately. Based on this, the standard deviation of these wavebands is calculated, and a probability density function of a normal distribution is constructed. The integral value of the data for the current second on the normal distribution is calculated. Then, the integral value is normalized and converted into a standard score. Based on these standard scores and combined with preset weights, seven emotion indicators are calculated: light (happiness), dominance, alertness, clearness, nervousness, attention, and meditation. The calculation of each of these seven emotion indicators is based on the comprehensive weight of different waveband signals to ensure the accuracy and reliability of emotion analysis. The specific steps are as follows:

[0357] Step 1: Calculate the mean:

[0358] The input is the value x of the i-th data point. i The total number of data points is n. The output is the mean μ of the data points.

[0359] Step 2: Calculate the standard deviation:

[0360]

[0361] The input is the mean μ of the data points, and the output is the standard deviation σ of the data points.

[0362] Step 3: Construct the probability density function (PDF) of the normal distribution:

[0363]

[0364] The input consists of the current data point value x, the mean μ of the data point, and the standard deviation σ of the data point. The output is the probability density value f(x) of the data point (x) under a normal distribution.

[0365] Step 4: Calculate the integral of the data for the current second on this distribution:

[0366]

[0367] The input is the value of the data point in the current second, x_current, and the probability density function f(x) of the data point (x) under a normal distribution. The output is the cumulative distribution function P(x) of the current data point under a normal distribution.current ).

[0368] Step 5: Normalize and convert to fractions:

[0369] Score = 100 × P(x) current )

[0370] The input is the cumulative distribution function value P(x) of the current data point on a normal distribution. current The output is the normalized score for the current data point.

[0371] Step 6: The overall process formula is:

[0372]

[0373] The process of analyzing multimodal emotions by the multimodal emotion analysis module is as follows: First, the values ​​of six indicators—audio_arousal, audio_dominance, audio_valence, face_arousal, face_intensity, and face_valence—are obtained from the cloud database and mapped to the PAD emotion table to obtain the predicted emotions for the current face and audio. The multimodal emotion analysis module visualizes audio_arousal, audio_dominance, and audio_valence as three-dimensional images and face_arousal and face_valence as two-dimensional images, and stores them in the cloud database along with the analysis results.

[0374] The process of the result feedback module sending back the emotion results is as follows: After the cloud server receives the data and finishes the analysis, it retrieves the real-time analysis results from the database to form a dictionary, packages it in JSON format, and sends it back to the computer via TCP / IP. Users can then view the currently predicted facial audio and EEG data, PAD emotion type, audio 3D stereoscopic image, and facial 2D planar image.

[0375] The process of generating the emotion analysis report in the results report display module is as follows: After the user finishes recording, the cloud server assigns different weights to the PAD emotion index obtained from the audio facial analysis and the emotion index obtained from the EEG analysis, resulting in an analysis report of positive emotions, negative emotions, and physiological emotions.

[0376] Positive emotion indicators include self-confidence, self-regulation ability, vitality, and mind-body balance.

[0377] Self-confidence is a probability indicator that reflects emotions within a moderately positive range in the emotional space. It is calculated by weighting the probabilities of all three dimensions of emotion in image and audio signals being greater than 0.

[0378] Self-regulation ability refers to the speed at which an intensity changes from high to low. It is calculated by summing the absolute changes between all adjacent elements, dividing it by the number of changes between elements, and obtaining the average rate of change. Self-regulation ability is assessed based on the average rate of change. Intensity is used for images and dominance is used for audio.

[0379] Vitality is derived by weighting the arousals of the two modalities from [-3,3] to [0,100], with the entire process calculated using the average sentiment value.

[0380] Mind-body balance is assessed through a comprehensive evaluation of tension, anxiety, and stress. The higher the anxiety, the lower the mind-body balance.

[0381] Negative emotion indicators include aggression, tension, facial stress, and suspicion.

[0382] Aggression was calculated using a combination of two dimensions: hostility and uncontrollability, using only audio data; the final result was obtained by weighting the percentages of the two emotions as a distance metric.

[0383] The stress index is comprehensively assessed by combining results from EEG, images, and audio. A distance metric percentage is used, and the final percentage is obtained by weighting the audio, video, and EEG data. Specifically, stress in the audio is replaced with anxiety, and the stress value is used directly from the EEG data.

[0384] Facial stress is assessed through image results, with intensity directly mapped from [-3,3] to [0,100], and calculated using the average emotional value over the entire process.

[0385] Suspiciousness is calculated using a comprehensive approach across three dimensions: vigilance, tension, and alertness. First, the distances between image, audio, and EEG data are calculated as percentages within a customized PAD emotional space. Then, the vigilance, tension, and alertness of the three modalities are weighted to arrive at the final suspiciousness score. Specifically, tension in the audio is derived through anxiety conversion, and alertness in the EEG is derived through tension conversion.

[0386] Physiological and emotional indicators include attention, relaxation, fatigue index, anxiety index, stress index, depression index, alertness, dominance, clarity, happiness, depression, and energy.

[0387] Attention and relaxation levels were both determined by EEG biofeedback analysis using Rawdata.

[0388] The fatigue index is assessed using the arousal of images, audio, and EEG. The arousals of the three are directly mapped to [0, 100], and then weighted proportionally to obtain the final percentage of non-fatigue. The fatigue index percentage is then obtained by subtracting the non-fatigue percentage from 100%.

[0389] The anxiety index is calculated using a distance metric. It measures the distance between the current emotion and the anxiety level based on the anxiety's location within the PAD emotional space.

[0390] The stress index is calculated based on arousal, valence, and dominance, and is assessed according to the PAD emotional space.

[0391] The depression index is calculated based on arousal, valence, and dominance, and is assessed according to the PAD emotional space.

[0392] The alertness is obtained by adding the weights of the beta wave (0.9) and the gamma wave (0.1).

[0393] The control index is used for a comprehensive assessment combining audio and EEG results. A distance metric percentage is used, and the final percentage is obtained by weighting the audio and EEG data.

[0394] Awareness indicators are comprehensively assessed by combining audio and EEG results. A distance metric percentage is used, and the audio and EEG data are weighted to obtain the final percentage, with EEG data having the primary weight.

[0395] Happiness is obtained by mapping arousal to [0,100].

[0396] Depression is represented by a weighted average of dominance and happiness. High dominance and low happiness represent depression, calculated by subtracting the weighted percentages of happiness and dominance from 100%.

[0397] Energy is typically calculated from arousal and intensity. The energy is obtained by multiplying the arousal and intensity at each time step, where arousal comes from the weighted sum of the three modalities and intensity comes from the image modality.

[0398] The calculation of the multimodal sentiment index uses various calculation formulas such as mapping, weighting, and PAD space mapping, which are explained in detail below.

[0399] The mapping process is represented by the following formula:

[0400] X i =m·L i +c

[0401] Among them, L i X represents the original probability value. i is the value after mapping, m is the multiplier of the mapping, and c is the constant term of the mapping.

[0402] The weighting process is represented by the following formula:

[0403]

[0404] Among them, X i For the i-th average value, w i Let be the weight corresponding to the i-th average value, and n represent the total number of values ​​involved in the calculation.

[0405] The mapping process of PAD's emotional space is as follows:

[0406] Calculate the maximum distance D from each extreme point in the PAD emotional space to the emotional coordinate T. max :

[0407] D max =max{D1,D2,…,D n}

[0408] Calculate the distance D between the current emotion and the emotion coordinate T. 当前情绪 :

[0409]

[0410] Where, T = [T x T y T z [x, y, z] represents the coordinates of a certain emotional position in the emotional space, and E = [x, y, z] represents the PAD emotional space coordinates of the current emotion.

[0411] Calculate the percentage of emotion using the following formula:

[0412]

[0413] The energy index is calculated as follows:

[0414] For each emotion pair (a i i i The energy is calculated as the product of its absolute values, as shown in the following formula:

[0415] energy i =|a i |×|i i |

[0416] If the energy list is empty, return 0.

[0417] The maximum energy value `max_energy` in the energy list is calculated to normalize the energy values ​​to the range [0, 100], as follows:

[0418]

[0419] Summing the normalized energy lists and dividing by the length of the energy lists yields the average energy value, as shown in the following formula:

[0420]

[0421] Here, len(normalized_energy) represents the length of the energy list.

[0422] The computer integrates an audio / video EEG data collection module and a user interaction module. These modules are used to collect audio / video EEG data via computer components and to interact with the algorithm through a computer interface. The audio / video EEG data collection module collects audio data (Raw_A) via a microphone, image data (Raw_V) via a camera, and EEG data (Raw_B) via the EEG data collection module. In the user interaction module, users interact with the algorithm using a mouse and the computer screen, starting and stopping recording, modifying parameters, and viewing results.

[0423] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A multimodal real-time mental health analysis system based on artificial intelligence methods, characterized in that: It includes an EEG biofeedback device, a cloud server, and a computer; The EEG biofeedback device communicates with the computer via USB serial port, while the computer communicates with the cloud server via TCP / IP. An EEG biofeedback device deploys an EEG acquisition module and a signal analysis module. The EEG acquisition module receives raw EEG signals, and the signal analysis module analyzes the raw EEG signals and transmits them to a computer. The process of the EEG acquisition module acquiring EEG signals is as follows: it receives raw EEG data, denoted as Rawdata, and converts it into five types of EEG signals: Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave, a total of five unsigned values. It uses a horizontal dynamic comparison method, and retains the data and transmits it to the signal analysis module if the packet loss rate is less than 10% and the checksum is passed. The signal analysis module performs signal analysis and transmission as follows: it calculates the attention and relaxation levels using the five types of brainwave signals and maps them to a range of 0 to 100. Then, it transmits the seven types of brainwaves (Attention, Meditation, Delta, Theta, Alpha, Beta, and Gamma) to the computer via USB serial communication. The computer program for deploying deep learning models and data analysis on a cloud server includes an interactive data storage module, an interactive parameter storage module, an audio processing module, an image processing module, an EEG processing module, a multimodal emotion analysis module, a result feedback module, and a result report display module. The functions of each module are as follows: the cloud server receives the audio stream, data stream, and EEG signal stream fed back from the audio-visual EEG collection module on the computer; the interactive data storage module stores the data in the cloud database in the order of timestamps. The cloud server accepts parameters input by the user during the interaction process, and the interaction parameter storage module stores them in the cloud database in timestamp order; The audio processing module performs emotion analysis based on audio data; the image processing module performs emotion analysis based on image data; the EEG processing module performs emotion analysis based on EEG data; the multimodal emotion analysis module receives emotion indicators from EEG, image, and audio modalities in real time and performs multimodal emotion analysis; the result feedback module sends the emotion analysis results back to the computer in real time for users to view; and the result report display module generates an emotion analysis report after recording is completed and sends it back to the computer. The specific process by which each module of the cloud server completes its function is as follows: The process of storing interactive data in the interactive data storage module is as follows: the computer transmits the audio signals collected by the microphone, the image data collected by the camera, and the brainwave signals collected by the brainwave collection module to the cloud server in the form of real-time data stream via TCP / IP. The interactive data storage module stores the data into the cloud database in the order of timestamps. The interaction parameter storage module stores interaction parameters as follows: the parameters entered by the user are stored in the local browser cache through the localStorage method. When the cloud server starts, it will retrieve the parameters entered by the user from the local browser cache. After the call is completed, the interaction data will be deleted to ensure the protection of user privacy. The audio processing module handles audio data and performs sentiment analysis as follows: First, the audio processing module converts the Raw_A audio signal received from the cloud server into mono data and resamples it to 16000Hz. Then, it divides the signal into segments of fixed duration. The preprocessed audio data is then used for sentiment recognition through a trained audio prediction model. The specific process is as follows: The input audio signal x(t) is first processed by a wav2vec pre-trained model as a feature encoder to obtain a feature vector sequence X = f_encoder(x(t)). The feature vector sequence X is then projected into a higher-dimensional space to match the input dimension of the Transformer model. After Transformer encoding and pooling operations, three emotion indicators are finally obtained: audio_arousal, audio_dominance, and audio_valence, with values ​​ranging from [0,1]. These emotion indicators are further mapped to the range of [-3,3]. The initial analysis period is set to 2, which can be modified according to actual user needs. Finally, the emotion indicator values ​​are stored in a cloud database according to timestamps. The image processing module processes image data and performs sentiment analysis as follows: First, the image processing module sends the image signal Raw_V received from the cloud server into a pre-trained DLIB face detection model to detect whether a face appears in the image and selects the largest face. Next, the selected face is sent into a pre-trained DLIB facial landmark detection model for landmark detection, thereby generating an object containing the detected facial landmarks, denoted as landmarks_object. Subsequently, the landmarks_object is passed to the pre-trained sentiment estimator. The sentiment estimator uses partial least squares regression analysis to construct a two-dimensional sentiment space prediction model and employs the Neutral to Apex morphing method to calculate sentiment intensity, resulting in three key facial sentiment indicators: face_arousal, face_intensity, and face_valence. The value range of face_intensity is [0,1], while the values ​​of the other two indicators range from [-1,1]. These three facial sentiment indicators are further mapped to the range of [-3,3]. The output period is once per frame, and the three facial sentiment indicators are stored in the cloud database according to the timestamp. The EEG processing module handles EEG data and performs emotion analysis as follows: First, the module segments the Raw_B EEG signal received from the cloud server according to a fixed period. For the data within each period, the mean period of Delta, Theta, Alpha, Beta, and Gamma waves is calculated separately. Based on this, the standard deviation of these wavebands is calculated, and a probability density function of a normal distribution is constructed. The integral value of the data for the current second on the normal distribution is calculated. Then, the integral value is normalized and converted into a standard score. Based on these standard scores and preset weights, seven emotion indicators are calculated: light (happiness), dominance, alertness, clearness, nervousness, attention, and meditation. The calculation of each emotion indicator is based on the comprehensive weight of different waveband signals to ensure the accuracy and reliability of emotion analysis. The multimodal sentiment analysis module analyzes multimodal emotions as follows: First, it retrieves the values ​​of six indicators—audio_arousal, audio_dominance, audio_valence, face_arousal, face_intensity, and face_valence—from the cloud database and maps them to the PAD sentiment table to obtain the predicted emotions for the current face and audio. The multimodal sentiment analysis module then visualizes audio_arousal, audio_dominance, and audio_valence as three-dimensional images and face_arousal and face_valence as two-dimensional images, storing them along with the analysis results in the cloud database. The process of the result feedback module sending back the emotion results is as follows: After the cloud server receives the data and finishes the analysis, it retrieves the real-time analysis results from the database to form a dictionary, packages it in JSON format, and sends it back to the computer via TCP / IP. Users can view the currently predicted facial audio and EEG data, PAD emotion type, audio 3D stereoscopic image and facial 2D planar image. The process of generating the emotion analysis report in the results report display module is as follows: After the user ends the recording, the cloud server assigns different weights to the PAD emotion index obtained from the audio facial analysis and the emotion index obtained from the EEG analysis, and obtains an analysis report of positive emotion, negative emotion and physiological emotion. The computer integrates an audio / video EEG data collection module and a user interaction module. These modules are used to collect audio / video EEG data via computer components and to interact with the algorithm through a computer interface. Specifically, the computer collects audio data (Raw_A) via a microphone, image data (Raw_V) via a camera, and EEG data (Raw_B) via the EEG data collection module. Users can operate the algorithm using a mouse and the computer screen to start and stop recording, modify parameters, and view results.

2. The multimodal real-time mental health analysis system based on artificial intelligence methods according to claim 1, characterized in that: The specific calculation method for the analysis report of positive emotions, negative emotions, and physiological emotions is as follows: Positive emotion indicators include self-confidence, self-regulation ability, vitality, and mind-body balance; The self-confidence mentioned is a probability index that the emotion is in the moderate positive range of the emotional space; it is calculated by weighting the probability that all three dimensions of emotion in the image and audio signals are greater than 0. The self-regulation ability refers to the speed at which the intensity changes from high to low. It is calculated as follows: calculate the sum of the absolute changes between all adjacent elements, divide it by the number of changes between elements, and obtain the average rate of change. The self-regulation ability is evaluated based on the average rate of change. Intensity is used for images and dominance is used for audio. The vitality is obtained by weighting the arousals of the two modalities from [-3,3] to [0,100], and the entire process is calculated using the average sentiment value. The mind-body balance is assessed through a comprehensive evaluation of tension, anxiety, and stress; the higher the anxiety, the lower the mind-body balance. Negative emotion indicators include aggression, tension, facial stress, and suspicion; The aggression was calculated using two dimensions: hostility and uncontrollability, using only audio data; the final result was obtained by weighting the percentages of the two emotions as a distance metric. The tension was comprehensively assessed using a combination of EEG, image, and audio results; a distance metric percentage was used, and the audio, video, and EEG data were weighted to obtain the final percentage; where tension in the audio was replaced with anxiety, and tension values ​​were used directly from the EEG data. The facial expression stress is evaluated through image results, mapping intensity directly from [-3,3] to [0,100], and calculated using the average emotional value throughout the process; The suspiciousness is calculated by comprehensively converting three dimensions: vigilance, tension, and alertness. First, the distance between image, audio, and EEG data is calculated as a percentage in a custom PAD emotional space. Then, the vigilance, tension, and alertness of the three modalities are weighted to obtain the final suspiciousness. Among them, the tension in the audio is obtained by anxiety conversion, and the alertness in the EEG is obtained by tension conversion. Physiological and emotional indicators include attention, relaxation level, fatigue index, anxiety index, stress index, depression index, alertness, dominance, clarity, happiness, depression, and energy. The levels of attention and relaxation were obtained by EEG biofeedback analysis using Rawdata. The fatigue index is assessed using the arousal of images, audio, and brainwaves. The arousals of the three are directly mapped to [0, 100], and then weighted proportionally to obtain the final percentage of non-fatigue. The fatigue index percentage is then obtained by subtracting the percentage of non-fatigue from 100%. The anxiety index is calculated using a distance metric; the distance between the current emotion and the anxiety is calculated based on the location of the anxiety in the PAD emotional space. The stress index is calculated based on arousal, valence, and dominance, and is assessed according to the PAD emotional space. The depression index is calculated based on arousal, valence, and dominance, and is assessed according to the PAD emotional space. The alertness is obtained by adding the proportion of beta waves (weighted at 0.9) and the proportion of gamma waves (weighted at 0.1). The assessment combines audio and EEG results for a comprehensive evaluation; a distance metric percentage is used, and the audio and EEG data are weighted to obtain the final percentage. The results of the awakening were comprehensively evaluated by combining audio and brainwave data; a distance metric percentage was used, and the audio and brainwave data were weighted to obtain the final percentage, with brainwave data having the main weight. The happiness is obtained by mapping arousal to [0, 100]. The depression is represented by a weighted average of dominance and pleasure; high dominance and low pleasure represent depression, calculated by subtracting the weighted percentages of happiness and dominance from 100%. The energy is calculated using arousal and intensity; the energy is obtained by multiplying the arousal and intensity at each time step, where arousal comes from the weighted sum of the three modalities and intensity comes from the image modality.

3. The multimodal real-time mental health analysis system based on artificial intelligence methods according to claim 2, characterized in that: The mapping process used in the calculation of positive emotions, negative emotions, and physiological emotions is represented by the following formula: X i =m·L i +c Among them, L i X represents the original probability value. i Let m be the value after mapping, m be the multiplier of the mapping, and c be the constant term of the mapping. The weighting process is represented by the following formula: Among them, X i For the i-th average value, w i Here, represents the weight corresponding to the i-th average value, and n represents the total number of values ​​involved in the calculation. The mapping process of the PAD emotional space is as follows: Calculate the maximum distance D from each extreme point in the PAD emotional space to the emotional coordinate T. max : D max =maX{D1,D2,...,D n } Calculate the distance D between the current emotion and the emotion coordinate T. 当前情绪 : Where, T = [T x T y T z [x, y, z] represents the coordinates of a certain emotional position in the emotional space, and E = [x, y, z] represents the PAD emotional space coordinates of the current emotion; Calculate the percentage of emotion using the following formula: The energy calculation is as follows: For each emotion pair (a i i i The energy is calculated as the product of its absolute values, as shown in the following formula: energy i =|a i ×|i i | If the energy list is empty, return 0; The maximum energy value `max_energy` in the energy list is calculated to normalize the energy values ​​to the range [0, 100], as follows: Summing the normalized energy lists and dividing by the length of the energy lists yields the average energy value, as shown in the following formula: Here, len(normalized_energy) represents the length of the energy list.

4. The multimodal real-time mental health analysis system based on artificial intelligence methods according to claim 3, characterized in that: The specific process of training the CNN14 model in the audio processing and audio prediction model of the audio processing module is as follows: Step 1: Generate a log-Mel spectrum; Step 1.1: Frame division; The audio signal x(t) is divided into several time frames. Each frame is weighted using a window function w(k). The purpose of the window function is to reduce spectral leakage and smooth edges, as shown in the following equation: x f [n]=x[n·H+k]·w[k],k=0,1,...,L-1 Where H is the step size, 10 milliseconds; L is the window size, 32 milliseconds; k represents the sampling point index within the window function w[k], from 0 to L-1; x f [n] represents the information of the nth frame after processing by the window function, and f is the frame index used to distinguish different frames; Step 1.2: Perform a short-time Fourier transform; Perform a Fourier transform on each frame according to the following formula: Where Π represents pi, X f [m, k] is the k-th frequency component of the m-th frame; Step 1.3: Convert to Mel frequency scale; Convert the spectrum to Mel spectrum using a Mel filter: Where f represents the index of the Mel frequency band, H[f, k] is the filter coefficient of the Mel filter bank, representing the response of the f-th Mel frequency band to the k-th frequency component; Step 1.4: Take the logarithm; Taking the logarithm of the Mel spectrum yields the log-Mel spectrum: Log-Mel(m,f) = log(M[m,f]) Step 2: Feed the log-Melogram obtained in Step 1 as a feature map into the CNN14 model. The CNN14 model consists of 6 convolutional blocks, each with two layers. Each layer has a max pooling operation, where the convolutional kernel size is 3*3 with a stride of 1, and the max pooling layer has a stride of 2. Details are as follows: Step 2.1: Convolutional layer; The output of each convolutional layer is represented as follows: C l =σ(W l *X l +b l ) Among them, W l It is the convolution kernel, * indicates the convolution operation, b l σ is the bias, X is the activation function, and X is the activation function. l It is the input feature log-Mel spectrogram; Step 2.2: Max pooling layer; The output of each pooling layer is represented as follows: P l =MaxPool(C l ) Step 2.3: Dropout layer; Apply Dropout after each convolutional block: D l =Dropout(P l ,p) Where p is the probability of Dropout; Step 2.4: Average pooling and max pooling; The output of the last convolutional layer is subjected to both average pooling and max pooling: Where N represents the total number of elements in the feature map, C i This represents the value of the i-th element in the feature map; Step 2.5: Fully connected layer; The pooled features are then input into the fully connected layer: F1=σ(W1·[MeamPool;MaxPool]+b1) F2=σ(W2·F+b2) Where W1 and W2 represent the weight matrices of the fully connected layer, b1 and b2 represent the corresponding bias terms, and σ represents the activation function; After convolutional layer transformation, the output time series feature representation is given, with a dimension of T×512, where T is the time step of the feature vector. The output feature is then transformed into a high-dimensional feature representation with a dimension of 1024, as shown in the following formula: H=f project (F2)=FW p +b p Among them, f project Denotes the projection function, W p and b p These represent the weight matrix and bias term of the projection layer, respectively; Step 3: Transformer encoding layer; The Transformer encoder consists of 24 Transformer layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network, with a dimension of 1024. Step 3.1: Multi-head self-attention mechanism; The input to the first layer is H, and for the input H of the l-th layer... (L-1) , as follows: Step 3.1.1: Linear Transformation: Q=H (l-1) W Q ,K=H (l-1) W K ,V=H (l-1) W V Among them, W Q 、W K 、W v All dimensions are 1024×1024; W o 、W K and W v These are the weight matrices for the query, key, and value, respectively. Q, K, and V are the query, key, and value matrices obtained through linear transformations. Step 3.1.2: Calculate the self-attention score: in, h is the number of heads; Step 3.1.3: Multi-headed attention: MultiHead(Q,K,V)=Concat(head1,…,head h )W O The output of each attention head is head. i =Attention(Q) i ,K i V i );W O It is a weight matrix used for linear transformation after splicing, with a dimension of 1024×1024; Step 3.2: Feedforward Neural Network; Step 3.2.1: First-level linear transformation and activation function: FFN1(x)=ReLU(xW1+b1) Where W1 has a dimension of 1024×4096, and b1 is the bias; Step 3.2.2: Second-level linear transformation: FFN2(x)=ReLU(xW1+b1)W2+b2 Where W2 has a dimension of 4096×1024, and b2 is the bias; Step 3.3: Perform layer normalization and residual connections after each sub-layer: Step 3.3.1: Self-attention sublayer: H (l) =LayerNorm(H (l-1) +MultiHead(Q,K,V)) Step 3.3.2: Feedforward Neural Network Sublayer: O=LayerNorm(H (l) +FFN(H (l) )) After 24 Transformer layers, the final output O is obtained; Step 4: Pooling operation: Pooling is applied to the output O of the last layer of the Transformer encoder to obtain a fixed-size output vector: p=f pool (O) Step 5: Output: The pooled feature vector p is passed through one or more fully connected layers, and then through an output layer to predict the sentiment dimension score: y=f output (P) Where y is the predicted three-dimensional vector, including scores for arousal, dominance, and sentiment value: y1=f arousal (P) y2=f dominance (P) y3=f valence (P) Among them, arousal, dominance, and valence represent arousal, dominance, and affectivity, respectively, and are three commonly used dimensions to describe emotional states.

5. The multimodal real-time mental health analysis system based on artificial intelligence methods according to claim 4, characterized in that: The image processing module performs image processing and uses a sentiment estimator based on geometric features, finite element analysis, and partial least squares regression as follows: Step 1: Obtain image key points; Step 1.1: Image preprocessing; After the camera captures an image, the "cvtColor" function in the OpenCV library is used to convert it into a grayscale image; Step 1.2: Key point detection; Face detection is performed using the DLIB frontal face detector get_frontal_face_detector() to extract the outer rectangle of the face; the DLIB library-trained 68-point face feature detector is used to extract the facial contour features and obtain the coordinates of 49 filtered facial coordinate points. Step 2: Facial positivity; Step 2.1: Scaling, Translation, and Rotation: First, scale, translate, and rotate the coordinate points so that all coordinate points are on the same frontal face: in: It is the position point or feature vector of the k-th sample in the original coordinate system, where o represents the original coordinate system; Indicates all The mean vector is the mean of all sample points in the original coordinate system, used for translation processing; R k This is the rotation matrix used to adjust the orientation of coordinate points; k: represents the index of the sample; o: the identifier in the original coordinate system, used to indicate the point in the original system; M: represents the coordinate system after scaling, translation, and rotation. Step 2.2: Construct a transpose matrix A containing the coordinates of all non-frontal expressions and a ground truth matrix Y containing the coordinates of the corresponding frontal expressions, and fill the normalized coordinates into the matrices; Step 2.3: Objective function: argmin X |Y-AX|2+λ|X|2 Where λ is the regularization parameter; X represents the parameter matrix to be optimized; Step 2.4: Predicted values ​​for positive mapping X: Where I represents the identity matrix, and its dimension is the same as A. T A is the same; T indicates transpose; Step 3: Partial Least Squares Regression; Step 3.1: Input and Output: X1: Initial feature, with a size of N×1176, is a 1176-dimensional feature calculated from the pairwise distances of 49 facial landmarks; Y1: Response feature, with a size of N×3, including three dimensions of output; Step 3.2: Model Representation: X1=TP T +E Y1=UQ T +F Where T and U are latent variable matrices: of size N×29, with 29 latent variables; P and Q are loading matrices; E and F are residual matrices; Step 3.3: Objective function; The goal is to maximize the covariance between the response features and the predicted features: Where t and u are column vectors of T and U, respectively; X1 represents the initial feature matrix, used for predicting feature data; Y1 represents the response feature matrix, containing the values ​​of the target variable; w and v are two direction vectors, used to project data into the feature space and response space, respectively; X1w represents the product of matrix X1 and vector w, the result of which is the projection along the w direction in the feature space; Y1v represents the product of matrix Y1 and vector v, the result of which is the projection along the v direction in the response space. Step 3.4: Solve for the regression coefficient matrix B using the following formula, which is used to map the feature matrix X1 to the predicted response: B=X1 T U(T T X1X1 T U) -1 T T Y1 Step 3.5: Predict the response matrix: The final predicted response matrix is ​​calculated using the following formula: in, The predicted response feature matrix includes scores for arousal, sentiment, and intensity.

6. The multimodal real-time mental health analysis system based on artificial intelligence methods according to claim 4, characterized in that: The specific steps for processing EEG data and scoring emotion indicators are as follows: Step 1: Calculate the mean: The input is the value x of the i-th data point. i The total number of data points, n; The output is the mean μ of the data points; Step 2: Calculate the standard deviation: The input is the mean μ of the data points, and the output is the standard deviation σ of the data points. Step 3: Construct the probability density function of the normal distribution: The inputs are the current data point value x, the mean μ of the data point, and the standard deviation σ of the data point; The output is the probability density value f(x) of the data point (x) under the normal distribution; Step 4: Calculate the integral of the data for the current second on this distribution: The input is the value of the data point in the current second, x_current, and the probability density function f(x) of the data point (x) under a normal distribution; the output is the cumulative distribution function P(x) of the current data point under a normal distribution. current ); Step 5: Normalize and convert to fractions: Score=100×P(x current ) The input is the cumulative distribution function value P(x) of the current data point on a normal distribution. current ); The output is the normalized score for the current data point; Step 6: The overall process formula is:

Images