Multi-modal real-time mental health analysis system based on artificial intelligence method
Through a multimodal real-time mental health analysis system based on artificial intelligence, combined with brain waves, images and audio data, detailed emotion analysis is provided, and the problems of insufficient accuracy and insufficient real-time performance in the existing technology are solved, and the problem of insufficient single-modal analysis is achieved. Highly accurate and efficient multimodal sentiment analysis.
Patent Information
- Application Number
- CN202411150269.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-21
AI Technical Summary
The existing emotion analysis technology has limitations when expressing more multi-dimensional and three-dimensional emotional indicators, and cannot provide detailed and accurate emotional data. It is mainly focused on single-modal emotion analysis, and it is not able to effectively combine multi-modal data for synchronous analysis. This results in insufficient accuracy of the analysis results and cannot meet the needs of real-time application scenarios.
Using a multimodal real-time mental health analysis system based on artificial intelligence methods, it provides a comprehensive, accurate and real-time multimodal emotion analysis algorithm by integrating data from three modalities: brain waves, images and audio, combining deep learning models and data. Analytical technology processes audio, image and brain wave signals, generates emotional indicators, and provides a more intuitive display of emotional changes through multimodal fusion emotional expression.
Improve the accuracy of emotion recognition and the robustness of algorithms, realize the synchronous analysis of multimodal data, improve the accuracy of emotion analysis results, ensure the timeliness and accuracy of emotion analysis, and provide comprehensive emotional indicators. Including positive emotions, negative emotions and physiological emotions, it improves users' cognition and understanding of emotional state.
Smart Images

Figure CN119069081A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of emotion analysis, and in particular to a multimodal real-time mental health analysis system based on an artificial intelligence method. Background Art
[0002] In recent years, with the rapid development of artificial intelligence and sensor technology, sentiment analysis technology has received widespread attention and application. Traditional sentiment analysis technology still has limitations in expressing more multi-dimensional and three-dimensional sentiment indicators, and cannot provide detailed and accurate sentiment data. Its sentiment analysis and expression capabilities have not yet been perfected.
[0003] To overcome these limitations, researchers have proposed a variety of new sentiment analysis techniques, including:
[0004] Emotional audio analysis technology: It analyzes the user's audio signals to determine their emotional state, which is a widely used way of expressing emotions;
[0005] Emotional image analysis technology: Emotional analysis based on the user's facial expressions, identifying emotions by capturing facial features;
[0006] Emotional brainwave analysis technology: Analyzes the user's brainwave data to obtain specific emotional indicators, providing an internal method for evaluating emotional states.
[0007] Although these technologies have made significant progress in their respective fields, existing sentiment analysis algorithms still have shortcomings in the following aspects:
[0008] (1) Failure to reasonably associate numerical indicators with emotional states and lack of in-depth understanding of the interactions between different indicators;
[0009] (2) There are deficiencies in the display and expression of emotional indicators, and it is impossible to intuitively present emotional changes;
[0010] (3) Existing technologies mainly focus on single-modal sentiment analysis and fail to effectively combine multimodal data for synchronous analysis, resulting in insufficient accuracy of analysis results;
[0011] (4) Existing sentiment analysis technologies have certain delays in data processing and feedback, and cannot meet the needs of real-time application scenarios, such as real-time customer service, online education, etc. The lack of real-time performance may reduce the timeliness and practicality of sentiment analysis. Summary of the invention
[0012] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a multimodal real-time mental health analysis system based on artificial intelligence methods. By fusing data from three modalities, namely brain waves, images and audio, a comprehensive, accurate and real-time multimodal emotion analysis algorithm is provided. The algorithm can not only process and analyze multimodal data in real time and improve the accuracy of emotion recognition, but also significantly improve the robustness and accuracy of the algorithm.
[0013] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0014] A multimodal real-time mental health analysis system based on artificial intelligence method includes an electroencephalogram (EEG) biofeedback instrument, a cloud server and a computer; the EEG biofeedback instrument communicates with the computer using a USB serial port, and the computer communicates with the cloud server using TCP / IP;
[0015] A brain wave acquisition module and a signal analysis module are deployed in the EEG biofeedback instrument. The brain wave acquisition module receives the original brain wave signal, and the signal analysis module analyzes the original brain wave signal and transmits the signal to the computer.
[0016] A computer program for deep learning model and data analysis is deployed on a cloud server, including an interactive data storage module, an interactive parameter storage module, an audio processing module, an image processing module, an EEG processing module, a multimodal emotion analysis module, a result feedback module and a result report display module. The functions of each module are as follows: the cloud server receives the audio stream, data stream and EEG signal stream returned by the audio and video EEG collection module in the computer, and the interactive data storage module stores them in the cloud database in the order of timestamps; the cloud server receives the parameters input by the user during the interaction process, and the interactive parameter storage module stores them in the cloud database in the order of timestamps; the audio processing module performs emotion analysis based on the audio data; the image processing module performs emotion analysis based on the image data; the EEG processing module performs emotion analysis based on the EEG data; the multimodal emotion analysis module is used to receive the emotion indicators of the EEG-image-audio modality in real time and perform multimodal emotion analysis; the result feedback module is used to transmit the emotion analysis results back to the computer in real time for the user to view on the computer; the result report display module is used to generate the emotion analysis report of this time after the recording is completed, and transmit it back to the computer;
[0017] An audio and video brainwave collection module and a user interaction module are integrated in the computer, which are used to collect audio and video brainwave data through computer components and to perform algorithm interaction through a computer interface, specifically: collecting audio data Raw_A through a microphone, collecting image data Raw_V through a camera, and collecting brainwave data Raw_B through a brainwave collection module; users use a mouse and a computer screen to perform algorithm operations, start and stop recording, modify parameters, and view results.
[0018] Furthermore, the process of collecting brain waves by the brain wave acquisition module is as follows: receiving raw brain wave data, recorded as Rawdata, and converting it into five brain wave signals: Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave, a total of five unsigned numerical values, using a horizontal dynamic comparison method, retaining the data and passing it to the signal analysis module when the packet loss rate is less than 10% and passes the checksum;
[0019] The signal analysis module performs signal analysis and transmission as follows: the concentration and the relaxation are calculated through the five brain wave signals and mapped to between 0 and 100, and the seven brain waves, Attention, Meditation, Delta, Theta, Alpha, Beta and Gamma, are transmitted to the computer through the USB serial port communication.
[0020] Furthermore, the process of each module of the cloud server completing its function is as follows:
[0021] The process of storing interactive data in the interactive data storage module is as follows: the computer transmits the audio signal collected by the microphone, the image data collected by the camera, and the brain wave signal collected by the brain wave collection module to the cloud server via TCP / IP in the form of real-time data stream, and the interactive data storage module stores the data in the cloud database in the order of timestamps;
[0022] The process of storing interaction parameters in the interaction parameter storage module is as follows: the parameters input by the user will be stored in the local browser cache through the localStorage method. When the cloud server starts, the parameters input by the user will be called from the local browser cache. After the call is completed, the interaction data will be deleted to ensure the protection of user privacy.
[0023] The process of audio processing module processing audio data and emotion analysis is as follows: First, the audio processing module converts the audio signal Raw_A received by the cloud server into mono data, resamples it to 16000Hz, and then divides it into segments of fixed length; the pre-processed audio data is used for emotion recognition through the trained audio prediction model. The specific process is as follows:
[0024] The input audio signal x(t) is first processed as a feature encoder through the wav2vec pre-trained model to obtain a feature vector sequence X = f_encoder(x(t)); the feature vector sequence X is projected to a higher-dimensional space to match the input dimension of the Transformer model; after Transformer encoding and pooling operations, three emotion indicators, audio_arousal, audio_dominance, and audio_valence, are finally obtained. The value range of these emotion indicators is [0,1]; these emotion indicators are further mapped to the range of [-3,3]; the analysis cycle is initially set to 2, and the user can modify it according to actual needs during use; finally, the emotion indicator values will be stored in the cloud database according to the timestamp;
[0025] The process of image processing module processing image data and emotion analysis is as follows: first, the image processing module sends the image signal Raw_V received by the cloud server to the pre-trained DLIB face detection model to detect whether there is a face in the picture and select the face with the largest size; next, the selected face is sent to the pre-trained DLIB facial key point detection model for key point detection, thereby generating an object containing the detected facial key points, recorded as landmarks_object; then, the object landmarks_object is passed to the pre-trained emotion estimator, which uses partial least squares regression analysis to construct a two-dimensional emotion space prediction model and adopts Neutral to Apex The morphing method calculates the intensity of emotion and obtains three key facial emotion indicators: facial arousal, facial intensity, and facial valence. The value range of facial intensity is [0, 1], and the value range of the other two indicators is [-1, 1]. The three facial emotion indicators are further mapped to the range of [-3, 3]. The output cycle is once per frame, and the three facial emotion indicators are stored in the cloud database according to the timestamp.
[0026] The process of brainwave processing module processing brainwave data and emotion analysis is as follows: First, the brainwave processing module processes the brainwave signal Raw_B received by the cloud server in segments according to a fixed period; for the data in each period, the period mean of Delta wave, Theta wave, Alpha wave, Beta wave and Gamma wave is calculated separately; on this basis, the standard deviation of these bands is calculated, and the probability density function of the normal distribution is constructed to calculate the integral value of the data of the current second on the normal distribution; then, the integral value is normalized and converted into a standard score; based on these standard scores, combined with the preset weights, seven emotion indicators are calculated: light, dominance, alert, clear, nervous, attention and meditation; the calculation of each emotion indicator is based on the comprehensive weight of signals in different bands to ensure the accuracy and reliability of emotion analysis;
[0027] The process of analyzing multimodal emotions by the multimodal emotion analysis module is as follows: first, the values of six indicators, namely audio_arousal, audio_dominance, audio_valence, face_arousal, face_intensity and face_valence, are obtained from the cloud database, and are mapped to the PAD emotion table respectively to obtain the predicted emotions of the current face and audio; the multimodal emotion analysis module visualizes audio_arousal, audio_dominance and audio_valence into three-dimensional stereograms, and visualizes face_arousal and face_valence into two-dimensional plane graphs, and stores them in the cloud database together with the analysis results;
[0028] The process of the result feedback module returning the emotion results is as follows: after the cloud server receives the data and finishes the analysis, it obtains the real-time analysis results from the database to form a dictionary, packages it in json format, and returns it to the computer via TCP / IP. The user can view the currently predicted facial audio and brain wave data, PAD emotion type, audio 3D stereogram and facial 2D plane map;
[0029] The process of generating an emotion analysis report in the result report display module is as follows: after the user finishes recording, the cloud server assigns different weights to the PAD emotion index obtained from the audio facial analysis and the emotion index obtained from the brain wave analysis to obtain an analysis report on positive emotions, negative emotions, and physiological emotions.
[0030] Furthermore, the specific calculation method of the analysis report of the positive emotions, negative emotions and physiological emotions is as follows:
[0031] Positive emotion indicators include self-confidence, self-regulation, vitality, and physical and mental balance;
[0032] The self-confidence is a probability index of the emotion being in a medium-intensity positive interval in the emotion space; it is calculated by weighting the probability that the three dimensions of the emotion in the image and audio signals are all greater than 0;
[0033] The self-regulation ability refers to the speed of changing from high intensity to low intensity, which is calculated by calculating the sum of absolute changes between all adjacent elements and dividing it by the number of changes between elements to obtain the average change rate; the self-regulation ability is evaluated based on the average change rate, using intensity for images and dominance for audio;
[0034] The vitality is obtained by mapping the arousal of the two modes from [-3, 3] to [0, 100] and then weighting them. The whole process is calculated using the average sentiment.
[0035] The physical and mental balance is comprehensively assessed through tension, anxiety and stress. The higher the anxiety, the lower the physical and mental balance.
[0036] Negative emotion indicators include aggression, tension, expression stress, and suspiciousness;
[0037] The aggression is calculated by combining the two dimensions of hostility and uncontrollability, using only audio data; the distance is measured as a percentage, and the percentages of the two emotions are weighted to obtain the final result;
[0038] The tension index is comprehensively evaluated by combining the results of brain waves, images and audio; the distance measurement percentage is used, and the audio, video and brain wave data are weighted to obtain the final percentage; wherein the tension in the audio is replaced by anxiety, and the tension value is directly used for the brain wave data;
[0039] The expression pressure is evaluated by image results, intensity is directly mapped from [-3, 3] to [0, 100], and the average value of the emotion in the whole process is used for calculation;
[0040] The suspicion is calculated through three dimensions: vigilance, tension and alertness. First, the distances of image, audio and brain wave data are calculated in the custom PAD emotional space to obtain percentages, and then the vigilance, tension and alertness of the three modalities are weighted to obtain the final suspicion. Among them, the tension of the audio is obtained by converting anxiety, and the alertness of the brain wave is obtained by converting tension.
[0041] Physiological and emotional indicators include attention, relaxation, fatigue index, anxiety index, stress index, depression index, alertness, control, wakefulness, happiness, depression, and energy;
[0042] The concentration and relaxation levels are obtained by EEG biofeedback instrument through Rawdata analysis;
[0043] The fatigue index is evaluated by the arousal of images, audio and brain waves. The arousal of the three are directly mapped to [0,100], and then weighted in equal proportion to obtain the final non-fatigue percentage. Then, the fatigue index percentage is obtained by subtracting the non-fatigue percentage from 100%.
[0044] The anxiety index is calculated by distance measurement; according to the position of anxiety in the PAD emotional space, the distance between the current emotion and anxiety is calculated;
[0045] The stress index is calculated according to arousal, valence and dominance, and is evaluated based on the PAD emotional space;
[0046] The depression index is calculated according to arousal, valence and dominance, and evaluated according to the PAD emotional space;
[0047] The alertness is obtained by adding the weight of 0.9 given to the proportion of beta waves and the weight of 0.1 given to the proportion of gamma waves;
[0048] The control index is combined with the audio and brain wave results for comprehensive evaluation; the distance measurement percentage is used, and the audio and brain wave data are weighted to obtain the final percentage;
[0049] The wakefulness index is comprehensively evaluated by combining the audio and brain wave results; using a distance metric percentage, and weighting the audio and brain wave data to obtain a final percentage, in which the brain wave data accounts for the main weight;
[0050] The happiness is obtained by mapping arousal to [0,100];
[0051] The depression is represented by the weighted dominance and happiness; high dominance and low happiness represent depression, calculated as 100% minus the weighted percentage of happiness and dominance;
[0052] The energy is converted by arousal and intensity; the energy is obtained by multiplying the arousal and intensity at each moment, where arousal comes from the weighted addition of the three modalities and intensity comes from the image modality.
[0053] Furthermore, the mapping process used in the calculation of the positive emotion, the negative emotion and the physiological emotion is expressed by the following formula:
[0054] X i =m·Li +c
[0055] Among them, L i is the original probability value, X i is the mapped value, m is the mapped multiplier, and c is the mapped constant term;
[0056] The weighting process is expressed by the following formula:
[0057]
[0058] Among them, X i is the i-th mean value, w i is the weight corresponding to the i-th average value, and n represents the total number of values involved in the calculation;
[0059] The mapping process of the PAD emotion space is as follows:
[0060] Calculate the maximum distance D between each extreme point in the PAD emotional space and the emotional coordinate T max :
[0061] D max =max{D1,D2,…,D n}
[0062] Calculate the distance D between the current emotion and the emotion coordinate T 当前情绪 :
[0063]
[0064] Where T = [T x , T y , T z ] is the coordinate of a certain emotion position in the emotion space, E = [x, y, z] is the PAD emotion space coordinate of the current emotion;
[0065] Calculate the percentage of emotion as follows:
[0066]
[0067] The calculation of the energy index is as follows:
[0068] For each emotion pair (a i ,i i ), calculate the energy by multiplying its absolute value as follows:
[0069] Energy i =|a i |×|i i |
[0070] If the energy list is empty, it returns 0;
[0071] Calculate the maximum energy value max_energy in the energy list to normalize the energy value to the range of [0,100] as follows:
[0072]
[0073] Sum the normalized energy lists and divide by the length of the energy lists to get the average energy value, as follows:
[0074]
[0075] Among them, len(normalized\_energy) represents the length of the energy list.
[0076] Furthermore, the specific process of the audio processing in the audio processing module and the CNN14 model training in the audio prediction model is as follows:
[0077] Step 1: Generate a log-Mel spectrogram;
[0078] Step 1.1: Frame division;
[0079] The audio signal x(t) is divided into several time frames, and each frame is weighted using the window function w(k). The purpose of the window function is to reduce spectrum leakage and smooth the edges, as shown in the following formula:
[0080] x f [n]=x[n·H+k]·w[k],k=0,1,…,L-1
[0081] Where H is the step size, 10 milliseconds; L is the window size, 32 milliseconds, k represents the sampling point index within the window function w[k], from 0 to L-1, x f [n] represents the information of the nth frame after being processed by the window function, and f is the index of the frame, which is used to distinguish different frames;
[0082] Step 1.2: Perform short-time Fourier transform;
[0083] Perform the Fourier transform on each frame as follows:
[0084]
[0085] Among them, Π represents pi, which is approximately equal to 3.14159, and X f [m,k] is the kth frequency component of the mth frame;
[0086] Step 1.3: Convert to Mel frequency scale;
[0087] Use the Mel filter to convert the spectrum to a Mel spectrum:
[0088]
[0089] Where, f represents the index of the Mel band, H[f,k] is the filter coefficient of the Mel filter bank, which represents the response of the f-th Mel band to the k-th frequency component;
[0090] Step 1.4: Take the logarithm;
[0091] Take the logarithm of the Mel spectrum to get the log-Mel spectrum:
[0092] Log-Mel(m,f)=log(M[m,f])
[0093] Step 2: The log-Mel spectrogram obtained in step 1 is sent to the CNN14 model as a feature map. The CNN14 model includes 6 convolution blocks, each of which has two layers. Each layer has a maximum pooling operation, where the convolution kernel size is 3*3, the step size is 1, and the maximum pooling layer step size is 2; the details are as follows:
[0094] Step 2.1: Convolutional layer;
[0095] The output of each convolutional layer is expressed as:
[0096] C l =σ(W l *X l +b l )
[0097] Among them, W l is the convolution kernel, * indicates the convolution operation, b l is the bias, σ is the activation function, X l is the characteristic log-mel spectrogram of the input;
[0098] Step 2.2: Max pooling layer;
[0099] The output of each pooling layer is represented as:
[0100] P l =MaxPool(C l )
[0101] Step 2.3: Dropout layer;
[0102] Dropout is applied after each convolutional block:
[0103] D l = Dropout(P l ,p)
[0104] Among them, p is the probability of Dropout;
[0105] Step 2.4: Average pooling and maximum pooling;
[0106] The output of the last convolutional layer is subjected to both average pooling and maximum pooling:
[0107]
[0108]
[0109] Among them, N represents the total number of elements in the feature map, C i Represents the value of the i-th element in the feature map;
[0110] Step 2.5: Fully connected layer;
[0111] The pooled features are input into the fully connected layer:
[0112] F1=σ(W1·[MeanPool;MaxPool]+b1)
[0113] F2=σ(W2·F1+b2)
[0114] Among them, W1 and W2 represent the weight matrices of the fully connected layer, b1 and b2 represent the corresponding bias terms, and σ represents the activation function;
[0115] After the convolutional layer transformation, the feature representation of the output time series has a dimension of T×512, where T is the number of time steps of the feature vector; then the output feature is transformed into a high-dimensional feature through projection, and the dimension after transformation is 1024. The transformation formula is as follows:
[0116] H=f project (F2) = FW p +b p
[0117] Among them, f project represents the projection function, W p and b p Represent the weight matrix and bias term of the projection layer respectively;
[0118] Step 3: Transformer encoding layer;
[0119] The Transformer encoder consists of 24 Transformer layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network, with a dimension of 1024;
[0120] Step 3.1: Multi-head self-attention mechanism; the input of the first layer is H, and for the input of the lth layer H (L-1) , as follows:
[0121] Step 3.1.1: Linear transformation:
[0122] Q=H (l-1) W Q ,K=H (l-1) W K ,V=H (l-1) W V
[0123] Among them, W Q , W K , W v The dimensions of W are 1024×1024; Q , W K and W v are the weight matrices of query, key, and value, respectively. Q, K, and V are the query, key, and value matrices obtained through linear transformation;
[0124] Step 3.1.2: Calculate the self-attention score:
[0125]
[0126] in, h is the number of heads;
[0127] Step 3.1.3: Multi-head attention:
[0128] MultiHead(Q,K,V)=Concat(head1,…,head h )W O
[0129] Among them, the output of each attention head is head i =Attention(Q i ,K i ,V i );W O It is the weight matrix used for linear transformation after splicing, with a dimension of 1024×1024;
[0130] Step 3.2: Feedforward neural network;
[0131] Step 3.2.1: First layer linear transformation and activation function:
[0132] FFN1(x)=ReLU(xW1+b1)
[0133] Among them, the dimension of W1 is 1024×4096, b1 is the bias;
[0134] Step 3.2.2: Second layer linear transformation:
[0135] FFN2(x)=ReLU(xW1+b1)W2+b2
[0136] Among them, the dimension of W2 is 4096×1024, and b2 is the bias;
[0137] Step 3.3: Each sub-layer is followed by layer normalization and residual connection:
[0138] Step 3.3.1: Self-attention sublayer:
[0139] H (l) =LayerNorm(H (l-1) +MultiHead(Q,K,V)
[0140] Step 3.3.2: Feedforward Neural Network Sublayer:
[0141] O=LayerNorm(H (l) +FFN(H (l) ))
[0142] After 24 layers of Transformer, the final output O is obtained;
[0143] Step 4: Pooling operation:
[0144] Apply a pooling operation to the output O of the last layer of the Transformer encoder to obtain a fixed-size output vector:
[0145] p=f pool (O)
[0146] Step 5: Output:
[0147] The pooled feature vector p passes through one or more fully connected layers and then through an output layer to predict the sentiment dimension score:
[0148] y=f output (P)
[0149] Among them, y is the predicted three-dimensional vector, including the scores of arousal, dominance and emotional value:
[0150]
[0151] y1=f arousal (P)
[0152] y2=f dominance (P)
[0153] y3=f valence (P)
[0154] Among them, arousal, dominance and valence represent arousal, dominance and affectivity respectively, which are three commonly used dimensions to describe emotional states.
[0155] Furthermore, the image processing performed by the image processing module and the emotion estimator based on finite element analysis and partial least squares regression of geometric features are specifically as follows:
[0156] Step 1: Get image key points;
[0157] Step 1.1: Image preprocessing;
[0158] After the camera captures the image, it is converted into a grayscale image using the "cvtColor" function in the OpenCV library;
[0159] Step 1.2: key point detection;
[0160] Use DLIB's forward face detector get_frontal_face_detector() to perform face detection and extract the outer rectangular frame of the face; use the 68-point face feature detector trained by the DLIB library to extract the facial contour features and obtain the coordinates of the 49 facial coordinate points after screening;
[0161] Step 2: Face positive orientation;
[0162] Step 2.1: Scaling, translation and rotation:
[0163] First, scale, translate, and rotate the coordinate points so that all coordinate points are on the same frontal expression:
[0164]
[0165] in: is the position point or feature vector of the kth sample in the original coordinate system, where o represents the original coordinate system; Indicates all The mean vector of is the mean of all sample points in the original coordinate system and is used for translation processing; R k is the rotation matrix, which is used to adjust the orientation of the coordinate point; k: represents the index of the sample; o: the identifier in the original coordinate system, which is used to indicate the point in the original system; M: represents the coordinate system after scaling, translation and rotation;
[0166] Step 2.2: Construct a transposed matrix A containing the coordinate points of all non-positive expressions and a true value matrix Y containing the corresponding coordinate points of positive expressions, and fill the normalized coordinates into the matrix;
[0167] Step 2.3: Objective function:
[0168] argmin X |Y-AX|2+λ|X|2
[0169] Among them, λ is the regularization parameter; X represents the parameter matrix to be optimized;
[0170] Step 2.4: Positive mapping of the predicted value of X:
[0171]
[0172] Where I represents the identity matrix, whose dimensions are Same; T means transpose;
[0173] Step 3: Partial least squares regression;
[0174] Step 3.1: Input and Output:
[0175] X1: initial feature, size is N×1176, 1176-dimensional feature calculated by the pairwise distance of 49 facial landmarks;
[0176] Y1: response feature, size N×3, including outputs of three dimensions;
[0177] Step 3.2: Model representation:
[0178]
[0179]
[0180] Among them, T and U are latent variable matrices: the size is N×29, there are 29 latent variables; P and Q are loading matrices; E and F are residual matrices;
[0181] Step 3.3: Objective function;
[0182] The goal is to maximize the covariance between the response feature and the predictor feature:
[0183]
[0184] Among them, t and u are the column vectors of T and U respectively; X1 represents the initial feature matrix, which is used to predict the feature data; Y1 represents the response feature matrix, which contains the value of the target variable; w and v are two direction vectors, which are used to project data in the feature space and response space respectively; X1w represents the product of matrix X1 and vector w, and the result is the projection along the w direction in the feature space; Y1v represents the product of matrix Y1 and vector v, and the result is the projection along the v direction in the response space;
[0185] Step 3.4: Solve the regression coefficient matrix B as follows to map the feature matrix X1 to the predicted response:
[0186]
[0187] Step 3.5: Predicted response matrix:
[0188] The final predicted response matrix is calculated by the following formula:
[0189]
[0190] in, Represents the predicted response feature matrix, including scores for arousal, emotion, and intensity:
[0191]
[0192] Furthermore, the specific steps of brain wave data processing and emotion index scoring are as follows:
[0193] Step 1: Calculate the mean:
[0194] The input is the value x of the i-th data point i , the total number of data points n;
[0195] The output is the mean μ of the data points;
[0196] Step 2: Calculate the standard deviation:
[0197]
[0198] The input is the mean μ of the data point, and the output is the standard deviation σ of the data point;
[0199] Step 3: Construct the probability density function of the normal distribution:
[0200]
[0201] The input is the value x of the current data point, the mean μ of the data point, and the standard deviation σ of the data point;
[0202] The output is the probability density value f(x) of the data point (x) under normal distribution;
[0203] Step 4: Calculate the integral of the data of the current second over the distribution:
[0204]
[0205] The input is the value x_current of the data point at the current second and the probability density value f(x) of the data point (x) under the normal distribution;
[0206] The output is the cumulative distribution function value P(x current );
[0207] Step 5: Normalize and convert to score:
[0208] Score = 100 × P (x current )
[0209] The input is the cumulative distribution function value P(x current );
[0210] The output is the normalized score Score of the current data point;
[0211] Step 6: The overall process formula is:
[0212]
[0213] The beneficial effects of adopting the above technical scheme are as follows: the multimodal real-time mental health analysis system based on artificial intelligence method provided by the present invention proposes a multimodal emotion analysis algorithm based on brain waves, images and audio signals, which can comprehensively evaluate the user's emotional state by combining multiple biological signals and behavioral data, thereby improving the accuracy of emotion recognition and the robustness of the algorithm; it overcomes the shortcomings of the prior art in single-modal emotion analysis, realizes the synchronous analysis of multimodal data, and improves the accuracy of emotion analysis results; through deep learning models and data analysis technology, audio, image and brain wave signals are processed to generate emotion indicators, and through multimodal fusion of emotional expression, a more intuitive display of emotion changes is provided; it is designed The cloud-edge collaborative framework deploys deep learning models and data analysis on the cloud server, and integrates the audio and video EEG collection module and the user interaction module on the local computer, realizing efficient data transmission and processing, making the algorithm more scalable and robust; an efficient data transmission and processing mechanism is established between the EEG biofeedback instrument, the local computer and the cloud server, and real-time data transmission and analysis are realized through USB serial port communication and TCP / IP communication, ensuring the timeliness and accuracy of emotion analysis; the multimodal emotion analysis algorithm of the present invention provides comprehensive emotion indicators, including positive emotions, negative emotions and physiological emotions, and can perform detailed analysis and evaluation of the user's emotional state, thereby enhancing the user's cognition and understanding of the emotional state. BRIEF DESCRIPTION OF THE DRAWINGS
[0214] Figure 1 An architecture diagram of a multimodal real-time mental health analysis system based on an artificial intelligence method provided by an embodiment of the present invention;
[0215] Figure 2 A flow chart of a multimodal real-time mental health analysis algorithm provided by an embodiment of the present invention;
[0216] Figure 3 A schematic diagram of the main interface of the multimodal algorithm provided by an embodiment of the present invention;
[0217] Figure 4A schematic diagram of a brain wave modality interface provided by an embodiment of the present invention;
[0218] Figure 5 A graph showing the results of brain wave emotion analysis provided by an embodiment of the present invention;
[0219] Figure 6 A schematic diagram of an image-audio modal interface provided by an embodiment of the present invention;
[0220] Figure 7 An image analysis result diagram displayed by the image display module provided by an embodiment of the present invention;
[0221] Figure 8 An audio analysis result diagram displayed by the audio analysis module provided in an embodiment of the present invention;
[0222] Fig. 9 A multimodal result display interface and a positive emotion radar chart provided in an embodiment of the present invention;
[0223] Fig.10 A negative emotion radar chart provided by an embodiment of the present invention;
[0224] Fig.11 A physiological emotion radar chart provided by an embodiment of the present invention;
[0225] Fig.12 A schematic diagram of a pop-up window for user setting parameters provided in an embodiment of the present invention.
[0226] In the figure: 1. Image display module; 2. Line chart module; 3. Image analysis module; 4. Audio analysis module; 5. User interaction button. DETAILED DESCRIPTION
[0227] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0228] A multimodal real-time mental health analysis system based on artificial intelligence methods, such as Figure 1 As shown in the figure, it includes an EEG biofeedback instrument, a cloud server and a computer. The EEG biofeedback instrument communicates with the computer using a USB serial port, and the computer and the cloud server communicate using TCP / IP. The complete process of the system for multi-modal real-time mental health analysis is as follows: Figure 2 As shown, the system welcome and function selection interface is as follows Figure 3 shown.
[0229] A brain wave acquisition module and a signal analysis module are deployed in the EEG biofeedback instrument. The brain wave acquisition module receives the original brain wave signal, and the signal analysis module analyzes the original brain wave signal and transmits the signal to the computer.
[0230] The process of collecting brain waves by the EEG acquisition module is as follows: receiving the original brain wave data, recorded as Rawdata, and converting it into five brain wave signals: Delta wave, Theta wave, Alpha wave, Beta wave, and Gamma wave, a total of five unsigned numerical values, using a horizontal dynamic comparison method. When the packet loss rate is less than 10% and the checksum is passed, the data is retained and passed to the signal analysis module.
[0231] The signal analysis module performs signal analysis and transmission as follows: the five brain wave signals are used to calculate the concentration and relaxation, and are mapped to a value between 0 and 100. The seven brain waves, Attention, Meditation, Delta, Theta, Alpha, Beta, and Gamma, are transmitted to the computer via the USB serial port communication. The real-time data will be displayed in two visual forms, such as Figure 4 After the recording is completed, the overall score will be displayed, such as Figure 5 shown.
[0232] A computer program for deep learning models and data analysis is deployed on a cloud server, including an interactive data storage module, an interactive parameter storage module, an audio processing module, an image processing module, an EEG processing module, a multimodal emotion analysis module, a result feedback module, and a result report display module. The functions of each module are:
[0233] The cloud server receives the audio and video brain wave collection module in the computer (such as Figure 6 The audio stream, data stream and brain wave signal stream returned by the interactive data storage module will be stored in the cloud database in the order of timestamps. Figure 6 The display module in the embodiment includes an image display module 1, a line graph module 2, an image analysis module 3, an audio analysis module 4 and a user interaction button 5. The cloud server accepts the parameters input by the user during the interaction process, such as Fig.12 As shown in the figure: "Do you want to save this data locally?", the interactive parameter storage module will store it in the cloud database in the order of timestamps; the audio processing module performs emotion analysis based on the audio data; the image processing module performs emotion analysis based on the image data; the brain wave processing module performs emotion analysis based on the brain wave data; the multimodal emotion analysis module is used to receive the emotion indicators of the brain wave-image-audio modality in real time and perform multimodal emotion analysis; the result feedback module is used to return the emotion analysis results to the computer in real time for the user to view on the computer, such as Figure 7 , Figure 8 As shown; the result report display module is used to generate the emotion analysis report after the recording is completed and send it back to the computer, such as Fig. 9 , Fig.10and Fig.11 shown.
[0234] The process of storing interactive data in the interactive data storage module is as follows: the computer transmits the audio signal collected by the microphone (recorded as Raw_A), the image data collected by the camera (recorded as Raw_V), and the brain wave signal collected by the brain wave collection module (recorded as Raw_B) to the cloud server in the form of real-time data stream through TCP / IP. The audio and video brain wave stream data contains timestamp information to ensure data integrity and timing accuracy. At the same time, the interactive data storage module stores it in the database in the order of timestamps.
[0235] The process of storing interaction parameters in the interaction parameter storage module is as follows: the parameters input by the user will be stored in the local browser cache through the localStorage method. When the cloud server starts, the parameters input by the user will be called from the local browser cache. After the call is completed, the interaction data will be deleted to ensure the protection of user privacy.
[0236] The process of audio processing module processing audio data and emotion analysis is as follows: the audio signal collected by the microphone (referred to as Raw_A) will be passed to the audio processing module for detailed analysis. First, the audio processing module converts the audio signal Raw_A received by the cloud server into mono data, resamples it to 16000Hz, and then divides it into segments of fixed length; the pre-processed audio data is used for emotion recognition through the trained audio prediction model. The specific process is as follows:
[0237] In audio emotion recognition based mainly on the Transformer architecture, the input audio signal x(t) is first processed as a feature encoder through the wav2vec pre-trained model to obtain a feature vector sequence X = f_encoder(x(t)); the feature vector sequence X is projected to a higher dimensional space to match the input dimension of the Transformer model; after Transformer encoding and pooling operations, three emotion indicators, audio_arousal, audio_dominance, and audio_valence, are finally obtained. The value range of these emotion indicators is [0,1]; for further analysis, these emotion indicators are further mapped to the range of [-3,3]; the default analysis cycle is once every 2 seconds, but it can be modified according to user needs. Finally, the emotion indicator values will be stored in the cloud database according to the timestamp. The specific process of CNN14 model training in audio processing and audio prediction models is as follows:
[0238] Step 1: Generate a log-Mel spectrogram.
[0239] Step 1.1: Frame.
[0240] The audio signal x(t) is divided into several time frames, and each frame is weighted using the window function w(k). The purpose of the window function is to reduce spectrum leakage and smooth the edges, as shown in the following formula:
[0241] x f [n]=x[n·H+k]·w[k],k=0,1,…,L-1
[0242] Where H is the step size, 10 milliseconds; L is the window size, 32 milliseconds, k represents the sampling point index within the window function w[k], from 0 to L-1, x f [n] represents the information of the nth frame after being processed by the window function, and f is the index of the frame, which is used to distinguish different frames.
[0243] Step 1.2: Perform a short-time Fourier transform.
[0244] Perform the Fourier transform on each frame as follows:
[0245]
[0246] Among them, Π represents pi, which is approximately equal to 3.14159, and X f [m,k] is the kth frequency component of the mth frame.
[0247] Step 1.3: Convert to Mel frequency scale.
[0248] Use the Mel filter to convert the spectrum to a Mel spectrum:
[0249]
[0250] Wherein, f represents the index of the Mel band, usually the number of Mel bands is 64, H[f,k] is the filter coefficient of the Mel filter bank, which represents the response of the f-th Mel band to the k-th frequency component.
[0251] Step 1.4: Take the logarithm.
[0252] Take the logarithm of the Mel spectrum to get the log-Mel spectrum:
[0253] Log-Mel(m,f)=log(M[m,f])
[0254] Step 2: The log-Mel spectrogram obtained in step 1 is sent to the CNN14 model as a feature map. The CNN14 model includes 6 convolution blocks, each of which has two layers. Each layer has a maximum pooling operation, where the convolution kernel size is 3*3, the step size is 1, and the maximum pooling layer step size is 2. The details are as follows:
[0255] Step 2.1: Convolutional layer.
[0256] The output of each convolutional layer is represented as:
[0257] C l =σ(W l *X l +b l )
[0258] Among them, W l is the convolution kernel, * indicates the convolution operation, b l is the bias, σ is the activation function, X l is the characteristic log-mel spectrogram of the input.
[0259] Step 2.2: Max pooling layer.
[0260] The output of each pooling layer is represented as:
[0261] P l =MaxPool(C l )
[0262] Step 2.3: Dropout layer.
[0263] Dropout is applied after each convolutional block:
[0264] D l = Dropout(P l ,p)
[0265] Among them, p is the probability of Dropout.
[0266] Step 2.4: Average pooling and maximum pooling;
[0267] The output of the last convolutional layer is subjected to both average pooling and maximum pooling:
[0268]
[0269]
[0270] Among them, N represents the total number of elements in the feature map, C i Represents the value of the i-th element in the feature map.
[0271] Step 2.5: Fully connected layer;
[0272] The pooled features are input into the fully connected layer:
[0273] F1=σ(W1·[MeanPool;MaxPool]+b1)
[0274] F2=σ(W2·F1+b2)
[0275] Among them, W1 and W2 represent the weight matrices of the fully connected layer, b1 and b2 represent the corresponding bias terms, and σ represents the activation function;
[0276] After the convolutional layer transformation, the feature representation of the output time series has a dimension of T×512, where T is the number of time steps of the feature vector; then the output feature is transformed into a high-dimensional feature through projection, and the dimension after transformation is 1024. The transformation formula is as follows:
[0277] H=f project (F2) = FW p +b p
[0278] Among them, f project represents the projection function, W p and b p Represent the weight matrix and bias term of the projection layer respectively;
[0279] Step 3: Transformer encoding layer;
[0280] The Transformer encoder consists of 24 Transformer layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network, with a dimension of 1024.
[0281] Step 3.1: Multi-Head Self-Attention Mechanism. The input of the first layer is H. For the input of the lth layer H (L-1) , as follows:
[0282] Step 3.1.1: Linear transformation:
[0283] Q=H (l-1) W Q ,K=H (l-1) W K ,V=H (l-1) W V
[0284] Among them, W Q , W K , W v The dimensions of W are 1024×1024; Q , W K and W v are the weight matrices of query, key and value respectively, and Q, K and V are the query, key and value matrices obtained by linear transformation.
[0285] Step 3.1.2: Calculate the self-attention score:
[0286]
[0287] in, h is the number of heads.
[0288] Step 3.1.3: Multi-head attention:
[0289] MultiHead(Q,K,V)=Concat(head1,…,head h )W O
[0290] Among them, the output of each attention head is head i =Attention(Q i ,K i ,V i );W O It is the weight matrix used for linear transformation after concatenation, with a dimension of 1024×1024.
[0291] Step 3.2: Feedforward Neural Network.
[0292] Step 3.2.1: First layer linear transformation and activation function:
[0293] FFN1(x)=ReLU(xW1+b1)
[0294] Among them, the dimension of W1 is 1024×4096, and b1 is the bias.
[0295] Step 3.2.2: Second layer linear transformation:
[0296] FFN2(x)=ReLU(xW1+b1)W2+b2
[0297] Among them, the dimension of W2 is 4096×1024, and b2 is the bias.
[0298] Step 3.3: Each sub-layer is followed by layer normalization and residual connection.
[0299] Step 3.3.1: Self-attention sublayer:
[0300] H (l) =LayerNorm(H (l-1) +MultiHead(Q,K,V)
[0301] Step 3.3.2: Feedforward Neural Network Sublayer:
[0302] O=LayerNorm(H (l) +FFN(H(l) ))
[0303] After 24 Transformer layers, the final output O is obtained.
[0304] Step 4: Pooling operation:
[0305] Apply a pooling operation to the output O of the last layer of the Transformer encoder to obtain a fixed-size output vector:
[0306] p=f pool (O)
[0307] Step 5: Output:
[0308] The pooled feature vector p passes through one or more fully connected layers (fine-tuning process) and then through an output layer to predict the sentiment dimension score:
[0309] y=f output (P)
[0310] Among them, y is the predicted three-dimensional vector, including the scores of arousal, dominance and emotional value:
[0311]
[0312] y1=f arousal (P)
[0313] y2=f dominance (P)
[0314] y3=f valence (P)
[0315] Among them, arousal, dominance and valence represent arousal, dominance and affectivity respectively, which are three commonly used dimensions to describe emotional states.
[0316] Arousal is defined as: describing the activity or intensity of an emotion, indicating the excitement of an emotion. It measures the degree of excitement or alertness of an individual in an emotional state; the range is usually from low to high, such as from calm to excited.
[0317] Dominance is defined as: describing the sense of control or dominance of an individual in an emotional state. It reflects the degree of control or dominance of an individual in an emotional state; the range is usually from low to high, such as from helpless to strong control;
[0318] Valence is defined as: describing the positive or negative nature of emotions, indicating how pleasant or unpleasant an emotion is; it measures the positive or negative nature of an emotion; the range is usually from negative to positive, such as from sadness to happiness.
[0319] The process of image processing module processing image data and emotion analysis is as follows: the video signal (denoted as Raw_V) collected by the camera will be passed to the image processing module for detailed analysis. First, the image processing module sends the image signal Raw_V received by the cloud server to the pre-trained DLIB face detection model to detect whether there is a face in the picture and select the face with the largest size; next, the selected face is sent to the pre-trained DLIB facial key point detection model for key point detection, thereby generating an object containing the detected facial key points, denoted as landmarks_object; then, the object landmarks_object is passed to the pre-trained emotion estimator, which uses partial least squares (PLS) regression analysis to construct a two-dimensional emotion space prediction model and uses the Neutral to Apex morphing method to calculate the emotion intensity. Through this process, three key facial emotion indicators can be obtained: facial arousal, facial intensity, and facial pleasure. Among them, the value range of face intensity is [0,1], and the value range of the other two indicators is [-1,1]. In order to facilitate further analysis, the three facial emotion indicators are further mapped to the range of [-3,3]. By default, the output cycle is once per frame, and the three facial emotion indicators are stored in the cloud database according to the timestamp. The image processing and emotion estimator based on finite element analysis and partial least squares regression based on geometric features are as follows:
[0320] Step 1: Get image key points.
[0321] Step 1.1: Image preprocessing;
[0322] After the camera captures the image, it is converted into a grayscale image using the “cvtColor” function in the OpenCV library.
[0323] Step 1.2: key point detection;
[0324] Use DLIB's forward face detector get_frontal_face_detector() to perform face detection and extract the outer rectangular frame of the face. Use the 68-point face feature detector trained by the DLIB library to extract the facial contour features and obtain the coordinates of the 49 facial coordinate points after screening.
[0325] Step 2: Face positive.
[0326] Step 2.1: Scaling, translation and rotation:
[0327] First, scale, translate, and rotate the coordinate points so that all coordinate points are on the same frontal expression:
[0328]
[0329] in: is the position point or feature vector of the kth sample in the original coordinate system, where o represents the original coordinate system; Indicates all The mean vector of R, which is the mean of all sample points in the original coordinate system and is used for translation processing; k is the rotation matrix, which is used to adjust the orientation of the coordinate point; k: represents the index of the sample; o: is the identifier in the original coordinate system, which is used to indicate the point in the original system; M: represents the coordinate system after scaling, translation and rotation.
[0330] Step 2.2: Construct a transposed matrix A containing the coordinate points of all non-positive expressions and a true value matrix Y containing the corresponding positive expression coordinate points, and fill the normalized coordinates into the matrix.
[0331] Step 2.3: Objective function:
[0332] argmin X |Y-AX|2+λ|X|2
[0333] Among them, λ is the regularization parameter; X represents the parameter matrix to be optimized.
[0334] Step 2.4: Positive mapping of the predicted value of X:
[0335]
[0336] Where I represents the identity matrix, whose dimensions are Same; T means transpose.
[0337] Step 3: Partial least squares regression.
[0338] Step 3.1: Input and Output:
[0339] Input X1: initial features, size N × 1176, 1176-dimensional features calculated by the pairwise distance of 49 facial landmarks;
[0340] Output Y1: response feature, size N×3, including outputs of three dimensions.
[0341] Step 3.2: Model representation:
[0342]
[0343]
[0344] Among them, T and U are latent variable matrices: the size is N×29, there are 29 latent variables; P and Q are loading matrices; E and F are residual matrices.
[0345] Step 3.3: Objective function;
[0346] The goal is to maximize the covariance between the response feature and the predictor feature:
[0347]
[0348] Among them, t and u are the column vectors of T and U respectively; X1 represents the initial feature matrix, which is used to predict the feature data; Y1 represents the response feature matrix, which contains the value of the target variable; w and v are two direction vectors, which are used to project data in the feature space and response space respectively; X1w represents the product of matrix X1 and vector w, and the result is the projection along the w direction in the feature space; Y1v represents the product of matrix Y1 and vector v, and the result is the projection along the v direction in the response space.
[0349] Step 3.4: Solve the regression coefficient matrix B as follows to map the feature matrix X1 to the predicted response Y1:
[0350]
[0351] Step 3.5: Predicted response feature matrix:
[0352] Final predicted response feature matrix Calculated by the following formula:
[0353]
[0354] in, The predicted response feature matrix, including scores for arousal, emotion, and intensity:
[0355]
[0356] The process of the brainwave processing module processing brainwave data and emotion analysis is as follows: the brainwave signal collected by the EEG biofeedback instrument (recorded as Raw_B) will be passed to the brainwave processing module for detailed analysis. First, the brainwave processing module processes the brainwave signal Raw_B received by the cloud server in segments according to a fixed period; for the data in each period, the period mean of Delta wave, Theta wave, Alpha wave, Beta wave and Gamma wave is calculated separately; on this basis, the standard deviation of these bands is calculated, and the probability density function of the normal distribution is constructed to calculate the integral value of the data of the current second on the normal distribution; then, the integral value is normalized and converted into a standard score; according to these standard scores, combined with the preset weights, seven emotional indicators are calculated: light, dominance, alert, clear, nervous, attention and meditation. The calculation of each of the seven emotional indicators is based on the comprehensive weight of signals in different bands to ensure the accuracy and reliability of emotional analysis. The specific steps are as follows:
[0357] Step 1: Calculate the mean:
[0358] The input is the value x of the i-th data point i , the total number of data points n. The output is the mean μ of the data points.
[0359] Step 2: Calculate the standard deviation:
[0360]
[0361] The input is the mean μ of the data points, and the output is the standard deviation σ of the data points.
[0362] Step 3: Construct the probability density function (PDF) of the normal distribution:
[0363]
[0364] The input is the value x of the current data point, the mean μ of the data point, and the standard deviation σ of the data point. The output is the probability density value f(x) of the data point (x) under the normal distribution.
[0365] Step 4: Calculate the integral of the data of the current second over the distribution:
[0366]
[0367] The input is the value x_current of the data point at the current second and the probability density value f(x) of the data point (x) under normal distribution. The output is the cumulative distribution function value P(x) of the current data point under normal distribution.current ).
[0368] Step 5: Normalize and convert to score:
[0369] Score = 100 × P (x current )
[0370] The input is the cumulative distribution function value P(x current ). The output is the normalized score Score of the current data point.
[0371] Step 6: The overall process formula is:
[0372]
[0373] The process of analyzing multimodal emotions by the multimodal emotion analysis module is as follows: first, the values of six indicators, namely audio_arousal, audio_dominance, audio_valence, face_arousal, face_intensity, and face_valence, are obtained from the cloud database, and mapped to the PAD emotion table respectively to obtain the predicted emotions of the current face and audio; the multimodal emotion analysis module visualizes audio_arousal, audio_dominance, and audio_valence as three-dimensional stereograms, and visualizes face_arousal and face_valence as two-dimensional plane graphs, and stores them in the cloud database together with the analysis results.
[0374] The process of the result feedback module returning the emotion results is as follows: after the cloud server receives the data and completes the analysis, it obtains the real-time analysis results from the database to form a dictionary, packages it in json format, and returns it to the computer via TCP / IP. Users can view the currently predicted facial audio and brain wave data, PAD emotion type, audio three-dimensional stereogram and facial two-dimensional plane map.
[0375] The process of generating an emotion analysis report in the result report display module is as follows: after the user finishes recording, the cloud server assigns different weights to the PAD emotion index obtained from the audio facial analysis and the emotion index obtained from the brain wave analysis to obtain an analysis report on positive emotions, negative emotions, and physiological emotions.
[0376] Positive emotion indicators include self-confidence, self-regulation, vitality, and physical and mental balance.
[0377] Self-confidence is a probability indicator of the emotion being in the medium-intensity positive interval of the emotion space. It is calculated by weighting the probability that the three dimensions of emotion in the image and audio signals are all greater than 0.
[0378] Self-regulation ability refers to the speed of changing from high intensity to low intensity. The calculation method is: calculate the sum of the absolute changes between all adjacent elements, divide it by the number of changes between elements, and get the average change rate; evaluate self-regulation ability based on the average change rate, using intensity for images and dominance for audio.
[0379] Vitality is calculated by mapping the arousal of the two modalities from [-3, 3] to [0, 100] and then weighting them, and the whole process is calculated using the average sentiment.
[0380] Physical and mental balance is comprehensively assessed through tension, anxiety and stress. The higher the anxiety, the lower the physical and mental balance.
[0381] Negative emotion indicators include aggression, tension, expressed stress, and suspiciousness.
[0382] Aggression was calculated by combining the two dimensions of hostility and uncontrollability, using only audio data; the percentage of the two emotions was weighted as a distance metric to obtain the final result.
[0383] The tension index is evaluated comprehensively by combining the results of brain waves, images, and audio. The distance measurement percentage is used, and the audio, video, and brain wave data are weighted to obtain the final percentage. Among them, the tension in the audio is replaced by anxiety, and the tension value is directly used for the brain wave data.
[0384] The expression pressure is evaluated by image results, directly mapping the intensity from [-3, 3] to [0, 100], and calculating it using the average emotion of the whole process.
[0385] Suspicion is calculated through the three dimensions of vigilance, tension, and alertness. First, the distances of image, audio, and brain wave data are calculated in the custom PAD emotional space to obtain percentages, and then the vigilance, tension, and alertness of the three modalities are weighted to obtain the final suspicion. Among them, the tension of the audio is obtained by converting anxiety, and the alertness of the brain wave is obtained by converting tension.
[0386] Physiological and emotional indicators include attention, relaxation, fatigue index, anxiety index, stress index, depression index, alertness, control, clarity, happiness, depression, and energy.
[0387] The concentration and relaxation levels are obtained through Rawdata analysis by the EEG biofeedback instrument.
[0388] The fatigue index is evaluated through the arousal of images, audio, and brain waves. The arousal of the three is directly mapped to [0,100], and then weighted in equal proportion to obtain the final non-fatigue percentage. The fatigue index percentage is then obtained by subtracting the non-fatigue percentage from 100%.
[0389] The anxiety index is calculated by distance measurement. According to the position of anxiety in the PAD emotional space, the distance between the current emotion and anxiety is calculated.
[0390] The stress index is calculated based on arousal, valence, and dominance, and is assessed based on the PAD emotional space.
[0391] The depression index was calculated according to arousal, valence, and dominance and evaluated according to the PAD affective space.
[0392] Be careful and add the weight of 0.9 given to the proportion of beta waves and the weight of 0.1 given to the proportion of gamma waves.
[0393] The control index combines the results of audio and brainwave for comprehensive evaluation. The distance metric percentage is used, and the audio and brainwave data are weighted to get the final percentage.
[0394] The wakefulness index combines the audio and brainwave results for a comprehensive assessment. The distance metric percentage is used, and the audio and brainwave data are weighted to get the final percentage, with the brainwave data accounting for the main weight.
[0395] Happiness is obtained by mapping arousal to [0,100].
[0396] Depression is represented by weighting dominance and happiness. High dominance and low happiness represent depression, calculated by subtracting the weighted percentages of happiness and dominance from 100%.
[0397] Energy is usually calculated from arousal and intensity. The energy is obtained by multiplying the arousal and intensity at each moment, where arousal comes from the weighted sum of the three modalities and intensity comes from the image modality.
[0398] In the calculation process of the multimodal sentiment index, various calculation formulas such as mapping, weighting, and PAD space mapping are used, as described below.
[0399] The mapping process is expressed as follows:
[0400] X i =m·L i +c
[0401] Among them, L i is the original probability value, X i is the mapped value, m is the mapped multiplier, and c is the mapped constant.
[0402] The weighting process is expressed as follows:
[0403]
[0404] Among them, X i is the i-th mean value, w i is the weight corresponding to the i-th average value, and n represents the total number of values involved in the calculation.
[0405] The mapping process of PAD emotional space is as follows:
[0406] Calculate the maximum distance D between each extreme point in the PAD emotional space and the emotional coordinate T max :
[0407] D max =max{D1,D2,…,D n}
[0408] Calculate the distance D between the current emotion and the emotion coordinate T 当前情绪 :
[0409]
[0410] Where T = [T x , T y , T z ] is the coordinate of a certain emotion position in the emotion space, and E=[x, y, z] is the PAD emotion space coordinate of the current emotion.
[0411] Calculate the percentage of emotion as follows:
[0412]
[0413] The calculation of the energy index is as follows:
[0414] For each emotion pair (a i ,i i ), calculate the energy by multiplying its absolute value as follows:
[0415] Energy Efficiency i =|a i |×|i i |
[0416] If the energy list is empty, returns 0.
[0417] Calculate the maximum energy value max_energy in the energy list to normalize the energy value to the range of [0,100] as follows:
[0418]
[0419] Sum the normalized energy lists and divide by the length of the energy lists to get the average energy value, as follows:
[0420]
[0421] Among them, len(normalized\_energy) represents the length of the energy list.
[0422] The audio and video brainwave collection module and the user interaction module are integrated in the computer, which are used to collect audio and video brainwave data through computer components and to interact with algorithms through the computer interface. The audio and video brainwave collection module collects audio data Raw_A through the microphone, image data Raw_V through the camera, and brainwave data Raw_B through the brainwave collection module. In the user interaction module, the user uses the mouse and the computer screen to operate the algorithm, start and stop recording, modify parameters, and view results.
[0423] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.
Claims
1. A multimodal real-time mental health analysis system based on artificial intelligence methods, Features: It includes an EEG biofeedback device, a cloud server and a computer; The EEG biofeedback device communicates with the computer using a USB serial port, and the computer and the cloud server communicate using TCP / IP; A brain wave acquisition module and a signal analysis module are deployed in the EEG biofeedback instrument. The brain wave acquisition module receives the original brain wave signal, and the signal analysis module analyzes the original brain wave signal and transmits the signal to the computer. A computer program for deep learning model and data analysis is deployed on a cloud server, including an interactive data storage module, an interactive parameter storage module, an audio processing module, an image processing module, an EEG processing module, a multimodal emotion analysis module, a result feedback module and a result report display module. The functions of each module are as follows: the cloud server receives the audio stream, data stream and EEG signal stream returned by the audio and video EEG collection module in the computer, and the interactive data storage module stores them in the cloud database in the order of timestamps; The cloud server accepts the parameters input by the user during the interaction process, and the interaction parameter storage module stores them in the cloud database in the order of timestamps; The audio processing module performs emotion analysis based on audio data; the image processing module performs emotion analysis based on image data; the brain wave processing module performs emotion analysis based on brain wave data; the multimodal emotion analysis module is used to receive emotion indicators of brain wave-image-audio modality in real time and perform multimodal emotion analysis; the result feedback module is used to transmit the emotion analysis results back to the computer in real time for users to view on the computer; the result report display module is used to generate the emotion analysis report after the recording is completed and transmit it back to the computer; An audio and video brainwave collection module and a user interaction module are integrated in the computer, which are used to collect audio and video brainwave data through computer components and to perform algorithm interaction through a computer interface, specifically: collecting audio data Raw_A through a microphone, collecting image data Raw_V through a camera, and collecting brainwave data Raw_B through a brainwave collection module; users use a mouse and a computer screen to perform algorithm operations, start and stop recording, modify parameters, and view results.
2. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 1, Features: The process of the EEG acquisition module collecting EEG is as follows: receiving raw EEG data, recorded as Rawdata, and converting it into five EEG signals: Delta wave, Theta wave, Alpha wave, Beta wave, Gamma wave, a total of five unsigned values, using a horizontal dynamic comparison method, retaining the data and passing it to the signal analysis module when the packet loss rate is less than 10% and passes the checksum; The signal analysis module performs signal analysis and transmission as follows: the concentration and the relaxation are calculated through the five brain wave signals and mapped to between 0 and 100, and the seven brain waves, Attention, Meditation, Delta, Theta, Alpha, Beta and Gamma, are transmitted to the computer through the USB serial port communication.
3. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 2, Features: The process of each module of the cloud server completing its function is as follows: The process of storing interactive data in the interactive data storage module is as follows: the computer transmits the audio signal collected by the microphone, the image data collected by the camera, and the brain wave signal collected by the brain wave collection module to the cloud server via TCP / IP in the form of real-time data stream, and the interactive data storage module stores the data in the cloud database in the order of timestamps; The process of storing interaction parameters in the interaction parameter storage module is as follows: the parameters input by the user will be stored in the local browser cache through the localStorage method. When the cloud server starts, the parameters input by the user will be called from the local browser cache. After the call is completed, the interaction data will be deleted to ensure the protection of user privacy. The process of audio processing module processing audio data and emotion analysis is as follows: First, the audio processing module converts the audio signal Raw_A received by the cloud server into mono data, resamples it to 16000Hz, and then divides it into segments of fixed length; the pre-processed audio data is used for emotion recognition through the trained audio prediction model. The specific process is as follows: The input audio signal x(t) is first processed as a feature encoder through the wav2vec pre-trained model to obtain a feature vector sequence X = f_encoder(x(t)); the feature vector sequence X is projected to a higher-dimensional space to match the input dimension of the Transformer model; after Transformer encoding and pooling operations, three emotion indicators, audio_arousal, audio_dominance, and audio_valence, are finally obtained. The value range of these emotion indicators is [0,1]; these emotion indicators are further mapped to the range of [-3,3]; the analysis cycle is initially set to 2, and the user can modify it according to actual needs during use; finally, the emotion indicator values will be stored in the cloud database according to the timestamp; The process of image processing module processing image data and emotion analysis is as follows: first, the image processing module sends the image signal Raw_V received by the cloud server to the pre-trained DLIB face detection model to detect whether there is a face in the picture, and selects the face with the largest size; next, the selected face is sent to the pre-trained DLIB facial key point detection model for key point detection, thereby generating an object containing the detected facial key points, recorded as landmarks_object; Subsequently, the object landmarks_object is passed to the pre-trained emotion estimator, which uses partial least squares regression analysis to build a two-dimensional emotion space prediction model and uses the Neutral to Apex morphing method to calculate the emotion intensity, and obtains three key facial emotion indicators: facial arousal, facial intensity, and facial valence. The value range of facial intensity is [0,1], and the value range of the other two indicators is [-1,1]. The three facial emotion indicators are further mapped to the range of [-3,3]. The output cycle is once per frame, and the three facial emotion indicators are stored in the cloud database according to the timestamp. The process of brainwave processing module processing brainwave data and emotion analysis is as follows: First, the brainwave processing module processes the brainwave signal Raw_B received by the cloud server in segments according to a fixed period; for the data in each period, the period mean of Delta wave, Theta wave, Alpha wave, Beta wave and Gamma wave is calculated separately; on this basis, the standard deviation of these bands is calculated, and the probability density function of the normal distribution is constructed to calculate the integral value of the data of the current second on the normal distribution; then, the integral value is normalized and converted into a standard score; based on these standard scores, combined with the preset weights, seven emotion indicators are calculated: light, dominance, alert, clear, nervous, attention and meditation; the calculation of each emotion indicator is based on the comprehensive weight of signals in different bands to ensure the accuracy and reliability of emotion analysis; The process of analyzing multimodal emotions by the multimodal emotion analysis module is as follows: first, the values of six indicators, namely audio_arousal, audio_dominance, audio_valence, face_arousal, face_intensity and face_valence, are obtained from the cloud database, and are mapped to the PAD emotion table respectively to obtain the predicted emotions of the current face and audio; the multimodal emotion analysis module visualizes audio_arousal, audio_dominance and audio_valence into three-dimensional stereograms, and visualizes face_arousal and face_valence into two-dimensional plane graphs, and stores them in the cloud database together with the analysis results; The process of the result feedback module returning the emotion results is as follows: after the cloud server receives the data and finishes the analysis, it obtains the real-time analysis results from the database to form a dictionary, packages it in json format, and returns it to the computer via TCP / IP. The user can view the currently predicted facial audio and brain wave data, PAD emotion type, audio 3D stereogram and facial 2D plane map; The process of generating an emotion analysis report in the result report display module is as follows: after the user finishes recording, the cloud server assigns different weights to the PAD emotion index obtained from the audio facial analysis and the emotion index obtained from the brain wave analysis to obtain an analysis report on positive emotions, negative emotions, and physiological emotions.
4. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 3, Features: The specific calculation method of the analysis report of positive emotions, negative emotions and physiological emotions is as follows: Positive emotion indicators include self-confidence, self-regulation, vitality, and physical and mental balance; The self-confidence is a probability index of the emotion being in a medium-intensity positive interval in the emotion space; it is calculated by weighting the probability that the three dimensions of the emotion in the image and audio signals are all greater than 0; The self-regulation ability refers to the speed of changing from high intensity to low intensity, which is calculated by calculating the sum of absolute changes between all adjacent elements and dividing it by the number of changes between elements to obtain the average change rate; the self-regulation ability is evaluated based on the average change rate, using intensity for images and dominance for audio; The vitality is obtained by mapping the arousal of the two modes from [-3, 3] to [0, 100] and then weighting them. The whole process is calculated using the average sentiment. The physical and mental balance is comprehensively assessed through tension, anxiety and stress. The higher the anxiety, the lower the physical and mental balance. Negative emotion indicators include aggression, tension, expression stress, and suspiciousness; The aggression is calculated by combining the two dimensions of hostility and uncontrollability, using only audio data; the distance is measured as a percentage, and the percentages of the two emotions are weighted to obtain the final result; The tension index is comprehensively evaluated by combining the results of brain waves, images and audio; the distance measurement percentage is used, and the audio, video and brain wave data are weighted to obtain the final percentage; wherein the tension in the audio is replaced by anxiety, and the tension value is directly used for the brain wave data; The expression pressure is evaluated by image results, intensity is directly mapped from [-3, 3] to [0, 100], and the average value of the emotion in the whole process is used for calculation; The suspicion is calculated through three dimensions: vigilance, tension and alertness. First, the distances of image, audio and brain wave data are calculated in the custom PAD emotional space to obtain percentages, and then the vigilance, tension and alertness of the three modalities are weighted to obtain the final suspicion. Among them, the tension of the audio is obtained by converting anxiety, and the alertness of the brain wave is obtained by converting tension. Physiological and emotional indicators include attention, relaxation, fatigue index, anxiety index, stress index, depression index, alertness, control, wakefulness, happiness, depression, and energy; The concentration and relaxation levels are obtained by EEG biofeedback instrument through Rawdata analysis; The fatigue index is evaluated by the arousal of images, audio and brain waves. The arousal of the three are directly mapped to [0,100], and then weighted in equal proportion to obtain the final non-fatigue percentage. Then, the fatigue index percentage is obtained by subtracting the non-fatigue percentage from 100%. The anxiety index is calculated by distance measurement; according to the position of anxiety in the PAD emotional space, the distance between the current emotion and anxiety is calculated; The stress index is calculated according to arousal, valence and dominance, and is evaluated based on the PAD emotional space; The depression index is calculated according to arousal, valence and dominance, and evaluated according to the PAD emotional space; The alertness is obtained by adding the weight of 0.9 given to the proportion of beta waves and the weight of 0.1 given to the proportion of gamma waves; The control index is combined with the audio and brain wave results for comprehensive evaluation; the distance measurement percentage is used, and the audio and brain wave data are weighted to obtain the final percentage; The wakefulness index is comprehensively evaluated by combining the audio and brain wave results; using a distance metric percentage, and weighting the audio and brain wave data to obtain a final percentage, in which the brain wave data accounts for the main weight; The happiness is obtained by mapping arousal to [0,100]; The depression is represented by the weighted dominance and happiness; high dominance and low happiness represent depression, calculated as 100% minus the weighted percentage of happiness and dominance; The energy is converted by arousal and intensity; the energy is obtained by multiplying the arousal and intensity at each moment, where arousal comes from the weighted addition of the three modalities and intensity comes from the image modality.
5. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 4, Features: The mapping process used in the calculation of the positive emotion, negative emotion and physiological emotion is expressed by the following formula: X i =m·L i +c Among them, L i is the original probability value, X i is the mapped value, m is the mapped multiplier, and c is the mapped constant term; The weighting process is expressed by the following formula: Among them, X i is the i-th mean value, w i is the weight corresponding to the i-th average value, and n represents the total number of values involved in the calculation; The mapping process of the PAD emotion space is as follows: Calculate the maximum distance D between each extreme point in the PAD emotional space and the emotional coordinate T max : D max =max{D 1 ,D 2 ,...,D n } Calculate the distance D between the current emotion and the emotion coordinate T 当前情绪 : Where T = [T x , T y , T z ] is the coordinate of a certain emotion position in the emotion space, E = [x, y, z] is the PAD emotion space coordinate of the current emotion; Calculate the percentage of emotion as follows: The calculation of the energy index is as follows: For each emotion pair (a i ,i i ), calculate the energy by multiplying its absolute value as follows: energy i =|a i |×|i i | If the energy list is empty, it returns 0; Calculate the maximum energy value max_energy in the energy list to normalize the energy value to the range of [0,100] as follows: Sum the normalized energy lists and divide by the length of the energy lists to get the average energy value, as follows: Among them, len(normalized\_energy represents the length of the energy list.
6. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 5, Features: The specific process of audio processing in the audio processing module and CNN14 model training in the audio prediction model is as follows: Step 1: Generate a log-Mel spectrogram; Step 1.1: Frame division; The audio signal x(t) is divided into several time frames, and each frame is weighted using the window function w(k). The purpose of the window function is to reduce spectrum leakage and smooth the edges, as shown in the following formula: x f [n]=x[n·H+k]·w[k],k=0,1,...,L-1 Where H is the step size, 10 milliseconds; L is the window size, 32 milliseconds, k represents the sampling point index within the window function w[k], from 0 to L-1, x f [n] represents the information of the nth frame after being processed by the window function, and f is the index of the frame, which is used to distinguish different frames; Step 1.2: Perform short-time Fourier transform; Perform the Fourier transform on each frame as follows: Among them, π represents pi, which is approximately equal to 3.14159, and X f [m, k] is the kth frequency component of the mth frame; Step 1.3: Convert to Mel frequency scale; Use the Mel filter to convert the spectrum to a Mel spectrum: Where, f represents the index of the Mel band, H[f, k] is the filter coefficient of the Mel filter bank, which represents the response of the f-th Mel band to the k-th frequency component; Step 1.4: Take the logarithm; Take the logarithm of the Mel spectrum to get the log-Mel spectrum: Log-Mel(m,f)=log(M[m,f]) Step 2: The log-Mel spectrogram obtained in step 1 is sent to the CNN14 model as a feature map. The CNN14 model includes 6 convolution blocks, each of which has two layers. Each layer has a maximum pooling operation, where the convolution kernel size is 3*3, the step size is 1, and the maximum pooling layer step size is 2; the details are as follows: Step 2.1: Convolutional layer; The output of each convolutional layer is represented as: C l =σ(W l *X l +b l ) Among them, W l is the convolution kernel, * indicates the convolution operation, b l is the bias, σ is the activation function, X l is the characteristic log-mel spectrogram of the input; Step 2.2: Max pooling layer; The output of each pooling layer is represented as: P l =MaxPool(C l ) Step 2.3: Dropout layer; Dropout is applied after each convolutional block: D l =Dropout(P l ,p) Among them, p is the probability of Dropout; Step 2.4: Average pooling and maximum pooling; The output of the last convolutional layer is subjected to both average pooling and maximum pooling: Among them, N represents the total number of elements in the feature map, C i Represents the value of the i-th element in the feature map; Step 2.5: Fully connected layer; The pooled features are input into the fully connected layer: F 1 =σ(W 1 ·[MeanPool;MaxPool]+b 1 ) F 2 =σ(W 2 ·F 1 +b 2 ) Among them, W 1 and W 2 represents the weight matrix of the fully connected layer, b 1 and b 2 represents the corresponding bias term, σ represents the activation function; After the convolutional layer transformation, the feature representation of the output time series has a dimension of T×512, where T is the number of time steps of the feature vector; then the output feature is transformed into a high-dimensional feature through projection, and the dimension after transformation is 1024. The transformation formula is as follows: H=f project (F 2 )=FW p +b p Among them, f project represents the projection function, W p and b p Represent the weight matrix and bias term of the projection layer respectively; Step 3: Transformer encoding layer; The Transformer encoder consists of 24 Transformer layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network, with a dimension of 1024; Step 3.1: Multi-head self-attention mechanism; the input of the first layer is H, and for the input of the lth layer H (L-1) , as follows: Step 3.1.1: Linear transformation: Q=H (l-1) W Q ,K=H (l-1) W K ,V=H (l-1) W V Among them, W Q , W K , W v The dimensions of W are 1024×1024; Q , W K and W v are the weight matrices of query, key, and value, respectively. Q, K, and V are the query, key, and value matrices obtained through linear transformation; Step 3.1.2: Calculate the self-attention score: in, h is the number of heads; Step 3.1.3: Multi-head attention: MultiHead(Q,K,V)=Concat(head 1 ,...,head h )W O Among them, the output of each attention head is head i =Attention(Q i , K i , V i );W O It is the weight matrix used for linear transformation after splicing, with a dimension of 1024×1024; Step 3.2: Feedforward neural network; Step 3.2.1: First layer linear transformation and activation function: FFN 1 (x)=ReLU(xW 1 +b 1 ) Among them, W 1 The dimension is 1024×4096, b 1 is bias; Step 3.2.2: Second layer linear transformation: FFN 2 (x)=ReLU(xW 1 +b 1 )W 2 +b 2 Among them, W 2 The dimension of b is 4096×1024, 2 is bias; Step 3.3: Each sub-layer is followed by layer normalization and residual connection: Step 3.3.1: Self-attention sublayer: H (l) =LaverNorm(H (l-1) +MultiHead(O, K, V)) Step 3.3.2: Feedforward neural network sublayer: O=LayerNorm(H (l) +FFN(H (l) )) After 24 layers of Transformer, the final output O is obtained; Step 4: Pooling operation: Apply a pooling operation to the output O of the last layer of the Transformer encoder to obtain a fixed-size output vector: p=f pool (O) Step 5: Output: The pooled feature vector p passes through one or more fully connected layers and then through an output layer to predict the sentiment dimension score: y=f output (P) Among them, y is the predicted three-dimensional vector, including the scores of arousal, dominance and emotional value: y 1 =f arousal (P) y 2 =f dominance (P) y 3 =f valence (P) Among them, arousal, dominance and valence represent arousal, dominance and affectivity respectively, which are three commonly used dimensions to describe emotional states.
7. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 5, Features: The image processing performed by the image processing module and the emotion estimator based on finite element analysis and partial least squares regression of geometric features are as follows: Step 1: Get image key points; Step 1.1: Image preprocessing; After the camera captures the image, use the "cvtColor" function in the OpenCV library to convert it into a grayscale image; Step 1.2: key point detection; Use DLIB's forward face detector get_frontal_face_detector() to perform face detection and extract the outer rectangular frame of the face; use the 68-point face feature detector trained by the DLIB library to extract the facial contour features and obtain the coordinates of the 49 facial coordinate points after screening; Step 2: Face positive orientation; Step 2.1: Scaling, translation and rotation: First, scale, translate, and rotate the coordinate points so that all coordinate points are on the same frontal expression: in: is the position point or feature vector of the kth sample in the original coordinate system, where o represents the original coordinate system; Indicates all The mean vector of is the mean of all sample points in the original coordinate system and is used for translation processing; R k is the rotation matrix, which is used to adjust the orientation of the coordinate point; k: represents the index of the sample; o: the identifier in the original coordinate system, which is used to indicate the point in the original system; M: represents the coordinate system after scaling, translation and rotation; Step 2.2: Construct a transposed matrix A containing the coordinate points of all non-positive expressions and a true value matrix Y containing the corresponding coordinate points of positive expressions, and fill the normalized coordinates into the matrix; Step 2.3: Objective function: argmin X |Y-AX| 2 +λ|X| 2 Among them, λ is the regularization parameter; X represents the parameter matrix to be optimized; Step 2.4: Positive mapping of the predicted value of X: Where I represents the identity matrix, whose dimensions are Same; T means transpose; Step 3: Partial least squares regression; Step 3.1: Input and Output: X 1 : Initial features, size is N × 1176, 1176-dimensional features calculated by the pairwise distance of 49 facial landmarks; Y 1 : Response features, size is N×3, including outputs of three dimensions; Step 3.2: Model representation: Among them, T and U are latent variable matrices: the size is N×29, there are 29 latent variables; P and Q are loading matrices; E and F are residual matrices; Step 3.3: Objective function; The goal is to maximize the covariance between the response feature and the predictor feature: Where t and u are column vectors of T and U respectively; X 1 Represents the initial feature matrix, which is used to predict the feature data; X 1 represents the response feature matrix, which contains the value of the target variable; w and v are two direction vectors used to project data in the feature space and response space respectively; X 1 w represents the matrix X 1 The product of Y and vector w is the projection along the w direction in the feature space; 1 v represents the matrix Y 1 The product of vector v is the projection along the v direction in the response space; Step 3.4: Solve the regression coefficient matrix B as follows to convert the feature matrix X 1 Mapping to predicted response: Step 3.5: Predicted response matrix: The final predicted response matrix is calculated by the following formula: in, Represents the predicted response feature matrix, including scores for arousal, emotion, and intensity:
8. The multimodal real-time mental health analysis system based on artificial intelligence method according to claim 5, Features: The specific steps of brain wave data processing and emotion index scoring are as follows: Step 1: Calculate the mean: The input is the value x of the i-th data point i , the total number of data points n; The output is the mean μ of the data points; Step 2: Calculate the standard deviation: The input is the mean μ of the data point, and the output is the standard deviation σ of the data point; Step 3: Construct the probability density function of the normal distribution: The input is the value x of the current data point, the mean μ of the data point, and the standard deviation σ of the data point; The output is the probability density value f(x) of the data point (x) under normal distribution; Step 4: Calculate the integral of the data of the current second over the distribution: The input is the value x_current of the data point at the current second and the probability density value f(x) of the data point (x) under the normal distribution; The output is the cumulative distribution function value P(x current ); Step 5: Normalize and convert to score: Score=100×P(x current ) The input is the cumulative distribution function value P(x current ); The output is the normalized score Score of the current data point; Step 6: The overall process formula is:
Citation Information
Patent Citations
Human brain idea recognition system based on invasive BCI
CN110974219A
Emotion recognition method and system based on N170
CN112488002A
Multi-modal emotion recognition method and device, electronic equipment and storage medium
CN115359576A
Operating personnel on-line multi-mode identification system based on multi-mode feature fusion
CN116226715A
Emotion recognition method and system based on multiple modes
CN118312857A
Cited By
Pension institution remote monitoring system based on image processing
CN119324051A
Digital human emotion perception and dynamic response method, system and device and storage medium
CN119884327A
Personality perception pet multi-mode emotion recognition method
CN121331171A
Brain-computer training system and method based on electroencephalogram attention assessment
CN121927178A
