Training method of personality recognition model, personality recognition method and related product

Abstract

The invention discloses a personality recognition model training method, a personality recognition method and a related product, and relates to the technical field of electroencephalogram and electromyographic signal processing, the actual personality score of a target subject and corresponding electroencephalogram signals and electromyographic signals are acquired, and feature extraction is performed to obtain electroencephalogram features and electromyographic features; performing element-by-element multiplication on the electroencephalogram features and the myoelectricity features to obtain element-by-element multiplication features; splicing the element-by-element multiplication features, the electroencephalogram features and the myoelectricity features to obtain spliced features; performing attention adjustment on the splicing features through a relative position attention mechanism to obtain attention features; performing personality recognition according to the attention features to obtain personality prediction scores; and according to the difference between the personality prediction score and the actual personality score of the target subject, carrying out iterative training to obtain a personality recognition model. According to the method, personality identification is carried out by fusing various physiological signals, so that personality characteristics of individuals are reflected more comprehensively and accurately, and the accuracy of personality evaluation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electroencephalogram and electromyogram signal processing, and in particular to a personality recognition model training method, a personality recognition method and related products. BACKGROUND

[0002] Personality is the internal characteristic of individual behavior patterns, which can predict individual behavior, thinking and emotional response to some extent in different situations. In the field of mental health, by assessing the personality characteristics of individuals, professionals can understand the emotional needs of patients and provide more personalized treatment plans. Therefore, accurate assessment of personality is of great significance in many fields. Neurophysiological signals have great potential in personality assessment due to their strong objectivity and good practical feasibility. Studies have shown that the brain stores human emotional experiences and unique personality. Most of the current personality recognition is based on electroencephalogram (EEG), but the personality recognition method based on a single modality such as EEG faces the challenge of being unable to fully reflect personality. SUMMARY

[0003] The purpose of the present application is to provide a personality recognition model training method, a personality recognition method and related products, which can integrate multiple physiological signals for personality recognition, thereby more comprehensively and accurately reflecting the personality characteristics of individuals, and improving the accuracy of personality assessment.

[0004] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0005] In a first aspect, the present application provides a personality recognition model training method, comprising:

[0006] obtaining the actual personality score of a target subject and the corresponding electroencephalogram signal and electromyogram signal;

[0007] using a feature extraction module to respectively extract features from the electroencephalogram signal and the electromyogram signal, to obtain electroencephalogram features and electromyogram features;

[0008] using a cross-modal feature fusion module to multiply the electroencephalogram features and the electromyogram features element by element, to obtain element-by-element multiplication features;

[0009] using a cross-modal feature fusion module to concatenate the element-by-element multiplication features, the electroencephalogram features and the electromyogram features, to obtain concatenated features;

[0010] using a cross-modal feature fusion module to adjust the attention of the concatenated features through a relative position attention mechanism, to obtain attention features;

[0011] The personality recognition is performed according to the attention feature by using a full connection layer, and a personality prediction score is obtained.

[0012] According to the personality prediction score and an actual personality score of the target subject, the feature extraction module, the cross-modal feature fusion module and the full connection layer are iteratively trained to obtain a personality recognition model.

[0013] Optionally, the target subject includes a pilot and an ordinary subject; and the electromyography signal includes a corrugator electromyography signal and a zygomatic electromyography signal.

[0014] The electroencephalogram signal includes an electroencephalogram signal in a calm task-free state and an electroencephalogram signal in a target task state.

[0015] The electromyography signal includes an electromyography signal in a calm task-free state and an electromyography signal in a target task state.

[0016] Optionally, the feature extraction module is used to perform feature extraction on the electroencephalogram signal and the electromyography signal respectively to obtain electroencephalogram features and electromyography features, and specifically includes:

[0017] The differential entropy feature of the electroencephalogram signal is calculated.

[0018] The trained identity classification convolutional neural network is used to divide the corresponding target subject into a pilot and an ordinary subject according to the differential entropy feature.

[0019] The feature extraction module is used to perform feature extraction on the modality signals of the pilot and the ordinary subject respectively and independently to obtain pilot features and ordinary subject features; the modality signals include electroencephalogram signals and electromyography signals; the pilot features and the ordinary subject features both include electroencephalogram features and electromyography features; the electroencephalogram signal is a result of feature extraction on the electroencephalogram signal; and the electromyography feature is a result of feature extraction on the electromyography signal.

[0020] Optionally, the training process of the identity classification convolutional neural network specifically includes:

[0021] The training data of the first subject is obtained; the training data includes the identity, the electroencephalogram signal and the big five personality score of the first subject; and the identity includes a pilot and an ordinary subject.

[0022] The differential entropy feature of the electroencephalogram signal is calculated to obtain a training differential entropy feature, and the calculation formula is as follows: Wherein, H Δx (X) represents the differential entropy feature, x i represents a sample point in the sequence of the electroencephalogram signal, Δx represents the difference between two adjacent sample points in the sequence of the electroencephalogram signal, and X represents the entire sequence of the electroencephalogram signal.

[0023] training the convolutional neural network with the training differential entropy features and the Big Five personality scores as inputs, the identity as a label, and an identity classification result as an output to obtain an identity classification convolutional neural network.

[0024] Optionally, the feature extraction module is configured to extract features from the EEG signal and the EMG signal respectively to obtain EEG features and EMG features, and specifically includes:

[0025] The two-layer two-dimensional convolution is configured to extract features from the EEG signal and the EMG signal respectively to obtain EEG extracted features and EMG extracted features.

[0026] The improved Transformer model is configured to extract global time sequence features from the EEG extracted features and the EMG extracted features respectively to obtain the EEG features and the EMG features, wherein the improved Transformer model is a model in which the absolute position encoding and the multi-head attention mechanism in the Transformer are replaced by a relative position attention mechanism.

[0027] Optionally, the full connection layer is configured to perform personality recognition according to the attention features to obtain personality prediction scores, and specifically includes:

[0028] The attention features are globally averaged and pooled to obtain pooled features.

[0029] The full connection layer is configured to perform personality classification according to the pooled features to obtain personality prediction scores, and the personality prediction scores include openness, extraversion, neuroticism, conscientiousness, or agreeableness.

[0030] In a second aspect, the present application provides a personality recognition method, including:

[0031] obtaining EEG signals and EMG signals of a subject to be recognized;

[0032] inputting the EEG signals and the EMG signals of the subject to be recognized into a personality recognition model to obtain personality prediction scores of the subject to be recognized, and the personality recognition model is trained by the training method of the personality recognition model described above.

[0033] In a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the training method of the personality recognition model or the personality recognition method described above.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the above-mentioned methods for training the personality recognition model or the above-mentioned personality recognition method.

[0035] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the above-mentioned methods for training the personality recognition model or the above-mentioned personality recognition method.

[0036] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0037] The present application provides a training method for a personality recognition model, a personality recognition method, and related products, wherein the training method for the personality recognition model includes: obtaining an actual personality score of a target subject and corresponding EEG signals and EMG signals; using a feature extraction module to perform feature extraction on the EEG signals and EMG signals respectively to obtain EEG features and EMG features; using a cross-modal feature fusion module to perform element-by-element multiplication on the EEG features and EMG features to obtain element-by-element multiplication features; using the cross-modal feature fusion module to splice the element-by-element multiplication features, the EEG features, and the EMG features to obtain spliced ​​features; using the cross-modal feature fusion module to adjust the attention of the spliced ​​features through a relative position attention mechanism to obtain an attention feature; using a fully connected layer to perform personality recognition based on the attention feature to obtain a personality prediction score; and iteratively training the feature extraction module, the cross-modal feature fusion module, and the fully connected layer based on the personality prediction score and the actual personality score of the target subject to obtain a personality recognition model. This application combines EEG signals and EMG signals for personality recognition. In the existing technology, the single signal source of EEG signals is difficult to fully capture the complex personality characteristics of an individual and cannot reflect the multidimensional characteristics of personality. This application uses multimodal physiological signals and adopts a dynamic multiplicative attention fusion strategy, which can not only enhance the synergy between different signal modalities, but also better retain the pattern information of the signal and dynamically adjust the importance of features across time and space dimensions. This fusion method enables the system to more comprehensively and accurately reflect the personality characteristics of an individual, improves the accuracy of personality assessment, and thus more comprehensively and accurately reflects the personality characteristics of an individual. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0039] Figure 1 The flowchart of the training method of the personality recognition model provided in an embodiment of the present application.

[0040] Figure 2 The schematic diagram of the acquisition process of the original electroencephalogram and electromyogram signals provided in an embodiment of the present application.

[0041] Figure 3 The schematic diagram of the structure and execution process of the personality recognition model provided in an embodiment of the present application.

[0042] Figure 4 The schematic diagram of the structure and data processing process of the personality recognition system provided in an embodiment of the present application.

[0043] Figure 5 The schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0045] The above purposes, features and advantages of the present application can be more obvious and easy to understand. The present application will be further described in detail below with reference to the drawings and specific embodiments.

[0046] In an exemplary embodiment, as shown in Figure 1 A training method of a personality recognition model is provided, which is executed by a computer device, specifically, can be executed by a terminal or a server, or can be executed by a terminal and a server together. In the embodiments of the present application, the method is applied to a server as an example, which includes the following steps S1 to S7. Wherein:

[0047] S1, obtaining the actual personality score of a target subject and the corresponding electroencephalogram signal and electromyogram signal.

[0048] The target subject includes a pilot and a common subject; the electromyography signal includes a corrugator electromyography signal and a zygomatic electromyography signal; the electroencephalogram signal includes an electroencephalogram signal in a calm task-free state and an electroencephalogram signal in a target task state; and the electromyography signal includes an electromyography signal in a calm task-free state and an electromyography signal in a target task state.

[0049] In the embodiment, the electroencephalogram signal and the electromyography signal of the target subject are acquired. In actual application, first, experts are organized to analyze the application scene and possible conditions, and a relatively complete experiment paradigm that can accurately reflect the electroencephalogram and electromyography signals in the cognitive state, i.e., a target task, is designed. Then, the discharge signals of the cerebral cortex and the electrical activity signals of the facial muscles of the target subject in different cognitive activities (target tasks) are recorded and collected under the interference of external environment. The electroencephalogram signal and the electromyography signal of the target subject in the target task state are recorded, and at the same time, as a comparison, the electroencephalogram signal and the electromyography signal of the target subject in the calm task-free state are recorded. The effectiveness and feasibility of the acquired electroencephalogram signal and electromyography signal are evaluated, and the original electroencephalogram and electromyography signals that are disordered, disturbed, and cannot accurately perform subsequent analysis are removed.

[0050] As a specific implementation, as shown in Figure 2 The acquisition process of the original electroencephalogram and electromyography signal includes steps 11 to 14.

[0051] Step 11: Before the experiment, a piece of stimulating material is selected, including four emotional movie clip segments of happiness, sadness, calmness, and tension. The emotional video material is a Chinese movie clip selected from a previously published data set. All the clips have Chinese subtitles and the audio is played synchronously.

[0052] Step 12: The target subject performs the electroencephalogram and electromyography acquisition experiment in a special environment that can shield external interference. The computer screen and the computer volume are controlled at a level that the target subject feels completely comfortable. Before the experiment starts, the target subject is required to complete a personality measurement questionnaire to obtain the actual personality score data of the target subject.

[0053] The personality measurement questionnaire is used to obtain the data of the five personality traits of the target subject, which is used as the personality attribute of the target subject. When performing the personality classification task, the classification accuracy of the classification task is obtained. The personality measurement questionnaire adopts the Chinese version of the Big Five Inventory (BFI), which includes 44 questions.

[0054] Step 13: Two experiments are performed in total, each experiment including two sections, each section playing one emotional video stimulus to the subject. The first experiment contains two emotions, nervous and happy, and the second experiment contains calm to sad emotions. Each experiment is separated by at least one week to ensure the independence of each emotional experiment and improve experimental accuracy. After the experiment starts, a prompt and precautions are played for about 5s, then the EEG and EMG signals in the no-task state are collected (2s), then the emotional stimulus video is played, and the EEG and EMG signals after emotional stimulation are collected (90s). After the target subject completes a random emotional video, he needs to complete a self-evaluation questionnaire (15s) according to the real feelings of watching the emotional video.

[0055] The self-evaluation questionnaire is to obtain the real feelings of the target subject after watching the movie clips, and to obtain the excitation degree of the target subject in valence and arousal at this moment, whether it is the same as the emotion of the movie clip segment. The self-evaluation questionnaire uses the emotion self-evaluation scale (Self-Assessment Manikin, SAM).

[0056] Step 14: Set a rest time (15s), and then continuously cycle the above steps 12 and 13 until the target subject completes the viewing of all videos.

[0057] This embodiment selects 30 target subjects to participate in the EEG and EMG signal collection experiment, of which 10 are pilots, aged 22.55±1.5, and 20 are ordinary subjects, aged 23.42±2.2. All participants have normal hearing and vision and are right-handed, and are all informed and agree to the experimental content.

[0058] The EEG signal and the EMG signal are preprocessed, the EEG signal is removed from the power frequency interference and the artifact, and the EMG signal is removed from the power frequency interference and filtered, which specifically includes steps 21 to 28.

[0059] Step 21: The EEG signal is processed by electrode re-referencing. That is, the reference electrode of all electrodes is converted.

[0060] The conversion process of the electrode needs to select a reference electrode, so as to compare the potential difference between the potential recorded by other electrodes and the potential of the reference point. This embodiment uses global average as the reference electrode, that is, the average value of all electrodes is used as the reference electrode. After electrode conversion, the reference electrode of the EEG signal is changed or recalculated to change the reference benchmark of the EEG signal. The influence of different positions of the scalp surface electrodes can be eliminated or reduced, and the EEG signal is more comparable.

[0061] Step 22: The filtered brain electrical signal data is obtained by filtering the brain electrical signal after the reference of the electrode. That is, the data of the unnecessary frequency band is removed, that is, the frequency bands of δ (1-3 Hz), θ (4-7 Hz), α (8-13 Hz), β (14-30 Hz) and γ (31-50 Hz) are removed.

[0062] Step 23: The brain electrical signal resampling data is obtained by resampling the brain electrical signal filtering data, which can improve the calculation efficiency while retaining the information related to the cognitive state.

[0063] Step 24: The brain electrical signal after removing artifacts is obtained by removing artifacts from the brain electrical signal resampling data. That is, the electrooculogram, electrocardiogram and electromyogram signals that interfere with the brain electrical signal are removed, including eyelid movement, limb shaking and 50 Hz power frequency interference.

[0064] Step 25: The baseline correction and segmented condition classification processing are performed on the brain electrical signal after removing artifacts, and the brain electrical signal is saved. The obtained brain electrical and electromyogram signals of each subject are segmented to obtain brain electrical and electromyogram signals of 90s per segment. These data are classified according to emotions and classified as happy, sad, calm and nervous.

[0065] Step 26: The electromyogram signal is filtered and notched, that is, the electromyogram signal (30-250 Hz) of the required frequency band is filtered from the electromyogram signal by a band-pass filter to obtain electromyogram signal filtering data, that is, irrelevant information outside the effective frequency band of electromyogram is removed. The power frequency interference (such as 50 Hz or 60 Hz power line interference) in the electromyogram signal is removed by a notch filter.

[0066] Step 27: The baseline correction and segmented condition classification processing are performed on the electromyogram signal after removing filtering, and the electromyogram signal is saved.

[0067] In this embodiment, the target subject watches 64 movie clips in the target task state, each movie clip has a duration of 90s. The obtained brain electrical and electromyogram signals are segmented to obtain brain electrical and electromyogram signals of 90s per segment. The 64 movie clips watched by the target subject are divided into 4 emotion categories (happy, calm, sad), each emotion category has 16 segments. According to the emotions of the movie clips watched by the target subject in the target task state, the brain electrical and electromyogram signals of all target subjects are conditionally classified. Finally, the brain electrical and electromyogram signals of the 4 emotion categories are obtained.

[0068] Step 28: The segmented condition classified brain electrical and electromyogram signals are cut by a time window of a set duration, and the preprocessed brain electrical and electromyogram signals are obtained.

[0069] Specifically, the segmented and conditioned electroencephalogram signals are windowed, and a set of electroencephalogram features under different tasks are extracted, and information of different channels is extracted according to the electrode channel distribution. In this embodiment, a time window of 1s is selected, and a sliding time window with 50% overlap is superimposed to extract continuous electroencephalogram signals and electromyogram signals.

[0070] The preprocessed electroencephalogram signals are subjected to feature extraction to obtain electroencephalogram features in different frequency bands, combined with big five personality scores, and a multi-layer one-dimensional convolutional neural network is used to classify the target subject into ordinary subjects and pilots; the multi-layer convolutional one-dimensional neural network model is obtained by pre-training using a training sample set; the training sample set includes a plurality of electroencephalogram sample features, corresponding big five personality score features, and corresponding subject identity category labels.

[0071] Specifically, the differential entropy features of the five frequency bands of the electroencephalogram signals are calculated. The differential entropy features and the big five personality scores of the target subject are subjected to standardization processing. Input into a 4-layer 1-dimensional convolutional neural network for binary classification. For each sample, the 4-layer 1-dimensional convolutional neural network will perform binary classification on the input electroencephalogram signal features. The network will be trained according to the differential entropy features and their related big five personality score features. For each input sample, the network will output a probability value. If the network output probability value is greater than or equal to 0.5, the sample will be classified as a pilot; if less than 0.5, the sample will be classified as an ordinary subject.

[0072] The network is trained using a training data set, inputting electroencephalogram signal features and big five personality scores, and outputting the class prediction of each sample. The weights and biases in the neural network are adjusted through the backpropagation algorithm. Each training sample in the training sample set includes an electroencephalogram sample signal (differential entropy features after feature extraction), a corresponding personality score, and a subject label (pilot or ordinary subject). Since each subject includes multiple samples, these samples may have different classification results - i.e. some samples are classified as pilots, while others are classified as ordinary subjects. To solve this problem, a majority vote method is used to determine the class of each subject according to the classification results of all samples of each subject, and the two classes of subjects are independently subjected to subsequent experiments. The five frequency bands of the electroencephalogram signal data include the delta (1-3 Hz) band, the theta (4-7 Hz) band, the alpha (8-13 Hz) band, the beta (14-30 Hz) band, and the gamma (31-50 Hz) band.

[0073] Differential entropy feature H Δx (X) is: where x irepresents a sample point in the electroencephalogram signal sequence, represents a difference value between two adjacent sample points in the electroencephalogram signal sequence, that is, a differential (difference), and X represents the entire electroencephalogram signal sequence.

[0074] S2, feature extraction is performed on the electroencephalogram signal and the electromyogram signal respectively by using a feature extraction module to obtain electroencephalogram features and electromyogram features.

[0075] In this embodiment, the differential entropy features of the electroencephalogram signal are calculated, and the trained identity classification convolutional neural network is used to divide the corresponding target subject into a pilot and an ordinary subject according to the differential entropy features.

[0076] The feature extraction module is used to independently extract features from the modal signals of the pilot and the ordinary subject respectively to obtain pilot features and ordinary subject features; the modal signals include electroencephalogram signals and electromyogram signals; the pilot features and the ordinary subject features both include electroencephalogram features and electromyogram features; the electroencephalogram signal is the result of feature extraction on the electroencephalogram signal; and the electromyogram feature is the result of feature extraction on the electromyogram signal.

[0077] The training process of the identity classification convolutional neural network specifically includes:

[0078] Training data of a first subject are obtained; the training data include the identity, the electroencephalogram signal, and the big five personality score of the first subject; and the identity includes a pilot and an ordinary subject.

[0079] Differential entropy features of the electroencephalogram signal are calculated to obtain training differential entropy features.

[0080] The training differential entropy features and the big five personality score are taken as inputs, the identity is taken as a label, and an identity classification result is taken as an output, and a convolutional neural network is trained to obtain an identity classification convolutional neural network.

[0081] In this embodiment, the two types of subjects after division are subjected to personality recognition by using a personality recognition model based on fusion of electroencephalogram signals and electromyogram signals. As shown in Figure 3 The personality recognition model based on electroencephalogram signals and electromyogram signals includes a feature extraction module, a cross-modal feature fusion module, and a classification module.

[0082] In this embodiment, feature extraction is performed on the electroencephalogram signal and the electromyogram signal respectively by using a feature extraction module to obtain electroencephalogram features and electromyogram features, specifically including the following steps:

[0083] The feature extraction module extracts spatial and temporal features from the electroencephalogram signals and the electromyogram signals, and the feature extraction module comprises a multi-layer two-dimensional convolutional neural network (CNN) and an improved Transformer.

[0084] Specifically, first, two layers of two-dimensional convolution are adopted, the height of the convolution kernel of the first layer is set to be the same as the number of electrodes. The width of the convolution kernel is allocated to the length of the time slice, which is used to extract the spatial information on each time slice. The height of the convolution kernel of the second layer is set to 1. The width of the convolution kernel is allocated to the length of the time slice, which is used to extract the temporal features of each time slice. After each layer of convolution, a batch normalization layer and a maximum pooling are added to further compress the size of the feature map and enhance the generalization ability of the model. Then, the input is input to the improved Transformer to further extract global time sequence features to obtain the spatio-temporal features of each modality. The improved Transformer model replaces the absolute position encoding and multi-head attention mechanism in the traditional Transformer with a relative position attention mechanism. The features are projected into a space with the same length of features: wherein and represents the projected electroencephalogram features, w e and w m represent weight matrices, b e and b m are bias terms for adjusting the position of the projection, tan h is an activation function for nonlinear processing of the input, and the purpose is to map the two features into a new space so that they have the same length and are suitable for subsequent processing. F EEG and F EMG are the features output by the convolution module, respectively. The improved Transformer includes a relative position attention mechanism, a feedforward neural network, and layer normalization. The relative position attention mechanism dynamically encodes the relative distance between tokens, solving the problem that the role of absolute position gradually weakens, the limitation increases, and the computational complexity is high when the traditional Transformer model processes long sequences. The feedforward neural network transforms the features through two fully connected layers, and the layer normalization is used to stabilize the training process and avoid gradient vanishing and explosion.

[0085] The relative position attention mechanism encodes the relative position into the calculation of the attention score, and the input features are mapped to the query (Q), key (K) and value (V) matrices through linear transformation, and the expression is as follows:

[0086]

[0087] wherein, where b denotes the batch size, l is the sequence length, and d model is the model dimension, is a learnable parameter matrix.

[0088] Next, the Q, K, V matrices are split along the feature dimension into h attention heads:

[0089]

[0090] where d k = d model / h is the dimension of each attention head, and h represents the number of attention heads, each head corresponding to an independent attention mechanism.

[0091] The similarity between Q and K is calculated through dot product and scaling. Additionally, in relative position attention, a relative position bias B is added in the attention score. Then, the Softmax function is applied to the resulting score to calculate the attention weights:

[0092]

[0093] where the calculation formula of B is:

[0094]

[0095] where Einsum(·) represents the summation operation in d k dimension. is a learnable position encoding matrix, is a temporary tensor dynamically generated from it. For each pair of positions (i, j) in the input sequence, the dynamic position encoding is generated by the following formula:

[0096] P dynamic [i, j, :] = P[i - j + l max -1, :];

[0097] where l max denotes the maximum sequence length. P is initialized by Glorot uniform distribution and optimized by gradient descent during training. This method reduces the number of parameters and improves computational efficiency.

[0098] The results from all attention heads are then concatenated and passed through a linear transformation. This process can be represented as:

[0099] RelativePositionAttention(Q, K, V) = Concat(head1,..., head h )W o ;

[0100] wherein is a weight matrix for the combined output, is the output of the i-th attention head.

[0101] S3, element-wise multiplication is performed on the EEG features and the EMG features by using the cross-modal feature fusion module to obtain element-wise multiplication features.

[0102] S4, the element-wise multiplication features, the EEG features and the EMG features are spliced by using the cross-modal feature fusion module to obtain spliced features.

[0103] S5, the spliced features are adjusted by attention by using the cross-modal feature fusion module through a relative position attention mechanism to obtain attention features.

[0104] In this embodiment, the cross-modal feature fusion module fuses the EEG features and the EMG features through a dynamic multiplicative attention fusion strategy. The dynamic multiplicative attention fusion strategy includes three processes: element-wise multiplication, feature splicing and relative attention mechanism.

[0105] Specifically, element-wise multiplication is first used to capture the correlation between the two modalities. This interaction process is represented as:

[0106]

[0107] wherein is an element-wise multiplication operation.

[0108] Next, the features after interaction are spliced with the original EEG features and EMG features, retaining the information of a specific modality while emphasizing its complementary aspects. The fused features are represented as:

[0109]

[0110] Finally, the importance of the features is dynamically adjusted by a relative position attention mechanism, and the formula is as follows:

[0111] F atended =RelativePositionAttention(F fusion ).

[0112] S6, personality recognition is performed according to the attention features by using a fully connected layer to obtain a personality prediction score.

[0113] In this embodiment, the classification module globally averages the attention features, reducing the feature dimension while preserving the global information of the data. Then, classification is performed through a fully connected layer. Two different personality recognition strategies are adopted, mainly distinguished by the emotional integration method and the prediction granularity.

[0114] Specifically, the first strategy is to input all emotions into the EEG and EMG-based personality recognition model. The reduced dimension features are classified through a fully connected layer to generate a vector Then, a Softmax activation function is used to generate a probability distribution for each personality category, and the specific process is as follows:

[0115]

[0116] wherein, is the probability distribution of each personality, is the output of the Softmax function. is the fused feature of the i-th dimension, i represents a specific target personality dimension, and j is an index for traversing all personality dimensions and used for normalization calculation.

[0117] In the second strategy, personality recognition is performed for specific emotions, and EEG and EMG data are independently processed for each emotion e∈{happy, sad, nervous, calm}. For each emotion, the reduced dimension features are classified through a fully connected layer to generate a personality category prediction vector Then, a Softmax activation function is used to generate a probability distribution of the predicted personality category under each emotion, which is specifically represented as:

[0118]

[0119] wherein, is the probability distribution of each personality category under each emotion. is the output of the Softmax function. is the fused feature of the i-th dimension under each emotion.

[0120] The personality categories include openness, extraversion, neuroticism, conscientiousness, and agreeableness.

[0121] S7. According to the personality prediction score and the actual personality score of the target subject, the feature extraction module, the cross-modal feature fusion module, and the fully connected layer are iteratively trained to obtain a personality recognition model.

[0122] In the training process of the personality recognition model, the parameter optimization of the deep learning model is performed by using Bayesian optimization, cross-validation and other methods. The model is trained using the training set, and the model performance is evaluated using the validation set. According to the evaluation result, the optimal model parameters are selected. And deployed and used in practical applications.

[0123] The embodiment extracts the differential entropy features of the EEG and the Big Five personality scores, classifies the subjects, and performs personality recognition on each class of subjects based on the EEG and EMG fusion personality recognition model. In feature extraction, CNN and improved Transformer learn deeper temporal and spatial features. For the improvement of the Transformer, the relative position attention mechanism is introduced to replace the original absolute encoding and multi-head attention mechanism, so as to better capture the spatio-temporal dependence in long sequence data. In cross-modal feature fusion, a dynamic multiplicative attention fusion strategy is adopted, which fully considers the correlation and heterogeneity of each modality, and uses the relative position attention mechanism to weight all feature vectors and dynamically adjust the feature weights. The fused features are compressed by global average pooling, and the personality recognition task is performed by the softmax layer.

[0124] The embodiment proposes a personality recognition method based on multi-modal physiological signal fusion. EEG and EMG signals are collected simultaneously through the set experimental paradigm, differential entropy features of preprocessed EEG signals are extracted, and Big Five personality scores are used as auxiliary features. A one-dimensional convolutional neural network is used to divide the subjects into pilots and ordinary subjects, and the two types of subjects are independently tested thereafter. A personality recognition model based on EEG and EMG fusion is used to recognize the personality of each type of subject. The personality recognition model first uses CNN and improved Transformer to extract the temporal and spatial features of preprocessed EEG signals and preprocessed EMG signals. Then, a dynamic multiplicative attention fusion strategy is used for cross-modal feature fusion, fully considering the correlation and heterogeneity of each modality. Finally, the feature dimension is reduced by global average pooling, and classification is performed by the fully connected layer, which improves the accuracy and efficiency of personality recognition in practical scenarios.

[0125] Based on the same inventive concept, the embodiment of the present application also provides a personality recognition method, comprising the following steps:

[0126] Obtaining the EEG and EMG signals of the subject to be recognized.

[0127] Inputting the EEG and EMG signals of the subject to be recognized into the personality recognition model to obtain the personality prediction score of the subject to be recognized; the personality recognition model is trained by the personality recognition model training method described above.

[0128] Based on the same inventive concept, the embodiments of the present application also provide a personality recognition system, as shown in the drawings, which comprises an electroencephalogram and electromyogram data acquisition module, a data preprocessing module, a target subject classification module, and a personality recognition module based on electroencephalogram and electromyogram fusion. Figure 4 The data preprocessing module is used for preprocessing the raw electroencephalogram and electromyogram signals. The raw electroencephalogram is removed from power frequency interference and artifacts to obtain a preprocessed electroencephalogram signal. The preprocessing includes electrode re-reference, filtering, downsampling, artifact removal, and segmented condition classification. The raw electroencephalogram is removed from power frequency interference and filtering to obtain a preprocessed electromyogram signal. The preprocessing includes filtering, downsampling, and segmented condition classification.

[0129] The electroencephalogram and electromyogram data acquisition module is used to acquire raw electroencephalogram and electromyogram signals of a target subject. The raw electroencephalogram and electromyogram signals include electroencephalogram and electromyogram signals in a calm and task-free state and electroencephalogram and electromyogram signals in a target task state. The data acquisition module is also used to evaluate the collected electroencephalogram and electromyogram signals. The raw electroencephalogram and electromyogram signals that are cluttered and cannot accurately perform subsequent analysis due to interference are removed.

[0130] The data preprocessing module is used for preprocessing the raw electroencephalogram and electromyogram signals. The raw electroencephalogram is removed from power frequency interference and artifacts to obtain a preprocessed electroencephalogram signal. The preprocessing includes electrode re-reference, filtering, downsampling, artifact removal, and segmented condition classification. The raw electroencephalogram is removed from power frequency interference and filtering to obtain a preprocessed electromyogram signal. The preprocessing includes filtering, downsampling, and segmented condition classification.

[0131] Specifically, the data preprocessing module removes cluttered signals in the electroencephalogram and electromyogram signals and classifies them according to experimental conditions to obtain an electroencephalogram and electromyogram data set. That is, the electroencephalogram is removed from 50Hz power frequency interference and artifacts, and the frequency bands other than the delta (1-3Hz) band, the theta (4-7Hz) band, the alpha (8-13Hz) band, the beta (14-30Hz) band, and the gamma (31-50Hz) band are removed. The electromyogram signal is also removed from 50Hz power frequency interference and filtering, and the frequency bands other than the (30-250Hz) band are removed. Then, the electroencephalogram and electromyogram signals are segmented and classified under different conditions. Finally, the preprocessed electroencephalogram and electromyogram signals are windowed, a sliding time window with a window of 1s and an overlap of 50% is used, and the preprocessed electroencephalogram and electromyogram signals are obtained.

[0132] The target subject classification module is used to distinguish subjects into two categories: pilots and ordinary subjects.

[0133] Specifically, the differential entropy features of each channel of the preprocessed electroencephalogram signal in five frequency bands are extracted, combined with the big five personality scores, and a one-dimensional convolutional neural network is used to perform a two-class classification task of the target subject. The two classes of subjects are independently subjected to subsequent experiments.

[0134] The personality recognition module based on electroencephalogram and electromyogram fusion is used to perform personality recognition on the two target subjects respectively. This includes feature extraction, cross-modal feature fusion, and classification of the preprocessed electroencephalogram and electromyogram signals.

[0135] For feature extraction, CNN and improved Transformer are used to extract time and spatial features of preprocessed electroencephalogram signals and electromyogram signals respectively.

[0136] Specifically, one layer of two-dimensional convolution is used to extract spatial features, and then one layer of two-dimensional convolution is used to extract spatial features and local time features, and then improved Transformer is used to extract global time features. For the improvement of Transformer, considering the long sequence characteristics of EEG and EMG, a relative position attention mechanism is introduced to replace the original absolute encoding and multi-head attention mechanism, so as to better capture the spatial and temporal dependencies in long sequence data. The relative position attention mechanism is to introduce relative position into the multi-head attention mechanism, which not only enhances the modeling ability of relative position, but also effectively reduces the computational complexity.

[0137] For cross-modal feature fusion, a dynamic multiplicative attention fusion strategy is used to fuse the features of the two modalities. Element-wise multiplication operation, concatenation and relative position attention mechanism are used to effectively combine the information of the two modalities, enhance the representation ability of the features, and fully utilize the advantages of each modality.

[0138] Specifically, element-wise multiplication operation is performed on the extracted electromyogram features and electroencephalogram features to fully consider the correlation between modalities. The obtained features are concatenated with the electromyogram features and electroencephalogram features to fully consider the heterogeneity of each modality. Finally, the obtained features are dynamically adjusted in importance through the relative position attention mechanism.

[0139] For personality classification, global average pooling is used to compress the features, and then a fully connected layer is used for classification.

[0140] The embodiment first collects the electroencephalogram signal data and electromyogram signal data of the target subject, removes the power frequency interference and artifacts of the original electroencephalogram and electromyogram signals, then extracts features from the preprocessed electroencephalogram signals to obtain electroencephalogram features in different frequency bands, and uses a multi-layer one-dimensional convolutional neural network to classify the target subject into ordinary subjects and pilots. Then, for each type of subject after classification, an electroencephalogram and electromyogram personality recognition model based on electromyogram and electroencephalogram is used for personality recognition. The electroencephalogram and electromyogram personality recognition model based on electromyogram and electroencephalogram includes a feature extraction module, a cross-modal feature fusion module and a classification module. Specifically, in the feature extraction module, two layers of two-dimensional convolutional neural networks and an improved Transformer model are used to extract features from the preprocessed electroencephalogram signal data and electromyogram signal data to obtain original electroencephalogram features and electromyogram features. Then, the cross-modal feature fusion module performs cross-modal feature fusion through dynamic multiplicative attention fusion. Finally, the classification module performs classification through global average pooling and a fully connected layer, which improves the accuracy and efficiency of personality recognition.

[0141] In an example embodiment, a computer device is provided, which can be a server or a terminal, and an internal structure diagram thereof can be as shown in FIG. 1. Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a personality recognition model training method or a personality recognition method.

[0142] Those skilled in the art can understand that Figure 5 the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In an example embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0144] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0145] In an example embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0148] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0149] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0150] The principles and implementation modes of the present application are described by applying specific examples herein, and the above-mentioned embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A training method for a personality recognition model, characterized in that: include: Obtain the actual personality score of the target subject and the corresponding EEG and EMG signals; Using a feature extraction module, extracting features from the EEG signal and the EMG signal to obtain EEG features and EMG features; Using a cross-modal feature fusion module, perform element-by-element multiplication on the EEG feature and the EMG feature to obtain an element-by-element multiplication feature; Using a cross-modal feature fusion module, the element-by-element multiplication feature, the EEG feature, and the EMG feature are concatenated to obtain a concatenated feature; Using a cross-modal feature fusion module, the spliced ​​features are adjusted for attention through a relative position attention mechanism to obtain an attention feature; Using a fully connected layer, personality recognition is performed based on the attention features to obtain a personality prediction score; According to the personality prediction score and the actual personality score of the target subject, the feature extraction module, the cross-modal feature fusion module and the fully connected layer are iteratively trained to obtain a personality recognition model.

2. The method for training a personality recognition model according to claim 1, wherein: The target subjects include pilots and ordinary subjects; The myoelectric signals include frown myoelectric signals and zygomatic myoelectric signals; The EEG signals include: EEG signals in a calm and non-task state and EEG signals in a target task state; The electromyographic signals include electromyographic signals in a quiet, non-task state and electromyographic signals in a target task state.

3. The method for training a personality recognition model according to claim 2, wherein: The feature extraction module is used to extract features from the EEG signal and the EMG signal to obtain EEG features and EMG features, specifically including: Calculating the differential entropy feature of the EEG signal; Using the trained identity classification convolutional neural network, the corresponding target subjects are divided into pilots and ordinary subjects according to the differential entropy features; A feature extraction module is used to independently extract features from the modal signals of the pilot and the general subject to obtain pilot features and general subject features. The modal signals include: electroencephalogram (EEG) signals and electromyographic (EMG) signals. Both the pilot features and the general subject features include EEG features and EMG features. The EEG signals are the result of feature extraction from the EEG signals. The EMG features are the result of feature extraction from the EMG signals.

4. The method for training a personality recognition model according to claim 3, wherein: The training process of the identity classification convolutional neural network specifically includes: Acquire training data of a first subject; the training data includes the identity, EEG signal, and Big Five personality score of the first subject; the identities include: a pilot and an ordinary subject; The differential entropy feature of the EEG signal is calculated to obtain the training differential entropy feature. The calculation formula is: Among them, H Δx (X) represents the differential entropy feature, x i represents a sample point in the sequence of EEG signals, Δx represents the difference between two adjacent sample points in the sequence of EEG signals, and X represents the entire sequence of EEG signals; The training differential entropy feature and the Big Five personality score are used as input, the identity is used as a label, and the identity classification result is used as output to train a convolutional neural network to obtain an identity classification convolutional neural network.

5. The method for training a personality recognition model according to claim 1, wherein: The feature extraction module is used to extract features from the EEG signal and the EMG signal to obtain EEG features and EMG features, specifically including: Using two layers of two-dimensional convolution to extract features from the EEG signal and the EMG signal, respectively, to obtain EEG extraction features and EMG extraction features; An improved Transformer model is used to extract the global temporal features in the EEG extraction features and the EMG extraction features respectively to obtain EEG features and EMG features; wherein, the improved Transformer model is a model in which the absolute position encoding and multi-head attention mechanism in the Transformer are replaced by the relative position attention mechanism.

6. The method for training a personality recognition model according to claim 1, wherein: The method of using the fully connected layer to perform personality recognition based on the attention features to obtain a personality prediction score specifically includes: Perform global average pooling on the attention features to obtain pooled features; Using a fully connected layer, personality classification is performed based on the pooled features to obtain a personality prediction score; the personality prediction score includes: openness, extraversion, neuroticism, conscientiousness, or agreeableness.

7. A personality recognition method, characterized in that: include: Acquiring EEG signals and EMG signals of the subject to be identified; Inputting the EEG signal and the EMG signal of the subject to be identified into a personality recognition model to obtain a personality prediction score of the subject to be identified; The personality recognition model is obtained by training using the personality recognition model training method according to any one of claims 1 to 6.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training method for a personality recognition model according to any one of claims 1 to 6 or the personality recognition method according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a personality recognition model according to any one of claims 1 to 6 or the method for personality recognition according to claim 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a personality recognition model according to any one of claims 1 to 6 or the method for personality recognition according to claim 7 is implemented.

Images