Multi-mode-based user personality psychological prediction method and device

Abstract

The invention relates to the technical field of medical equipment and psychological health management, and discloses a user personality psychological prediction method and device based on multiple modalities, and the method comprises the following steps: collecting personality psychological answer sheet data of a user, and carrying out the statistics of scores corresponding to all question options, and obtaining a total score related to each personality psychological category; collecting user face image data to perform personality type prediction analysis to obtain a personality psychological classification result; voice data and physiological data of the user are collected for emotion prediction, and an emotion prediction result of the voice data and the physiological data is obtained; and determining a final personality psychological prediction result according to the total score related to each personality psychological category, the personality psychological classification result and the emotion prediction result of the voice and physiological data. Therefore, the technical problems that data sources of existing personality psychological assessment methods such as questionnaire survey and interviews are single, subjectivity is high, real-time processing cannot be achieved, and the accuracy and comprehensiveness of assessment results are insufficient are solved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of medical equipment and mental health management, and in particular to a method and device for predicting user personality psychology based on multimodality. Background Art

[0002] In the field of psychology, the identification and evaluation of personality traits are of great significance for understanding individual behavior, emotional reactions, and social interactions. Traditional personality psychology mainly relies on subjective evaluation methods such as questionnaires and interviews. Although these methods can reflect individual personality traits to a certain extent, they are often affected by the self-reports of the subjects and have the following limitations: Single data source: relying only on questionnaire scores, it is difficult to reflect the real-time emotional state or dynamic changes of individuals. Strong subjective dependence: The results may be affected by the emotional state, answering habits, and social expectation effects of the subjects, resulting in limited accuracy. Inability to process in real time: The questionnaire method usually requires manual participation and cannot meet the needs of efficient and real-time psychological evaluation. In addition, if the evaluation is only based on questionnaire data or facial expression analysis, this limits the accuracy and comprehensiveness of the evaluation. Questionnaires may also be affected by the subjects' answer tendencies. In addition, voice, as an important carrier of individual emotional expression, also plays an important role in personality assessment, but existing technologies often ignore this aspect of information.

[0003] With the development of artificial intelligence and big data technology, it has become possible to use machine learning algorithms to objectively and automatically evaluate individual personality traits. However, how to design a scientific fusion strategy to integrate these heterogeneous information into a unified personality psychological classification output is a key technical difficulty. Therefore, there is an urgent need for a method that can comprehensively utilize multiple data sources (such as questionnaire data, facial expression data, and voice data) for personality assessment to improve the accuracy and comprehensiveness of the assessment. Summary of the invention

[0004] In order to address the deficiencies of the prior art, the embodiments of the present disclosure provide a user personality psychology prediction method and device based on multimodality to solve the technical problems of existing subjective evaluation methods such as questionnaires and interviews, such as single data source, strong subjectivity, inability to process in real time, and insufficient accuracy and comprehensiveness of evaluation results.

[0005] In a first aspect, the present disclosure provides a method for predicting user personality psychology based on multimodality, comprising the following steps:

[0006] Collect the user's personality psychology answer sheet data and count the scores corresponding to each question option to obtain the total score related to each personality psychology category;

[0007] Collect user face image data to perform personality type prediction and analysis to obtain personality psychological classification results;

[0008] Collecting the user's voice data and physiological data for emotion prediction, obtaining the emotion prediction results of the voice and physiological data, and then calculating the contribution value of each personality psychological category according to the emotion prediction results of the voice and physiological data; wherein the physiological data includes the user's skin electricity, breathing frequency and pupil data;

[0009] Determining a final personality psychology prediction result according to the total scores related to each personality psychology category, the personality psychology classification result and the emotion prediction result of the voice and physiological data includes:

[0010] Calculating the contribution value of each personality psychological category based on the emotion prediction results of the voice and physiological data;

[0011] The total scores related to the personality psychology categories, the personality psychology classification results and the contribution value sets of the personality psychology categories are subjected to nonlinear weighted judgment to obtain the final personality psychology prediction result.

[0012] In a possible implementation, the method further includes:

[0013] The comprehensive score is obtained based on the scores corresponding to each question option. The estimated user psychological type is obtained by matching the comprehensive score with the personality psychological result with the closest score.

[0014] The estimated user psychological type and the final personality psychology prediction result are compared for differences, and the estimated user psychological type is corrected according to the final personality psychology prediction result.

[0015] In a possible implementation, the collecting of user face image data for personality type prediction and analysis to obtain personality psychological classification results includes:

[0016] Preprocessing facial image data;

[0017] Performing feature extraction on the preprocessed face image data; wherein the feature extraction includes preliminary extraction and secondary extraction, wherein the preliminary extraction includes performing a depthwise separable convolution operation and a preliminary batch normalization transformation on the preprocessed face image data to obtain preliminary extracted image features; the secondary extraction includes performing a depthwise separable convolution operation on the preliminary extracted image features, and using an activation function to enhance feature representation, and finally fusing the preliminary extracted image features with the features processed by the depthwise separable convolution operation through a residual connection to obtain secondary extracted image features;

[0018] Based on the secondary extracted image features, the classification head is used to perform personality type prediction and analysis to obtain the personality psychology classification results.

[0019] In one possible implementation, the model used to predict personality type based on facial data is a deep learning convolutional neural network, and the loss function of the model includes a weighted sum of target confidence loss, non-target confidence loss and category loss.

[0020] In a possible implementation, the loss function is specifically:

[0021] L total =λ obj L obj +λ noobj L noobj +λ class L class

[0022] Where L obj , L noobj and L class Represent target confidence loss, non-target confidence loss and category loss respectively, λ obj , λ noobj and λ class It is the weight coefficient corresponding to each part, which is used to balance the impact of various losses.

[0023] In one possible implementation, the training process of the convolutional neural network model includes: collecting labeled face image data as a training set, inputting the model for forward propagation to calculate the loss, updating the model parameters through a back propagation algorithm, and iterating multiple times until the loss converges.

[0024] In a possible implementation, the collecting of the user's voice data and physiological data for emotion prediction to obtain the emotion prediction results of the voice and physiological data includes:

[0025] Preprocessing the voice data and physiological data;

[0026] Extract the acoustic and language features of the preprocessed speech data, including pitch, volume, timbre, speaking speed, and vocabulary usage; perform unsupervised learning on the user emotion classification based on the extracted acoustic and language feature vectors to obtain the emotion results predicted by the speech data;

[0027] Extract the features of preprocessed physiological data, including skin conductance level, respiratory variability, pupil diameter change rate, and blinking times; construct a physiological data emotion prediction model to process the time series feature data of physiological data and obtain the emotion results predicted by physiological data;

[0028] The emotion result predicted by the speech data is integrated with the emotion result predicted by the physiological data to obtain the emotion prediction results of the speech and physiological data.

[0029] In a possible implementation, the step of performing unsupervised learning on user emotion classification based on the extracted acoustic and language feature vectors to obtain the emotion result predicted by the speech data includes:

[0030] Calculate the Euclidean distance between feature vectors of different speech data;

[0031] Determine a core point according to the Euclidean distance and a predetermined distance threshold and a minimum number of neighborhood points;

[0032] The acoustic and language feature vectors are clustered and divided according to the core points to obtain the emotion results predicted by the speech data; wherein each cluster in the emotion results predicted by the speech data represents an emotion category or combination.

[0033] In a possible implementation, a nonlinear weighted decision is performed on the total scores related to each personality psychology category, the personality psychology classification results, and the contribution value sets of each personality psychology category to obtain a final personality psychology prediction result, including:

[0034] According to the emotion prediction results of the voice and physiological data and the correlation between the preset emotional state and personality psychology type, the contribution value of each personality psychology category of the user is calculated in turn to obtain a contribution value set of the user's personality psychology category;

[0035] According to the total scores related to each personality psychological category, the probability of each personality psychological category and the contribution value set of the personality psychological category of the user, a nonlinear weighted decision formula is adopted to obtain the predicted value M of the personality psychological category k by adjusting the weight index and the weighting coefficient. k :

[0036]

[0037] Among them, M k is the predicted value of personality psychological category k; γ, δ and τ are the exponents for adjusting weights to control nonlinear amplification; α, β and ρ are weighting coefficients used to balance the influence of prediction probability and answer sheet score, satisfying α+β+ρ=1; for each personality psychological category k, the total score vector related to each personality psychological category calculated by statistical answer sheet score is Score=[Score1,Score2,...,Score K ], where Score1, Score2, ..., Score K are the total scores corresponding to personality psychological categories 1, 2, ..., K, Score k is the total score corresponding to personality psychological category k; the classification probability distribution of personality psychological categories output by the model is P = [P1, P2, ..., P K], where P1, P2, ..., P K are the predicted probabilities of personality categories 1, 2, ..., K, P k is the predicted probability of personality psychological category k, and Estimate the contribution value set of each personality psychological category of the user based on voice and physiological data in, are the contribution values ​​of personality psychological categories 1, 2, …, K respectively;

[0038]

[0039] Among them, w i，k is the importance weight of the i-th question to the k-th personality category, n is the total number of questions, S i is the score of the ith question;

[0040] After calculating the predicted values ​​of all personality psychological categories of the user M1, M2, ..., M k , ..., M K After that, by selecting the k k is used to determine the final personality psychological prediction result M:

[0041]

[0042] In a second aspect, the present disclosure also provides a user personality psychology prediction device based on multimodality, comprising:

[0043] The data statistics module is used to collect the user's personality psychology answer data and count the scores corresponding to each question option to obtain the total score related to each personality psychology category;

[0044] The face analysis module is used to collect user face image data for personality type prediction and analysis to obtain personality psychological classification results;

[0045] An emotion prediction module collects the user's voice data and physiological data to predict the user's emotions, obtains the emotion prediction results of the voice and physiological data, and then calculates the contribution value of each personality psychological category according to the emotion prediction results of the voice and physiological data; wherein the physiological data includes the user's skin electricity, breathing frequency and pupil data;

[0046] The psychological judgment module is used to determine the final personality psychology prediction result according to the total scores related to each personality psychology category, the personality psychology classification result and the emotion prediction result of the voice and physiological data, including:

[0047] According to the emotion prediction results of the voice and physiological data, the contribution value of each personality psychology category is calculated; the total score related to each personality psychology category, the personality psychology classification result and the contribution value set of each personality psychology category are subjected to nonlinear weighted judgment to obtain the final personality psychology prediction result.

[0048] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects:

[0049] 1. Improve the accuracy of assessment: The present invention overcomes the limitations of a single data source and improves the accuracy of personality assessment by integrating multiple data sources (such as questionnaire data, facial image data, voice data and physiological data). Questionnaire data provides information about an individual's self-cognition, facial data reflects an individual's immediate emotional response and facial features, and voice data and physiological data reveal an individual's voice characteristics and emotional state. Data from different modalities complement each other and together constitute a comprehensive assessment of an individual's personality traits.

[0050] 2. Enhance the objectivity and real-time nature of the assessment: Compared with traditional subjective assessment methods, the present invention uses machine learning algorithms to automatically process and analyze multimodal data, reducing the interference of human factors, enhancing the objectivity of the assessment, and helping to eliminate the deviations caused by factors such as individual subjective feelings, answering habits, and social expectations; it can achieve real-time personality assessment and greatly improve the efficiency of the assessment. This is especially important for scenarios that require rapid response (such as online psychological counseling, emergency rescue, etc.).

[0051] 3. Personalized service support: Through accurate and comprehensive personality assessment, the present invention can provide individuals with more personalized services, such as psychological counseling, career planning, social guidance, etc., to help individuals better understand themselves and improve their quality of life.

[0052] In summary, the present invention proposes a personality psychology assessment method and device based on multimodal data fusion, which conducts personality psychology assessment by fusing multimodal data questionnaire data, face data and voice data, which not only improves the accuracy and objectivity of the assessment, but also can realize real-time assessment, providing more personalized service support for individuals. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1A flowchart of a multi-modal user personality psychology prediction method provided in an embodiment of the present disclosure;

[0055] Figure 2 A schematic diagram of a user personality psychology prediction method based on multimodality provided in an embodiment of the present disclosure;

[0056] Figure 3 A unit module diagram of a multimodal user personality psychology prediction device provided by an embodiment of the present disclosure;

[0057] Figure 4 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0058] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0059] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor represent the necessary logical order between them. It should also be understood that in the embodiments of the present disclosure, "multiple" can refer to two or more, and "at least one" can refer to one, two or more. It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, in the absence of explicit limitation or contrary revelation given in the context, it can generally be understood as one or more. In addition, the term "and / or" in the present disclosure is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are an "or" relationship. It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments, and the same or similar parts can refer to each other. For the sake of brevity, they will not be repeated one by one.

[0060] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0062] Figure 1 A flowchart of a multi-modal user personality psychology prediction method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 The schematic diagram of a multi-modal user personality psychology prediction method provided by the embodiment of the present disclosure. Figure 1 and Figure 2 As shown, the method 100 comprises the following steps:

[0063] S110: Collecting the personality psychology answer sheet data of the user and counting the scores corresponding to each question option to obtain the total score related to each personality psychology category;

[0064] First, in step S110, the personality psychology answer sheet data filled in by the user is collected through an online questionnaire or an APP, and then the scores corresponding to the various question options are counted. The specific calculation process is as follows:

[0065] Definition S i，j represents the score of the user who selected the jth option in the ith question. Each question has multiple options, and the user selects an option based on his or her answer, and each option corresponds to a specific score. Assume that the ith question has m i options, the score of the jth option is S i，j We can get the total score S of the i-th question by taking the weighted sum of the scores of all options. i (i.e. the score of the selected answer):

[0066]

[0067] Among them, x i，j is the decision variable for the user to choose the jth option for the i-th question (if option j is chosen, then x i，j =1, otherwise x i，j =0); where the score of each question option is S i，j The calculation formula is:

[0068] S i，j =w j ·r i，j

[0069] Among them, S i，j The score for choosing the jth option for the i-th question; w j is the weight of the jth option; r i，j is the original score of the jth option.

[0070] Then, according to the score S of each question option i，j , calculate the total score related to each personality psychological category, the specific process is as follows:

[0071] S i Relationship with personality psychological category k:

[0072] S i S represents the score of the ith question, which is usually related to the user's preference or tendency towards a specific personality trait. For example, if the ith question is related to personality dimensions such as extraversion, neuroticism, and responsibility, then S i Indicates the user's score on this dimension.

[0073] Optionally, according to the MBTI personality type classification, there are 16 personalities in total. These 16 personalities are composed of 4 dimensions, namely introversion and extroversion, intuition and sensing, emotion and thinking, perception and judgment. Each dimension has two letters, and the four dimensions have a sequence of: E or I, N or S, F or T, P or J. By weighted summing up the scores of all questions or other appropriate methods, the total score under K = 16 dimensions (i.e., personality psychological categories) can be obtained, thereby inferring the overall personality characteristics of the user. The contribution of each question i to a certain personality psychological category k will be weighted based on its importance and relevance, including the following steps:

[0074] Weighted score: The score S for each question i Questions are weighted according to their relevance to a certain personality category. For example, question i has different impact on personality category k.

[0075] Comprehensive score: S is the weighted sum of the scores of all questions. i , get the final total score of each personality category k:

[0076]

[0077] Among them, Score k is the total score associated with the kth personality psychological category, w i，k is the importance weight of the i-th question to the k-th personality psychological category, S i is the score of the ith question, and n is the total number of questions.

[0078] S120: Collecting user face image data to perform personality type prediction analysis to obtain personality psychological classification results;

[0079] The step S120 specifically includes the following steps:

[0080] S121, user face image data collected by the terminal camera;

[0081] Secondly, the facial image data is preprocessed, such as cropping, normalization, and enhancement, to improve the image quality and make it meet the model input requirements.

[0082] S122, performing feature extraction on the preprocessed face image data; wherein the feature extraction includes preliminary extraction and secondary extraction, the preliminary extraction includes performing a depth-separable convolution operation and a preliminary transformation of batch normalization on the preprocessed face image data to obtain preliminary extracted image features; the secondary extraction includes performing a depth-separable convolution operation on the preliminary extracted image features, and using an activation function to enhance feature representation, and finally fusing the preliminary extracted image features with the features processed by the depth-separable convolution operation through a residual connection to obtain secondary extracted image features;

[0083] S1221, preliminary extraction: the face image data Z after preprocessing in S121 l-3 Perform a preliminary transformation of 3×3 depth-separable convolution operation and batch normalization:

[0084]

[0085] Among them, Z l-2 is the image feature initially extracted; Z l-3 is the input preprocessed face image data; BN is the batch normalization operation; is the activation function (GELU); DWConv 3×3 It is a 3×3 depth-wise separable convolution operation; l represents the index to distinguish whether the variable is an input or an output.

[0086] Dimension 3×3 depth-wise separable convolution operation DWConv 3×3 The formula is:

[0087]

[0088] Where F(g, h) usually represents the feature value extracted from the input image after a 3×3 depth-wise separable convolution operation at the image position (g, h); is the depth convolution result of the c-th channel output; W c (p, q) is the 3×3 convolution kernel of the cth channel; The value of the cth channel at the image position (g, h), g represents the index of the pixel in the horizontal direction (row) of the image, and h represents the index of the pixel in the vertical direction (column) of the image; W c，d is a 1×1 convolution kernel with channel mixing; C is the number of channels.

[0089] The normalization operation BN formula is:

[0090]

[0091] Among them, y v，u is the output value of the vth sample in the uth feature dimension after normalization; μ u is the sample mean, Sample variance; x v，u is the input data; u is the feature dimension; ∈ is a small positive number to avoid the denominator being zero; Y u is the scaling parameter; β u is the offset parameter; m is the number of samples in the current batch.

[0092] The activation function (GELU) formula is:

[0093]

[0094] In the above formula, x represents the input of the activation function, which can be a scalar (a single value), a vector (a set of values) or a matrix (a two-dimensional array), depending on the layer structure of the neural network and the dimension of the input data. During the forward propagation of the neural network, the output of each layer will be used as the input of the next layer. For the GELU activation function, x is the output of its layer, that is, the input data it processes. Among them, erf(x) is the error function (ErrorFunction), which calculates the cumulative distribution function of the Gaussian error; The function of is to convert the input x into a value between -1 and 1, which represents the cumulative probability of x relative to the standard normal distribution.

[0095] The GELU activation function implements a nonlinear transformation of the input by multiplying the input x and the output of the error function (after scaling and translation). This transformation not only preserves the positive and negative nature of the input (because x itself is multiplied by a value between 0 and 1), but also introduces nonlinearity, allowing the neural network to learn more complex feature representations.

[0096] Among them, preferably, the model used for predictive analysis of personality type based on face data is a deep learning convolutional neural network, and the loss function used by the model for preliminary extraction includes the weighted sum of target confidence loss, non-target confidence loss and category loss, and the loss function is specifically:

[0097] L total =λ obj L obj +λ noobj L noobj +λ class L class

[0098] Where L obj , L noobj and L class Represent target confidence loss, non-target confidence loss and category loss respectively, λ obj , λ noobj and λ clasx It is the weight coefficient corresponding to each part, which is used to balance the impact of various losses.

[0099] Target confidence loss L obj It measures the error in the model's confidence level about the existence of the target. It is expressed by the predicted confidence p a and the true target confidence The difference is calculated as:

[0100]

[0101] Among them, p a is the confidence level of the target in the prediction box; is the actual target confidence, usually 1 (if the target exists) or 0 (if the target does not exist); is an indicator function, indicating whether there is an object; a is an index used to identify a specific object box or prediction box, specifically used to distinguish the indexes of these different prediction boxes. In the object detection task, the model generates multiple prediction boxes, each of which has a corresponding confidence score, indicating the probability that the model believes that there is an object in the prediction box.

[0102] Confidence It refers to the model's predicted probability that the target exists in the a-th prediction box. It is a value between 0 and 1. The higher the value, the more confident the model is that the target exists in the prediction box.

[0103] True target confidence Refers to the true value of whether there is actually an object in the a-th prediction box. During the training process, this value is usually known and used to calculate the loss function and guide the learning of the model. If there is indeed an object in the a-th prediction box, then Normally 1; if no target exists, Usually 0.

[0104] For regions without objects, the non-object confidence loss L noobj The error between the predicted confidence and 0 is calculated:

[0105]

[0106] in, is an indicator function, indicating that there is no target in this area.

[0107] Class loss L class The difference between the predicted class probability and the true class is calculated.

[0108]

[0109] Among them, c a is the predicted class probability of the target; is the true category probability of the target; C is the number of categories; Specifically, it means that there is a target in the box. This is because the category loss is based on the premise that the model has already determined that there is a target in the predicted box (that is, the box is the target box), and further measures the difference between the predicted category probability and the true category probability.

[0110] (2) Secondary extraction: including the initial extraction of image features Z l-2 Through the depth-separable convolution operation and the use of activation function to enhance the feature representation, the initially extracted image feature Z is finally l-2 The features processed by the depthwise separable convolution operation are fused through residual connections to obtain secondary extracted image features, including:

[0111] Through 3×3 depth-separable convolution operation and using activation function to enhance feature representation, the initial extracted image feature Z of the original input is finally l-2 The processed features are fused through residual connections, and the loss function is L set in the above steps. total :

[0112]

[0113] Among them, Z l-1 is the output feature after secondary extraction, is the activation function (GELU), DWConv 3×3 is a 3×3 depth-wise separable convolution operation, represents a residual connection.

[0114] S123, based on the secondary extracted image features, a classification head is used to perform personality type prediction analysis to obtain a personality psychology classification result.

[0115] In step S123, based on the secondary extracted image feature Z l-1 , using the classification head to predict personality type:

[0116]

[0117] P(y=k|Z l-1 ) is given the input feature Z l-1 The probability that the sample belongs to the kth personality psychological category under the condition Z l-1 =[z1,z2,...,z K ],z k is the vector Z l-1 The kth element in , K is the total number of classification categories (if classified according to MBTI, the personality psychological category K is 16); It is a normalization factor after calculating the exponential of all categories to ensure that the output is a valid probability distribution.

[0118] In the personality type prediction or classification task disclosed in the present invention, z b Represents the original score or unnormalized log probability of the model for the input image (after feature extraction) belonging to the bth category. b Perform exponential calculation These raw scores can be converted to positive numbers, which amplifies the differences between the scores. This helps to convert these scores into probability distributions through the subsequent normalization operation.

[0119] In the step S120 of predicting and analyzing personality types based on face data, the model selected is a deep learning convolutional neural network, such as YOLOv3 or EfficientDet, which has excellent performance in processing image data. The model training process is as follows:

[0120] Data preparation: Collect a large amount of face image data with emotion labels as training sets and validation sets.

[0121] Model construction: Build a convolutional neural network model and define the loss function as the aforementioned Ltotal.

[0122] Training process: Input the training set data into the model, perform forward propagation to calculate the loss, and then update the model parameters through the backpropagation algorithm (such as Adam optimizer). Iterate multiple times until the loss function converges and the model performance reaches the optimal level.

[0123] Model evaluation: Evaluate model performance on the validation set, such as accuracy, recall, and other indicators to ensure the effectiveness of the model.

[0124] S130: Collecting the user's voice data and physiological data to predict the user's emotions, and obtaining emotion prediction results of the voice and physiological data;

[0125] Specifically, step S130 includes:

[0126] S131: Preprocessing the voice data and physiological data;

[0127] In step S131, the preprocessing of the speech data includes noise removal (implemented by filtering algorithms, smoothing processing, etc.), endpoint detection, etc., to improve the clarity of the speech signal and remove redundant silent parts.

[0128] Preprocessing of physiological outputs includes noise removal (through filtering algorithms, smoothing, etc.), outlier processing and interpolation, etc., to remove noise interference and abnormal data, and to maintain the continuity and integrity of the data. It can also include standardization or normalization: converting physiological data to a uniform scale or range for subsequent analysis and processing. For example, the skin conductance level (GSR) can be standardized to between 0 and 1, or the respiratory rate can be converted to the number of breaths per minute.

[0129] S132: extracting acoustic and language features of the speech data, including pitch, volume, timbre, speaking speed, and vocabulary usage, and performing unsupervised learning user emotion classification based on the extracted acoustic and language feature vectors to obtain an emotion result predicted by the speech data; including:

[0130] S1321: Extracting acoustic and language feature vectors X from the user voice data collected by the microphone:

[0131] X = [x1, x2, ..., x N ]

[0132] x i’ is the feature vector of the i'th speech sample; where each x i ， It contains the following features:

[0133] Pitch: frequency characteristic, a numerical value in Hertz (Hz);

[0134] Volume: Sound pressure level, a numerical value in decibels (dB);

[0135] Timbre: a vector containing the spectrum envelope and harmonic components;

[0136] Speech rate: the number of speech segments per second, a numerical value;

[0137] Vocabulary usage: term frequency - inverse document frequency value

[0138] Among them, n i’j’ Indicates the number of times word i' appears in document j', ∑ k’ n k’j’ Indicates the total number of occurrences of all words in document j'; N represents the total number of documents, df i’ Represents the number of documents containing word i'.

[0139] Among them, k' is used to index different feature dimensions. For example, in the extracted acoustic and language feature vectors x = [x1, x2, ..., x N ], each x i’ is a feature vector, which is an array or vector containing multiple feature values, which represent the attributes of the speech sample in different dimensions, such as pitch, volume, timbre, etc. k' is used to specify which dimension's feature value we want to view, that is, these dimensions can be indexed by k'.

[0140] S1322: Perform unsupervised learning on the user emotion classification based on the extracted acoustic and language feature vector x to obtain the emotion result predicted by the speech data.

[0141] The step S1322 specifically includes:

[0142] S13221: Calculate the Euclidean distance between feature vectors of different speech data;

[0143] Calculate the Euclidean distance d(x) between the feature vectors of different speech data i’ , x j’ ):

[0144]

[0145] Among them, x i’ , x j’ is the feature vector of two speech data, which comes from the acoustic and language feature vector X=[x1, x2, ..., x N ]; x i’，k’， x j’，k’are respectively the value of the k'th dimension in the feature vector of the i'th speech sample and the value of the k'th dimension in the feature vector of the j'th speech sample; m' is the dimension of the feature vector.

[0146] S13222: Determine a core point according to the Euclidean distance and a predetermined distance threshold and a minimum number of neighborhood points;

[0147] The purpose of step S1322 is to set the core point. Given the distance threshold ∈ and the minimum number of neighborhood points MinPts, if a point x i’ Neighborhood N(x i’ ) satisfies the following conditions, then x i’ The core points:

[0148] N(x i’ )={x j’ |d(x i’ , x j’ )≤∈}

[0149] |N(x i ,)|≥MinPts

[0150] Among them, N(x i’ ) is the point x i’ The point set in the neighborhood; |N(x i’ )| is the number of points in the neighborhood; ∈ is the neighborhood radius (density threshold); MinPts: the minimum number of neighborhood points.

[0151] S13223: Clustering the acoustic and language feature vectors according to the core points to obtain the emotion result predicted by the speech data; wherein each cluster in the emotion result predicted by the speech data represents an emotion category or combination from any core point x i’ Starting from this, all density-reachable points are grouped into the same cluster; density-reachable is defined as: if point x j’ At point x i’ In the neighborhood of j’ is a core point, then point x i’ x j’ The density can be reached. If a point x i’ A point that is neither a core point nor a density-reachable point of any core point is considered a noise point.

[0152] In clustering algorithms such as DBSCAN, density reachability is an important concept that is used to determine which points should be classified into the same cluster. i’ Another point x can be reached through a series of densely connected core points j’ , then x i’ x j’are density-reachable. Here, "density-connected" means that the two core points are in each other's neighborhood. Noise points are isolated points that are neither core points nor density-reachable points of any core points. In the clustering process, these noise points are usually ignored or treated separately.

[0153] Each cluster in the clustering result represents an emotion category or combination. Let the final cluster label be C:

[0154] C = {c1, c2, ..., c K ,}

[0155] Where K′ is the number of clusters obtained by clustering. If the clusters are divided according to “joy, anger, sadness, fear, disgust, surprise, anxiety, expectation, calmness, love, jealousy, shame, pride, depression, boredom, and satisfaction”, the value of K′ is 16; c k’ is the probability of the emotion category of cluster k'.

[0156] S133: extracting features of the preprocessed physiological data, including skin conductance level, respiratory variability, pupil diameter change rate, and blinking times; constructing a physiological data emotion prediction model to process the time series feature data of the physiological data, and obtaining the emotion results predicted by the physiological data;

[0157] In this step S133, feature extraction is performed on the preprocessed physiological data, including:

[0158] For skin conductance data: extract features such as skin conductance level (GSR).

[0159] For respiratory rate data: calculate parameters such as respiratory rate and respiratory depth, and extract features such as respiratory variability (such as the standard deviation of the breathing interval).

[0160] For pupil data: calculate features such as the rate of change of pupil diameter and number of blinks.

[0161] In this step S133, it also includes constructing a physiological data emotion prediction model to process the time series feature data of the physiological data to obtain the emotion result predicted by the physiological data, specifically:

[0162] Build an emotion prediction model specifically for processing physiological data: For example, use recurrent neural networks (RNN), long short-term memory networks (LSTM), or gated recurrent units (GRU) to process time series data. And use physiological datasets with emotion labels to train the model so that it can accurately predict the user's emotional state.

[0163] S134: Fusing the emotion result predicted by the speech data with the emotion result predicted by the physiological data to obtain the emotion prediction result of the speech and physiological data;

[0164] In this step S134, it is preferred to achieve the fusion of the emotion result predicted by the speech data and the emotion result predicted by the physiological data through weighted averaging, voting or other fusion strategies.

[0165] S140: Determine a final personality psychology prediction result according to the total scores related to each personality psychology category, the personality psychology classification result, and the emotion prediction result of the voice and physiological data, including:

[0166] S141: Calculate the contribution value of each personality psychology category of the user in turn according to the emotion prediction result of the voice and physiological data and the correlation between the preset emotional state and personality psychology type, and obtain a contribution value set of the user's personality psychology category;

[0167] Specifically, in this step S141, for example, the correlation between 16 emotion types and 16 personality psychology categories may be preset as follows, see Table 1.

[0168] Table 1 MBTI personality psychological categories and emotional correlation weights

[0169]

[0170]

[0171] According to the correlation between the emotional state and different psychological types in Table 1, the estimated personality psychological type is obtained. The steps are as follows:

[0172] Assume that the emotion prediction result of the speech and physiological data predicted in step S130, i.e., the emotion probability distribution c k (sum is 1): C k ={Joy: 0.1, Anger: 0.05, Sadness: 0.2, Fear: 0.1, Disgust: 0.05, Surprise: 0.05, Anxiety: 0.2, Anticipation: 0.1, Calmness: 0.05, Love: 0.05, Jealousy: 0.02, Shame: 0.03, Pride: 0.02, Depression: 0.05, Boredom: 0.02, Satisfaction: 0.01}

[0173] Taking INTJ as an example, the weight matrix w of the emotional and psychological types in Table 1 is used to calculate the contribution value possibility of INTJ

[0174] 1 Joy: 0.1×0.1=0.01

[0175] 2Anger: 0.2×0.05=0.01

[0176] 3 Sadness: 0.4×0.2=0.08

[0177] 4 Fear: 0.3 × 0.1 = 0.03

[0178] 5 Disgust: 0.2×0.05=0.01

[0179] 6 Surprise: 0.1×0.05=0.005

[0180] 7 Anxiety: 0.4 × 0.2 = 0.08

[0181] 8 Expectation: 0.1×0.1=0.01

[0182] 9 Calm: 0.2×0.05=0.01

[0183] 10 love: 0.1×0.05=0.005

[0184] 11 Jealousy: 0.1×0.02=0.002

[0185] 12 Shame: 0.2 × 0.03 = 0.006

[0186] 13 pride: 0.3×0.02=0.006

[0187] 14 Depression: 0.4 × 0.05 = 0.02

[0188] 15 boredom: 0.3 × 0.02 = 0.006

[0189] 16 satisfies: 0.2×0.01=0.002

[0190]

[0191] Based on this, the contribution value set of the user's personality psychological category

[0192] S142: According to the total scores related to each personality psychological category, the probability of each personality psychological category and the contribution value set of the personality psychological category of the user, a nonlinear weighted decision formula is used to adjust the weight index and the weight coefficient to obtain the predicted value M of the personality psychological category k. k :

[0193] First, the calculation parameters obtained according to the above steps S110 to S130 include:

[0194] Step S110 obtains the answer sheet score: for each personality psychological category k, the total score vector related to each personality psychological category calculated by counting the answer sheet scores is:

[0195] Score=[Score1, Score2,..., Score K], where Score1, Score2, ..., Score K They are the total scores corresponding to personality psychological categories 1, 2,…, K respectively.

[0196] Step S120 obtains the image data prediction result: the classification probability distribution of personality psychology categories output by the model P = [P1, P2, ..., P k ], where P1, P2, ..., P K are the predicted probabilities of personality psychological categories 1, 2, …, K respectively.

[0197] Step S141 calculates the contribution value set of the user's personality psychological category based on the emotion prediction results of the voice and physiological data in, are the contribution values ​​of personality psychological categories 1, 2, …, K respectively;

[0198] Furthermore, step S142 specifically includes:

[0199] According to the total scores of each personality psychological category, the probability of each personality psychological category and the contribution value set of the user's personality psychological category, a nonlinear weighted decision formula is used to adjust the weight index and weight coefficient to obtain the predicted value M of personality category k. k :

[0200]

[0201] Among them, M k is the predicted value of personality psychological category k; Y, δ and τ are the exponents for adjusting weights to control nonlinear amplification; α, β and ρ are weighting coefficients used to balance the influence of prediction probability and answer sheet score, satisfying α+β+ρ=1; P k is the predicted probability of personality psychological category k, and is the contribution value of each personality psychological category k of the user.

[0202] Finally, after calculating the predicted values ​​of all personality psychological categories of the user, M1, M2, ..., M K After that, by selecting the k k is used to determine the final personality psychological prediction result M:

[0203]

[0204] The above-mentioned embodiment is a personality psychology assessment method based on multimodal data fusion. By fusing multimodal data questionnaire data, face data and voice data, personality psychology assessment is performed, which not only improves the accuracy and objectivity of the assessment, but also improves the comprehensiveness of the assessment, provides more personalized service support for individuals, and can also realize real-time personality assessment, greatly improving the assessment efficiency.

[0205] In some optional embodiments, the method further comprises:

[0206] The comprehensive scores are obtained according to the corresponding options of each question. The personality and psychological results with the closest scores are matched by the comprehensive scores to obtain the estimated user psychological type. Specifically, the scores of each question option on the answer sheet are counted and the closest personality and psychological category is matched as the estimated user psychological type, including:

[0207]

[0208] in, is the estimated user psychological type; M k is the kth type in the candidate set of psychological types; is the candidate set of psychological types; T is the comprehensive score; G k Psychological type M k Standard score of |TG k | is the absolute difference between the user's comprehensive score and the candidate's psychological type score;

[0209] The calculation formula of the comprehensive score T is:

[0210]

[0211] Where n is the total number of questions; m is the number of options for each question.

[0212] Compare the estimated user psychographic types and the final personality psychological prediction result M, and correct the estimated user psychological type by the final personality psychological prediction result M

[0213] According to this embodiment, a preliminary estimated user psychological type can be obtained through the comprehensive score matching of the questionnaire data, which provides a quick reference for the user and helps to have a general understanding of the user before the final evaluation result comes out. Then, by comparing the difference between the estimated psychological type and the final personality psychology prediction result, the difference between the two can be found, and the estimated type is corrected using the final prediction result, which can further improve the accuracy of the evaluation and make the result more accurate. In practical applications, sometimes it is necessary to quickly give a preliminary evaluation result, and at the same time, it is desired to ensure the accuracy of the final result. This embodiment provides such flexibility, which can quickly give the estimated result through the questionnaire data, and obtain a more accurate final evaluation result through multimodal data fusion.

[0214] Figure 3 This is a unit module diagram of a multimodal user personality psychology prediction device provided by an embodiment of the present disclosure. Figure 2 and 3 As shown, a user personality psychology prediction device 200 based on multimodality includes:

[0215] The data statistics module 210 is used to collect the personality psychology answer sheet data of the user and to count the scores corresponding to each question option to obtain the total score related to each personality psychology category;

[0216] The face analysis module 220 is used to collect user face image data to perform personality type prediction analysis and obtain personality psychological classification results;

[0217] The emotion prediction module 230 collects the user's voice data and physiological data to predict the user's emotions, obtains the emotion prediction results of the voice and physiological data, and then calculates the contribution value of each personality psychological category according to the emotion prediction results of the voice and physiological data; wherein the physiological data includes the user's skin electricity, breathing frequency and pupil data;

[0218] The psychological judgment module 240 is used to determine the final personality psychology prediction result according to the total scores related to each personality psychology category, the personality psychology classification result and the emotion prediction result of the voice and physiological data, including:

[0219] According to the emotion prediction results of the voice and physiological data, the contribution value of each personality psychology category is calculated; the total score related to each personality psychology category, the personality psychology classification result and the contribution value set of each personality psychology category are subjected to nonlinear weighted judgment to obtain the final personality psychology prediction result.

[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0221] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0222] Figure 4 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0223] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a ROM 302 or a computer program loaded from a storage unit 308 into a RAM 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0224] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0225] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as the user personality psychological prediction method based on multimodality. For example, in some embodiments, the user personality psychological prediction method based on multimodality may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the user personality psychological prediction method based on multimodality described above may be executed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the described method in any other appropriate manner (for example, by means of firmware).

[0226] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0227] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0228] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0229] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0230] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0231] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0232] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0233] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A user personality psychology prediction method based on multimodality, characterized in that: The following steps are involved: Collect the user's personality psychology answer sheet data and count the scores corresponding to each question option to obtain the total score related to each personality psychology category; Collect user face image data to perform personality type prediction and analysis to obtain personality psychological classification results; Collecting the user's voice data and physiological data for emotion prediction, and obtaining the emotion prediction results of the voice and physiological data; wherein the physiological data includes the user's skin electricity, breathing frequency and pupil data; Determining a final personality psychology prediction result according to the total scores related to each personality psychology category, the personality psychology classification result and the emotion prediction result of the voice and physiological data includes: Calculating the contribution value of each personality psychological category based on the emotion prediction results of the voice and physiological data; The total scores related to the personality psychology categories, the personality psychology classification results and the contribution value sets of the personality psychology categories are subjected to nonlinear weighted judgment to obtain the final personality psychology prediction result.

2. The method according to claim 1, characterized in that in, The method further comprises: The comprehensive score is obtained based on the scores corresponding to each question option. The estimated user psychological type is obtained by matching the comprehensive score with the personality psychological result with the closest score. The estimated user psychological type and the final personality psychology prediction result are compared for differences, and the estimated user psychological type is corrected according to the final personality psychology prediction result.

3. The method according to claim 1, characterized in that in, The collecting of user face image data to perform personality type prediction analysis to obtain personality psychological classification results includes: Preprocessing the face image data; extracting features from the preprocessed face image data; wherein the feature extraction includes preliminary extraction and secondary extraction, wherein the preliminary extraction includes performing a deep separable convolution operation and a preliminary batch normalization transformation on the preprocessed face image data to obtain preliminary extracted image features; the secondary extraction includes performing a deep separable convolution operation on the preliminary extracted image features, and using an activation function to enhance feature representation, and finally fusing the preliminary extracted image features with the features processed by the deep separable convolution operation through a residual connection to obtain secondary extracted image features; based on the secondary extracted image features, using a classification head to perform personality type prediction analysis to obtain a personality psychology classification result.

4. The method according to claim 3, characterized in that in, The model used for personality type prediction analysis based on facial data is a deep learning convolutional neural network, and the loss function of the model includes the weighted sum of target confidence loss, non-target confidence loss and category loss.

5. The method according to claim 4, characterized in that in, The loss function is specifically: L total =λ obj L obj +λ noobj L noobj +λ class L class Where L obj , L noobj and L class Represent target confidence loss, non-target confidence loss and category loss respectively, λ obj’ , noobj and λ class It is the weight coefficient corresponding to each part, which is used to balance the impact of various losses.

6. The method according to claim 5, characterized in that The training process of the convolutional neural network model includes: collecting labeled face image data as a training set, inputting the model for forward propagation to calculate the loss, updating the model parameters through the back propagation algorithm, and iterating multiple times until the loss converges.

7. The method according to claim 1, characterized in that in, Collect the user's voice data and physiological data for emotion prediction, and obtain the emotion prediction results of the voice and physiological data, including: Preprocessing the voice data and physiological data; Extract the acoustic and language features of the preprocessed speech data, including pitch, volume, timbre, speaking speed, and vocabulary usage; perform unsupervised learning on the user emotion classification based on the extracted acoustic and language feature vectors to obtain the emotion results predicted by the speech data; Extract the features of preprocessed physiological data, including skin conductance level, respiratory variability, pupil diameter change rate, and blinking times; construct a physiological data emotion prediction model to process the time series feature data of physiological data and obtain the emotion results predicted by physiological data; The emotion result predicted by the speech data is integrated with the emotion result predicted by the physiological data to obtain the emotion prediction results of the speech and physiological data.

8. The method according to claim 7, characterized in that in, The user emotion classification is performed by unsupervised learning based on the extracted acoustic and language feature vectors to obtain the emotion result predicted by the speech data, including: Calculate the Euclidean distance between feature vectors of different speech data; Determine a core point according to the Euclidean distance and a predetermined distance threshold and a minimum number of neighborhood points; The acoustic and language feature vectors are clustered and divided according to the core points to obtain the emotion results predicted by the speech data; wherein each cluster in the emotion results predicted by the speech data represents an emotion category or combination.

9. The method according to claim 1, characterized in that: in, The total scores related to each personality psychology category, the personality psychology classification results and the contribution value set of each personality psychology category are subjected to nonlinear weighted judgment to obtain the final personality psychology prediction result, including: According to the emotion prediction results of the voice and physiological data and the correlation between the preset emotional state and personality psychology type, the contribution value of each personality psychology category of the user is calculated in turn to obtain a contribution value set of the user's personality psychology category; According to the total scores related to each personality psychological category, the probability of each personality psychological category and the contribution value set of the personality psychological category of the user, a nonlinear weighted decision formula is adopted to obtain the predicted value M of the personality psychological category k by adjusting the weight index and the weighting coefficient. k : Among them, M k is the predicted value of personality psychological category k; γ, δ and τ are the exponents for adjusting weights to control nonlinear amplification; α, β and ρ are weighting coefficients used to balance the influence of prediction probability and answer sheet score, satisfying α+β+ρ=1; for each personality psychological category k, the total score vector related to each personality psychological category calculated by statistical answer sheet score is Score=[Score1,Score2,...,Score K ], where Score1, Score2, ..., Score K are the total scores corresponding to personality psychological categories 1, 2, ..., K, Score k is the total score corresponding to personality psychological category k; the classification probability distribution of personality psychological categories output by the model is P = [P1, P2, ..., P K ], where P1, P2, ..., P K are the predicted probabilities of personality categories 1, 2, ..., K, P k is the predicted probability of personality psychological category k, and The contribution value set of each personality psychological category of the user in, are the contribution values ​​of personality psychological categories 1, 2, …, K respectively; Among them, w i，k is the importance weight of the i-th question to the k-th personality psychological category, n is the total number of questions, S i is the score of the ith question; After calculating the predicted values ​​of all personality psychological categories of the user, M1, M2, ..., M k , ..., M K After that, by selecting the k k is used to determine the final personality psychological prediction result M:

10. A user personality psychology prediction device based on multimodality, characterized in that: include: The data statistics module is used to collect the user's personality psychology answer data and count the scores corresponding to each question option to obtain the total score related to each personality psychology category; The face analysis module is used to collect user face image data for personality type prediction and analysis to obtain personality psychological classification results; An emotion prediction module collects the user's voice data and physiological data for emotion prediction, obtains the emotion prediction results of the voice and physiological data, and then calculates the contribution value of each personality psychological category according to the emotion prediction results of the voice and physiological data; wherein the physiological data includes the user's skin electricity, breathing frequency and pupil data; The psychological judgment module is used to determine the final personality psychology prediction result according to the total scores related to each personality psychology category, the personality psychology classification result and the emotion prediction result of the voice and physiological data, including: According to the emotion prediction results of the voice and physiological data, the contribution value of each personality psychology category is calculated; the total score related to each personality psychology category, the personality psychology classification result and the contribution value set of each personality psychology category are subjected to nonlinear weighted judgment to obtain the final personality psychology prediction result.

Images