Multi-line text MBTI personality assessment system and method based on multi-modal large model
By fusing text, speech, and image data into a multimodal large model, and combining cross-modal adversarial learning and personalized reinforcement learning, the problem of insufficient full utilization of multimodal information and inadequate personalized adjustment in the MBTI personality assessment is solved, achieving a more accurate and personalized personality assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU YUNTONG LIANDA GOLDEN CLOTHING TECH CO LTD
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-12
AI Technical Summary
Existing MBTI personality assessment methods rely on single-modal data, fail to fully explore multimodal information, and lack personalized adjustment mechanisms, resulting in insufficient accuracy and personalization of personality assessments.
We employ a multimodal large model, combining cross-modal adversarial learning and personalized reinforcement learning, and integrate text, speech, and image data to conduct personality assessment through adaptive optimization strategies.
It improves the accuracy and personalization capabilities of personality assessments, enabling it to provide accurate MBTI personality evaluations in different situations and adapt to users' long-term behavioral changes.
Smart Images
Figure CN122196777A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of personality assessment, and in particular to a multi-line text MBTI personality assessment system and method based on a multimodal large model. Background Technology
[0002] In modern psychological research, the MBTI personality assessment, as a widely used psychological tool, has been extensively applied in areas such as personal development, career planning, and team building. Traditional MBTI personality assessments primarily rely on self-report questionnaires, where subjects determine their personality type by answering a series of questions. While this method can provide some personality assessment in the short term, it also has limitations. First, the design of traditional questionnaires often depends on the subject's self-awareness and self-description, which may be influenced by factors such as social expectations and emotional fluctuations, thus affecting the accuracy of the results. Second, the questionnaire's format makes it difficult to comprehensively capture an individual's behavioral characteristics in different scenarios and cannot effectively integrate multiple modal data such as voice, text, and images from real life.
[0003] With the rapid development of artificial intelligence technology, especially breakthroughs in natural language processing, speech recognition, and computer vision, traditional questionnaire-based MBTI personality assessment methods are gradually being replaced by new intelligent assessment methods. Deep learning-based multimodal analysis methods, by fusing different types of data (such as text, speech, and images), can more comprehensively and accurately analyze an individual's behavioral performance and personality traits, providing a more precise and personalized MBTI personality assessment than traditional questionnaires. However, existing multimodal personality assessment methods still have some significant shortcomings.
[0004] First, existing methods for fusing multimodal data are still immature. Although some studies have attempted to extract features from multimodal data using deep learning models, these methods often rely solely on feature extraction from a single modality, neglecting the interaction information between different modalities. In many cases, the interrelationship information between the three modalities of speech, text, and image has not been fully explored and utilized, resulting in limited accuracy and reliability of personality assessment models. In particular, the issue of feature alignment and information interaction across modalities remains a major challenge in multimodal learning.
[0005] Secondly, existing multimodal personality assessment methods mostly employ static classification models, meaning that all prediction parameters are fixed during the training phase, and they typically lack personalized adjustment mechanisms. While this approach can classify personality to some extent, it struggles to adaptively adjust based on long-term behavioral changes and feedback. Therefore, personality predictions for users may exhibit significant deviations at different times and in different contexts.
[0006] To address the shortcomings of existing technologies, this invention proposes a multi-line text MBTI personality assessment system and method based on a multimodal large model. Employing advanced multimodal deep learning technology, combined with cross-modal adversarial learning and personalized reinforcement learning, it can effectively improve the accuracy and personalized adjustment capabilities of personality assessment. By utilizing the interactive information of multimodal data such as text, voice, and images, and combining it with adaptive optimization strategies, this invention can provide more accurate MBTI personality assessments that are more in line with user characteristics in different periods and contexts. Summary of the Invention
[0007] One objective of this invention is to propose a multi-line text MBTI personality assessment system and method based on a multimodal large model. This invention can provide an efficient and scientific optimization scheme in MBTI personality testing, bringing significant technical value and economic benefits to practical applications.
[0008] According to an embodiment of the present invention, a multi-line text MBTI personality assessment system and method based on a multimodal large model includes the following steps: S1. Collect multimodal data input by the user, including text data, voice data, and image data; S2. Preprocess the text data and use the pre-trained language model BERT to generate text feature vectors; S3. Extract features from the speech data and convert the speech data into speech feature vectors; S4. Extract features from the image data and use the ViT model to extract image feature vectors; S5. Employ cross-modal adversarial learning to align text, speech, and image features, mapping different modal features to a unified feature space, using an attention mechanism to calculate the weights of different modal features, and obtaining the final fused feature vector through weighted summation. S6. Based on the final fused feature vector, the MBTI personality classification model is used to obtain the predicted MBTI personality type; S7. Personalize the predicted MBTI personality type, use self-supervised contrastive learning to optimize the MBTI prediction model, and obtain the final MBTI prediction type after personalized optimization. S8. Generate an assessment report, display the MBTI personality classification results, and provide personalized analysis and suggestions based on the assessment results.
[0009] Optionally, S1 includes the following steps: S11. Collect text data input by the user. The number of characters in the text data is L, and it satisfies the following conditions: , This represents the minimum character limit for text data. These represent the maximum character limit for text data; S12. Collect user-input voice data and convert it into text using automatic speech recognition technology. The duration of the voice data is T, satisfying... , The minimum duration of voice data, This indicates the maximum duration of the speech data, which is stored as text after being converted to it. ,in, This represents the i-th word, and n is the total number of words in the text; S13. Acquire image data input by the user, and set the resolution of the image data to [resolution setting missing]. H represents the height of the image, and W represents the width of the image. Image feature data is extracted using a face detection model. ; S14. Synchronize and timestamp the collected text data, voice data, and image data.
[0010] Optionally, S2 includes the following steps: S21. Regarding the collected text data and text data after speech conversion Perform word segmentation processing, and set the segmented text sequence as follows: ,in, This represents the i-th word, and n is the total number of words in the text sequence; S22. Perform stop word removal on the segmented text data. The stop word-removed text sequence is represented as follows: ; S23. Perform part-of-speech tagging and syntactic dependency analysis on the text data, and set the part-of-speech tag set as follows: ,in, For corresponding words The part-of-speech tags and syntactic dependency relations are represented using dependency trees, with the dependency tree structure set as follows: Where N is the set of word nodes in the text, E is the set of edges representing dependency relationships, and each edge... Connect word nodes and ; S24. Perform sentiment analysis on the text data and calculate the text sentiment score. Set the set of emotion category tags as Where k is the number of emotion categories; S25. Generating text feature vectors based on pre-trained language models : ; in, For the pre-trained model BERT; S26. Text feature vectors The text feature vectors are standardized by means normalization. .
[0011] Optionally, S3 includes the following steps: S31. Preprocess the collected voice data, converting the original voice signal into... Converted to spectral representation via short-time Fourier transform. : ; in, This represents the nth sampling point in the time series, where N represents the Fourier transform window length, f represents the frequency, and t represents the time parameter. S32. Extract features from the speech signal. The extracted features include: Mel frequency cepstral coefficients, pitch features, and speech rate features. S33. Perform dimensionality reduction on the speech feature vector to obtain the dimensionality-reduced speech feature vector. .
[0012] Optionally, S4 includes the following steps: S41. Preprocess the acquired image data, setting the input image data as... The image resolution is normalized using bilinear interpolation. S42. Use a ViT-based face detection algorithm to extract facial region features. ; S43. Perform key point detection on the facial region in the image to obtain a set of key points. ,in, Let i be the coordinates of the i-th key point. The number of key points detected; S44. Perform facial expression recognition on the facial regions in the image, and set the expression category set as follows: Where m represents the number of expression categories, and the expression feature vector is calculated. : ; Where f represents the expression classification model based on the Swing Transformer; S45. The extracted facial expression feature vector Dimensionality reduction is performed using a variational autoencoder for feature compression to obtain the final image feature vector. .
[0013] Optionally, S5 includes the following steps: S51. Regarding the obtained text feature vector Speech feature vectors Image feature vectors A cross-modal adversarial learning method is employed for feature alignment, and adversarial loss is used to optimize semantic consistency between different modalities. The cross-modal transformation function for text, speech, and image features is set as follows: Calculate the cross-modal embedding vectors respectively: ; ; ; in, Adversarial noise generated for cross-modal adversarial learning is minimized by reducing the cross-modal adversarial loss. This allows features from different modalities to be aligned in a shared embedding space. ; Where D is the adversarial discriminator, used to distinguish features of different modalities; S52. A Transformer-based dynamic alignment mechanism is used to calculate the relationships between multimodal features. The self-attention calculation method for text, speech, and image features is set as follows: ; ; in, , , Let d be the weight matrix, and d be the dimension of the feature vectors. This is the self-attention-weighted multimodal feature vector; S53. Calculate the importance weights of each modal feature using a mode weight adjustment method based on a gating mechanism: ; ; ; in, This is the weight matrix. Use the Sigmoid activation function; S54. Calculate the final fused feature vector using a weighted summation method. : ; in, This is the text feature vector after self-attention weighting. This is the speech feature vector after self-attention weighting. This is the image feature vector after self-attention weighting.
[0014] Optionally, S6 includes the following steps: S61. The final fused feature vector As input, MBTI is evaluated using a Transformer-based classification model: ; in, For trainable weights of fully connected layers, For bias terms, This indicates that the sample belongs to the MBTI personality category. The probability of; S62. Predicting MBTI Personality Categories Based on Maximum Probability Decision Rules: ; in, This is a prediction of the MBTI personality type.
[0015] Optionally, S7 includes the following steps: S71. Regarding the predicted MBTI personality categories Personalized adjustments are made, and a personalized adjustment model G is set based on user historical data. Optimize and calculate the adjusted personality prediction probability. : ; Where α is the personalized weighting factor, This includes the statistical distribution of users' historical text, voice, and image information, as well as historical evaluation results; S72. Revise the evaluation results and set personalized weights for each user. Calculate personalized adjustment loss : ; Where σ is the Sigmoid activation function, and D is a pair of historical user evaluation data, which is updated through gradient descent; S73. Optimize the MBTI prediction model using self-supervised contrastive learning, and define the positive sample set. and negative sample set Calculate the contrastive learning loss : ; in, The cosine similarity function is used. The feature vector representing a positive sample. The feature vector representing the negative sample is used for contrastive learning optimization, which makes the features of samples with similar personality categories more similar and samples of different categories more distinguishable. S74. Personalize the model and set up a user-specific memory matrix. The update method is as follows: ; in, To update the weights for memory, Stores the long-term distribution of users' individual characteristics; S75, Predicted probability based on personalized optimization Recalculate the final MBTI predicted categories: ; in, This is the final MBTI prediction type after personalized optimization.
[0016] The beneficial effects of this invention are: This invention employs a multimodal data fusion strategy, overcoming the limitations of existing personality assessment methods that can only rely on single-modal data. By integrating multimodal information such as text, voice, and images, this invention can comprehensively analyze an individual's behavioral characteristics from multiple perspectives, obtaining a more accurate and comprehensive personality assessment than traditional questionnaires. This cross-modal feature alignment and information interaction mechanism effectively improves the reliability and accuracy of personality assessment results, avoiding the bias that may be introduced by traditional methods that rely solely on self-reporting.
[0017] This invention introduces an optimization strategy based on cross-modal adversarial learning, which greatly enhances the system's personalized adjustment capabilities. Traditional personality assessment methods often cannot dynamically optimize based on users' long-term behavior and feedback, resulting in limited accuracy in personality assessment. However, this invention utilizes users' historical data and behavioral feedback, combined with a deep reinforcement learning model, enabling the personality assessment system to continuously adapt to user changes, thereby achieving more personalized personality predictions. By introducing an adaptive optimization mechanism, the system can not only classify personalities based on different user characteristics, but also dynamically adjust the prediction results during long-term interaction, optimizing the accuracy of personality assessment.
[0018] The multimodal large-scale model personality assessment method of this invention can make full use of the advantages of modern deep learning technology to process large-scale, high-dimensional multimodal data, providing richer information sources for personality prediction. This not only improves the accuracy of personality assessment, but also provides a more solid data foundation for application scenarios such as personalized recommendation and psychological counseling. Attached Figure Description
[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1This is a flowchart of a multi-line text MBTI personality assessment system and method based on a multimodal large model proposed in this invention; Figure 2 This is a flowchart of cross-modal adversarial learning feature alignment in a multi-line text MBTI personality assessment system and method based on a multimodal large model proposed in this invention. Detailed Implementation
[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0021] refer to Figures 1-2 A multi-line text MBTI personality assessment system and method based on a multimodal large model includes the following steps: S1. Collect multimodal data input by the user, including text data, voice data, and image data; S2. Preprocess the text data and use the pre-trained language model BERT to generate text feature vectors; S3. Extract features from the speech data and convert the speech data into speech feature vectors; S4. Extract features from the image data and use the ViT model to extract image feature vectors; S5. Employ cross-modal adversarial learning to align text, speech, and image features, mapping different modal features to a unified feature space, using an attention mechanism to calculate the weights of different modal features, and obtaining the final fused feature vector through weighted summation. S6. Based on the final fused feature vector, the MBTI personality classification model is used to obtain the predicted MBTI personality type; S7. Personalize the predicted MBTI personality type, use self-supervised contrastive learning to optimize the MBTI prediction model, and obtain the final MBTI prediction type after personalized optimization. S8. Generate an assessment report, display the MBTI personality classification results, and provide personalized analysis and suggestions based on the assessment results.
[0022] In this embodiment, S1 includes the following steps: S11. Collect text data input by the user. The number of characters in the text data is L, and it satisfies the following conditions: , This represents the minimum character limit for text data. These represent the maximum character limit for text data; S12. Collect user-input voice data and convert it into text using automatic speech recognition technology. The duration of the voice data is T, satisfying... , The minimum duration of voice data, This indicates the maximum duration of the speech data, which is stored as text after being converted to it. ,in, This represents the i-th word, and n is the total number of words in the text; S13. Acquire image data input by the user, and set the resolution of the image data to [resolution setting missing]. H represents the height of the image, and W represents the width of the image. Image feature data is extracted using a face detection model. ; S14. Synchronize and timestamp the collected text data, voice data, and image data.
[0023] In this embodiment, S2 includes the following steps: S21. Regarding the collected text data and text data after speech conversion Perform word segmentation processing, and set the segmented text sequence as follows: ,in, This represents the i-th word, and n is the total number of words in the text sequence; S22. Perform stop word removal on the segmented text data. The stop word-removed text sequence is represented as follows: ; S23. Perform part-of-speech tagging and syntactic dependency analysis on the text data, and set the part-of-speech tag set as follows: ,in, For corresponding words The part-of-speech tags and syntactic dependency relations are represented using dependency trees, with the dependency tree structure set as follows: Where N is the set of word nodes in the text, E is the set of edges representing dependency relationships, and each edge... Connect word nodes and ; S24. Perform sentiment analysis on the text data and calculate the text sentiment score. Set the set of emotion category tags as Where k is the number of emotion categories; S25. Generating text feature vectors based on pre-trained language models : ; in, For the pre-trained model BERT; S26. Text feature vectors The text feature vectors are standardized by means normalization. .
[0024] In this embodiment, S3 includes the following steps: S31. Preprocess the collected voice data, converting the original voice signal into... Converted to spectral representation via short-time Fourier transform. : ; in, This represents the nth sampling point in the time series, where N represents the Fourier transform window length, f represents the frequency, and t represents the time parameter. S32. Extract features from the speech signal. The extracted features include: Mel frequency cepstral coefficients, pitch features, and speech rate features. S33. Perform dimensionality reduction on the speech feature vector to obtain the dimensionality-reduced speech feature vector. .
[0025] In this embodiment, S4 includes the following steps: S41. Preprocess the acquired image data, setting the input image data as... The image resolution is normalized using bilinear interpolation. S42. Use a ViT-based face detection algorithm to extract facial region features. ; S43. Perform key point detection on the facial region in the image to obtain a set of key points. ,in, Let i be the coordinates of the i-th key point. The number of key points detected; S44. Perform facial expression recognition on the facial regions in the image, and set the expression category set as follows: Where m represents the number of expression categories, and the expression feature vector is calculated. : ; Where f represents the expression classification model based on the Swing Transformer; S45. The extracted facial expression feature vector Dimensionality reduction is performed using a variational autoencoder for feature compression to obtain the final image feature vector. .
[0026] In this embodiment, S5 includes the following steps: S51. Regarding the obtained text feature vector Speech feature vectors Image feature vectors A cross-modal adversarial learning method is employed for feature alignment, and adversarial loss is used to optimize semantic consistency between different modalities. The cross-modal transformation function for text, speech, and image features is set as follows: Calculate the cross-modal embedding vectors respectively: ; ; ; in, Adversarial noise generated for cross-modal adversarial learning is minimized by reducing the cross-modal adversarial loss. This allows features from different modalities to be aligned in a shared embedding space. ; Where D is the adversarial discriminator, used to distinguish features of different modalities; S52. A Transformer-based dynamic alignment mechanism is used to calculate the relationships between multimodal features. The self-attention calculation method for text, speech, and image features is set as follows: ; ; in, , , Let d be the weight matrix, and d be the dimension of the feature vectors. This is the self-attention-weighted multimodal feature vector; S53. Calculate the importance weights of each modal feature using a mode weight adjustment method based on a gating mechanism: ; ; ; in, This is the weight matrix. Use the Sigmoid activation function; S54. Calculate the final fused feature vector using a weighted summation method. : ; in, This is the text feature vector after self-attention weighting. This is the speech feature vector after self-attention weighting. This is the image feature vector after self-attention weighting.
[0027] In this embodiment, S6 includes the following steps: S61. The final fused feature vector As input, MBTI is evaluated using a Transformer-based classification model: ; in, For trainable weights of fully connected layers, For bias terms, This indicates that the sample belongs to the MBTI personality category. The probability of; S62. Predicting MBTI Personality Categories Based on Maximum Probability Decision Rules: ; in, This is a prediction of the MBTI personality type.
[0028] In this embodiment, S7 includes the following steps: S71. Regarding the predicted MBTI personality categories Personalized adjustments are made, and a personalized adjustment model G is set based on user historical data. Optimize and calculate the adjusted personality prediction probability. : ; Where α is the personalized weighting factor, This includes the statistical distribution of users' historical text, voice, and image information, as well as historical evaluation results; S72. Revise the evaluation results and set personalized weights for each user. Calculate personalized adjustment loss : ; Where σ is the Sigmoid activation function, and D is a pair of historical user evaluation data, which is updated through gradient descent; S73. Optimize the MBTI prediction model using self-supervised contrastive learning, and define the positive sample set. and negative sample set Calculate the contrastive learning loss : ; in, The cosine similarity function is used. The feature vector representing a positive sample. The feature vector representing the negative sample is used for contrastive learning optimization, which makes the features of samples with similar personality categories more similar and samples of different categories more distinguishable. S74. Personalize the model and set up a user-specific memory matrix. The update method is as follows: ; in, To update the weights for memory, Stores the long-term distribution of users' individual characteristics; S75, Predicted probability based on personalized optimization Recalculate the final MBTI predicted categories: ; in, This is the final MBTI prediction type after personalized optimization.
[0029] Example: In the modern workplace, the MBTI personality assessment is widely used in areas such as personal development, team building, and recruitment. Many companies and organizations hope to use MBTI assessments to help employees and candidates understand their personality types, thereby providing personalized career development advice or helping them choose the most suitable positions. However, traditional MBTI assessment methods typically rely on self-report questionnaires, which have certain limitations, especially when users lack proactive feedback on the test results, are emotionally unstable during the testing process, or have social expectation biases, which can easily lead to inaccurate assessment results. To better address this issue, this embodiment uses an MBTI personality assessment system based on a multimodal large model, combining users' text, voice, image, and other multimodal data, as well as personalized deep reinforcement learning models, to provide a more accurate and personalized personality assessment solution.
[0030] In one example, at a company, a candidate participated in a mock interview. During the interview, the candidate provided answers to a set of questions via video and audio devices, explaining the questions using both text and voice. Furthermore, the candidate's facial expressions and emotional tone were recorded during the interview. This data included text (such as the written content of the answers), voice (such as emotional expression and pauses in the voice), and images (such as facial expressions and eye contact).
[0031] The implementer uses the method of this invention to extract features from text, speech and image data through models such as BERT (for text analysis), Wav2Vec (for speech analysis) and Swin Transformer (for image analysis), and then fuses these features through a cross-modal adversarial learning model (CMA) to generate a comprehensive user-personalized feature vector.
[0032] Based on the fused multimodal feature vectors, the system of this invention predicts the MBTI personality of candidates and outputs a personalized personality assessment report based on the optimization results.
[0033] To verify the effectiveness and advantages of the method of the present invention, the implementers conducted a data experiment in a real recruitment scenario. The experiment involved 200 candidates, and personality assessments were conducted using both traditional questionnaires and multimodal large model-based evaluation methods. The specific experimental data are shown in Table 1 below: Table 1. Comparison of MBTI personality prediction results between the present invention and traditional methods.
[0034] Throughout the embodiments, the implementer not only solves the problems of poor accuracy and susceptibility to social expectation bias in traditional MBTI personality assessment methods through the method of the present invention, but also improves the ability to personalize adjustments, and achieves efficient personality prediction and recommendation based on multimodal data fusion.
[0035] This invention overcomes the limitations of traditional MBTI assessments, which rely solely on text questionnaires, by introducing multimodal data fusion technology. It combines multidimensional data with deep learning models for cross-modal adversarial learning, effectively improving the accuracy and diversity of personality assessment results. Compared to traditional assessment methods, this invention utilizes comprehensive features such as user behavioral data, emotional fluctuations, and facial expressions to make more accurate personality predictions and provide more personalized career recommendations.
[0036] Regarding personalized adjustments, this invention incorporates a deep learning model, enabling the system to adaptively optimize personality assessment results based on long-term feedback from user behavior data. This method not only overcomes the static and singular limitations of traditional methods but also allows for real-time adjustments to personality assessments, continuously adapting to changes in user behavior, thereby ensuring the sustained accuracy and personalized effectiveness of personality assessments.
[0037] This invention significantly improves the accuracy of personality assessments through multimodal fusion and reinforcement learning optimization, and provides career suggestions that highly match individual personality traits. Experimental data shows that compared to traditional questionnaire methods, the multimodal assessment system of this invention significantly improves prediction accuracy and personalized recommendations, effectively avoiding the biases and limitations of traditional assessment methods, and providing users with more accurate and practical personality analysis results.
[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A multi-line text MBTI personality assessment system and method based on a multimodal large model, characterized in that, Includes the following steps: S1. Collect multimodal data input by the user, including text data, voice data, and image data; S2. Preprocess the text data and use the pre-trained language model BERT to generate text feature vectors; S3. Extract features from the speech data and convert the speech data into speech feature vectors; S4. Extract features from the image data and use the ViT model to extract image feature vectors; S5. Employ cross-modal adversarial learning to align text, speech, and image features, mapping different modal features to a unified feature space, using an attention mechanism to calculate the weights of different modal features, and obtaining the final fused feature vector through weighted summation. S6. Based on the final fused feature vector, the MBTI personality classification model is used to obtain the predicted MBTI personality type; S7. Personalize the predicted MBTI personality type, use self-supervised contrastive learning to optimize the MBTI prediction model, and obtain the final MBTI prediction type after personalized optimization. S8. Generate an assessment report, display the MBTI personality classification results, and provide personalized analysis and suggestions based on the assessment results.
2. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S1 includes the following steps: S11. Collect text data input by the user. The number of characters in the text data is L, and it satisfies the following conditions: , This represents the minimum character limit for text data. These represent the maximum character limit for text data; S12. Collect user-input voice data and convert it into text using automatic speech recognition technology. The duration of the voice data is T, satisfying... , The minimum duration of voice data, This indicates the maximum duration of the speech data. The speech data is stored as text after being converted to text. ,in, This represents the i-th word, and n is the total number of words in the text; S13. Acquire image data input by the user, and set the resolution of the image data to [resolution setting missing]. H represents the height of the image, and W represents the width of the image. Image feature data is extracted using a face detection model. ; S14. Synchronize and timestamp the collected text data, voice data, and image data.
3. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S2 includes the following steps: S21. Regarding the collected text data and text data after speech conversion Perform word segmentation processing, and set the segmented text sequence as follows: ,in, This represents the i-th word, and n is the total number of words in the text sequence; S22. Perform stop word removal on the segmented text data. The stop word-removed text sequence is represented as follows: ; S23. Perform part-of-speech tagging and syntactic dependency analysis on the text data, and set the part-of-speech tag set as follows: ,in, For corresponding words The part-of-speech tags and syntactic dependency relations are represented using dependency trees, with the dependency tree structure set as follows: Where N is the set of word nodes in the text, E is the set of edges representing dependency relationships, and each edge... Connect word nodes and ; S24. Perform sentiment analysis on the text data and calculate the text sentiment score. Set the set of emotion category tags as Where k is the number of emotion categories; S25. Generating text feature vectors based on pre-trained language models : ； in, For the pre-trained model BERT; S26. Text feature vectors The text feature vectors are standardized by means normalization. .
4. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S3 includes the following steps: S31. Preprocess the collected voice data, converting the original voice signal into... Converted to spectral representation via short-time Fourier transform. : ； in, This represents the nth sampling point in the time series, where N represents the Fourier transform window length, f represents the frequency, and t represents the time parameter. S32. Extract features from the speech signal. The extracted features include: Mel frequency cepstral coefficients, pitch features, and speech rate features. S33. Perform dimensionality reduction on the speech feature vector to obtain the dimensionality-reduced speech feature vector. .
5. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S4 includes the following steps: S41. Preprocess the acquired image data, setting the input image data as... The image resolution is normalized using bilinear interpolation. S42. Use a ViT-based face detection algorithm to extract facial region features. ; S43. Perform key point detection on the facial region in the image to obtain a set of key points. ,in, Let i be the coordinates of the i-th key point. The number of key points detected; S44. Perform facial expression recognition on the facial regions in the image, and set the expression category set as follows: Where m represents the number of expression categories, and the expression feature vector is calculated. : ； Where f represents the expression classification model based on the Swing Transformer; S45. The extracted facial expression feature vector Dimensionality reduction is performed using a variational autoencoder for feature compression to obtain the final image feature vector. .
6. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S5 includes the following steps: S51. Regarding the obtained text feature vector Speech feature vectors Image feature vectors A cross-modal adversarial learning method is employed for feature alignment, and adversarial loss is used to optimize semantic consistency between different modalities. The cross-modal transformation function for text, speech, and image features is set as follows: Calculate the cross-modal embedding vectors respectively: ； ； ； in, Adversarial noise generated for cross-modal adversarial learning is minimized by reducing the cross-modal adversarial loss. This allows features from different modalities to be aligned in a shared embedding space: ； Where D is the adversarial discriminator, used to distinguish features of different modalities; S52. A Transformer-based dynamic alignment mechanism is used to calculate the relationships between multimodal features. The self-attention calculation method for text, speech, and image features is set as follows: ； ； in, , , Let d be the weight matrix, and d be the dimension of the feature vectors. This is the self-attention-weighted multimodal feature vector; S53. Calculate the importance weights of each modal feature using a mode weight adjustment method based on a gating mechanism: ； ； ； in, This is the weight matrix. Use the Sigmoid activation function; S54. Calculate the final fused feature vector using a weighted summation method. : ； in, This is the text feature vector after self-attention weighting. This is the speech feature vector after self-attention weighting. This is the image feature vector after self-attention weighting.
7. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S6 includes the following steps: S61. The final fused feature vector As input, MBTI is evaluated using a Transformer-based classification model: ； in, For trainable weights of fully connected layers, For bias terms, This indicates that the sample belongs to the MBTI personality category. The probability of; S62. Predicting MBTI Personality Categories Based on Maximum Probability Decision Rules: ； in, This is a prediction of the MBTI personality type.
8. The MBTI personality assessment system and method based on a multimodal large model according to claim 1, characterized in that, S7 includes the following steps: S71. Regarding the predicted MBTI personality categories Personalized adjustments are made, and a personalized adjustment model G is set based on user historical data. Optimize and calculate the adjusted personality prediction probability. : ； Where α is the personalized weighting factor, This includes the statistical distribution of users' historical text, voice, and image information, as well as historical evaluation results; S72. Revise the evaluation results and set personalized weights for each user. Calculate personalized adjustment loss : ； Where σ is the Sigmoid activation function, and D is a pair of historical user evaluation data, which is updated through gradient descent; S73. Optimize the MBTI prediction model using self-supervised contrastive learning, and define the positive sample set. and negative sample set Calculate the contrastive learning loss : ； in, The cosine similarity function is used. The feature vector representing a positive sample. The feature vector representing the negative sample is used for contrastive learning optimization, which makes the features of samples with similar personality categories more similar and samples of different categories more distinguishable. S74. Personalize the model and set up a user-specific memory matrix. The update method is as follows: ； in, To update the weights for memory, Stores the long-term distribution of users' individual characteristics; S75, Predicted probability based on personalized optimization Recalculate the final MBTI predicted categories: ； in, This is the final MBTI prediction type after personalized optimization.