A method and system for intelligent psychological state assessment based on multimodal alignment

Abstract

This invention discloses an intelligent psychological state assessment method and system based on multimodal alignment, belonging to the field of psychological assessment technology. The method includes the following steps: S1: Constructing a training sample set for a psychological state assessment model; S2: Preprocessing video data, audio data, and text data in the training data sample set; S3: Utilizing multimodal cross-attention, aligning and fusing text data as the primary modality and video and audio data as auxiliary data for text data to obtain multimodal fusion features for psychological state analysis; S4: Training a psychological state analysis model using the aligned and fused multimodal fusion features; S5: Obtaining multidimensional psychological analysis results of the test subjects through the trained psychological state analysis model. The method of this invention extracts deep multimodal features of adolescents through pre-training a large model and improves the feature representation ability of the data using modal alignment technology.

Description

Technical Field

[0001] This invention relates to the field of psychological assessment technology, and in particular to an intelligent psychological state assessment method and system based on multimodal alignment. Background Technology

[0002] According to the "Report on the Development of National Mental Health in China (2021-2022)," 14.8% of adolescents are at risk of depression to varying degrees, with 4% experiencing severe depression. However, there is a severe shortage of mental health teachers and professionals in my country, and this serious imbalance between supply and demand has resulted in a significant lack of coverage for mental health screening and intervention among adolescents.

[0003] Against this backdrop, AI-based psychological state assessment methods are gradually becoming an important means of solving this problem. Traditional psychological assessment methods mainly rely on questionnaires and interviews, which are difficult to comprehensively and objectively reflect the psychological state of adolescents. However, multimodal data fusion technology, by comprehensively utilizing multiple data sources such as video, audio, and text, can more comprehensively capture and analyze the psychological characteristics of adolescents, achieving a more accurate psychological state assessment.

[0004] Traditional deep learning methods often employ small-scale models to extract features for each modality, resulting in limited feature representation capabilities. These models struggle to capture complex psychological states and high-level expressions. Furthermore, because they are trained on small-scale and domain-specific data, their generalization ability is poor, making it difficult to adapt to data from different scenarios and individuals, leading to unstable performance in real-world applications. In addition, current multimodal fusion methods, such as CN116936098A, typically involve simple feature concatenation or weighted averaging, failing to comprehensively capture the interactions and correspondences between different modalities, thus resulting in generally low accuracy.

[0005] In view of this, this invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing an intelligent psychological state assessment method and system based on multimodal alignment. This method extracts multimodal deep features of adolescents through a pre-trained large model, and utilizes modal alignment technology to improve the feature representation ability of the data, thereby enabling the analysis of adolescents' psychological states and meeting the requirements for accuracy and robustness in adolescent mental health assessment.

[0007] To achieve the above objectives, the present invention also employs the following technical solution:

[0008] A method for assessing intelligent mental states based on multimodal alignment includes the following steps:

[0009] S1: Collect video, audio, and text data from subjects using psychological testing and assessment tools to construct a training sample set for the psychological state assessment model.

[0010] S2: Preprocess the video, audio, and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video, audio, and text data.

[0011] S3: Utilizing multimodal cross-attention, text data is used as the primary modality, while video and audio data are used as auxiliary text data for alignment and fusion, resulting in multimodal fusion features for psychological state analysis.

[0012] S4: The psychological state analysis model is obtained by training the aligned and fused multimodal fusion features.

[0013] S5: Obtain multi-dimensional psychological analysis results of the test subjects through the psychological state analysis model obtained through training.

[0014] Furthermore, step S1 includes the following steps:

[0015] S11: Provides a variety of assessment tools, including psychological assessment questionnaires.

[0016] S12: Acquire multimodal data of the subjects, including video and audio data during the answering process, as well as text data extracted from the audio data.

[0017] S13: After completing the questions, calculate the score of the psychological assessment questionnaire as the psychological state label of the subject's multimodal data.

[0018] S14: Construct a training sample set for the psychological state assessment model based on the multimodal data in S12 and the psychological state labels in S13.

[0019] Furthermore, step S2 includes the following steps:

[0020] S21: Preprocessing of video data includes cropping video frames to focus on facial regions, adjusting image sizes to meet model input requirements, and normalizing images.

[0021] S22: Preprocessing of speech data includes removing background noise, normalizing the speech signal, and segmenting long audio files into shorter segments.

[0022] S23: Preprocessing of text data includes data cleaning to remove irrelevant characters, punctuation marks, and special symbols; using the jieba Chinese word segmentation tool to segment the text into words or phrases and remove stop words; converting all text to a uniform uppercase and lowercase form; and standardizing numbers, dates, etc.

[0023] Furthermore, step S2 also includes the following steps:

[0024] S24: Input the preprocessed video data from S21 into the EMO-AffectNetModel and use this pre-trained model to extract deep facial features.

[0025] S25: Input the preprocessed speech data from S22 into emotion2vec, and obtain deep speech features with a sampling rate of 50Hz through the output of the last layer of the model.

[0026] S26: Input the preprocessed text data from S23 into Piccolo2 and output the embedding vector of the subject's text expression.

[0027] Furthermore, step S3 includes the following steps:

[0028] S31: Global information is captured from video data, audio data, and text data through a multi-head self-attention mechanism; among them, a cross-attention mechanism is constructed based on the structure of the Transformer network.

[0029] S32: Construct a text-aligned video representation cross-attention mechanism and a text-aligned speech representation cross-attention mechanism.

[0030] S33: Construct a cross-attention mechanism for video and speech-aligned text representations to obtain multimodal fusion features. .

[0031] Furthermore, step S31 includes the following steps:

[0032] S311: Represent the single-modal vector as X, and generate the query (Q), key (K), and value (V) matrix of the modality data through a linear layer, in the following form:

[0033] ;

[0034] ;

[0035] ;

[0036] in, , and This is the weight matrix.

[0037] S312: Divide Q, K, and V into multiple heads, calculate the attention weights for each head independently, and then perform a weighted sum. The formula for calculating the attention weights for each head is as follows:

[0038] ;

[0039] in, is the dimension of the key vector, used to scale the dot product attention, and i is the label of the segmentation head.

[0040] S313: Concatenate the outputs of all heads and pass them through a linear layer to obtain the output of the multi-head self-attention mechanism:

[0041] ;

[0042] Where h is the number of segment heads. It is the output weight matrix.

[0043] S314: By performing a residual join, Z is added to the original single-mode vector X to obtain... .

[0044] S315: The application layer normalization yields the output of each modality after multi-head self-attention, calculated using the following formula:

[0045] ;

[0046] The outputs of video, audio, and text modalities are respectively represented as follows: , and .

[0047] Furthermore, step S32 includes the following steps:

[0048] S321: Based on video features As a query (Q), with text features As keys (K) and values ​​(V), they are obtained through multi-head self-attention mechanism, residual linking, and layer normalization. .

[0049] S322: Represent the aligned video The output is fed into the feedforward network FFN to obtain The format is as follows:

[0050] .

[0051] S323: Yes Residual connections and layer normalization are performed to obtain the final video representation aligned with text. .

[0052] S324: Based on speech features As a query (Q), with text features As keys (K) and values ​​(V), they are obtained through multi-head self-attention mechanism, residual linking, and layer normalization. .

[0053] S325: Align the speech representation The output is fed into the feedforward network FFN to obtain The format is as follows:

[0054] .

[0055] S326: Yes Residual connections and layer normalization are performed to obtain the final text-aligned speech representation. .

[0056] Furthermore, step S33 includes the following steps:

[0057] S331: Using text features As the overall query (Q), based on video features As a key in the video modality ( ) and value ( Text representations aligned with video are obtained through a multi-head attention layer. .

[0058] S332: Using text features As the total query (Q), with speech features As a key in the video modality ( ) and value ( Text representations aligned with video are obtained through a multi-head attention layer. .

[0059] S333: Will and The text representation is obtained through a fusion layer, and then input into a layer normalization layer, a feedforward network layer, and another layer normalization layer to obtain the final text representation aligned with video and speech. .

[0060] S334: The final method will utilize text-aligned video representation. Finally, text-aligned speech representation is used. And ultimately, text representations that utilize video and audio alignment. Multimodal fusion features for psychological state analysis are obtained through fusion layer fusion. .

[0061] Furthermore, step S5 includes the following steps:

[0062] S51: The mental state analysis model consists of two fully connected layers. The input is multimodal fusion features. The first layer uses the ReLU activation function, and the second fully connected layer uses the Softmax activation function. The final output is the mental state analysis result. The calculation formula of the mental state analysis model is as follows:

[0063] ;

[0064] in, These are the weight matrix and bias vector of the first fully connected layer, respectively. These are the weight matrix and bias vector of the second fully connected layer, respectively.

[0065] The model's loss function is the cross-entropy loss function used in classification tasks, as shown in the following formula:

[0066] ,

[0067] Where c is the total number of categories, This represents the true label of the i-th class. This represents the predicted probability of the i-th class; during training, the model parameters are updated using the backpropagation algorithm. , To minimize the loss function L.

[0068] To achieve the above objectives, the present invention also employs the following technical solution:

[0069] A multimodal alignment-based intelligent psychological state assessment system includes:

[0070] The data acquisition module is used to collect video, audio, and text data from subjects using psychological testing and assessment tools to build a training sample set for the psychological state assessment model.

[0071] The preprocessing module is used to preprocess video data, speech data, and text data in the training data sample set, and to extract deep features from the preprocessed video data, speech data, and text data using a pre-trained large model.

[0072] The alignment and fusion module utilizes multimodal cross-attention to align and fuse text data as the primary modality, with video and audio data as auxiliary text data, to obtain multimodal fusion features for psychological state analysis.

[0073] The model training module is used to train a psychological state analysis model using aligned and fused multimodal fusion features.

[0074] The psychological analysis module is used to obtain multi-dimensional psychological analysis results of test subjects through a trained psychological state analysis model.

[0075] Compared with the prior art, the beneficial effects of this invention are as follows:

[0076] 1. Utilize large models to extract deep features from video, audio, and text data. By pre-training pre-trained models for each modality on a large amount of data in the corresponding domain, more subtle and abstract psychological features can be captured. Furthermore, large-scale pre-trained models can improve the robustness of the evaluation process against noise and anomalous data, reducing the impact of these factors on the evaluation results.

[0077] 2. Using text data as the primary modality, and video and audio data as supplementary enhancements to the text representation modality, modality fusion technology aligns video, audio, and text data, accurately matching correspondences between different modalities, capturing correlations between them, and dynamically adjusting weights based on the importance of information from different modalities to achieve effective feature fusion. Compared to traditional multimodal fusion methods (such as simple feature concatenation or weighted averaging), dynamic weight allocation better utilizes information from each modality, avoids information loss or redundancy, and improves the accuracy of psychological state assessment. Attached Figure Description

[0078] Figure 1 This is a flowchart of an intelligent mental state assessment method based on multimodal alignment.

[0079] Figure 2 A schematic diagram illustrating the principle of cross-attention mechanism for text-aligned video representation.

[0080] Figure 3 A schematic diagram illustrating the principle of the cross-attention mechanism for aligning text with speech representation.

[0081] Figure 4 A schematic diagram illustrating the principle of cross-attention mechanism for aligning text representations of video and audio.

[0082] Figure 5 This is a framework diagram of an intelligent mental state assessment method based on multimodal alignment.

[0083] Figure 6 This is a structural diagram of an intelligent mental state assessment method based on multimodal alignment. Detailed Implementation

[0084] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0085] Example 1:

[0086] A method for intelligent mental state assessment based on multimodal alignment, such as Figure 1-5 As shown, it includes:

[0087] S1: Collect video, audio, and text data from subjects using psychological testing and assessment tools to construct a training sample set for the psychological state assessment model.

[0088] S2: Preprocess the video, audio, and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video, audio, and text data.

[0089] S3: Utilizing multimodal cross-attention, text data is used as the primary modality, while video and audio data are used as auxiliary text data for alignment and fusion, resulting in multimodal fusion features for psychological state analysis.

[0090] In this embodiment, in the video, audio and text data, the subjects will answer the reasons for choosing the answer in the psychological test. Therefore, the text data will contain key clues for the assessment of the individual's psychological state. Thus, in the process of multimodal cross-attention alignment and fusion, the text data is the primary modality, and the video data and audio data are used as auxiliary text data to enhance the text representation modality.

[0091] S4: The psychological state analysis model is obtained by training the aligned and fused multimodal fusion features.

[0092] In this embodiment, the main improvement is made to the method for obtaining multimodal fusion features. In addition, for the method of training the model, existing methods can be used to train the model and the model can be continuously optimized to obtain the optimal psychological state analysis model.

[0093] S5: Obtain multi-dimensional psychological analysis results of the test subjects through the psychological state analysis model obtained through training.

[0094] The method in this embodiment utilizes large models to extract deep features from video, audio, and text data. By pre-training pre-trained models for each modality on a large amount of data in the corresponding domain, it is possible to capture more subtle and abstract psychological features. Furthermore, large-scale pre-trained models can improve the robustness of the evaluation process in the face of noise and anomalous data, reducing the impact of these factors on the evaluation results.

[0095] The method in this embodiment aligns video, audio, and text data using modality fusion technology, accurately matching the correspondences between different modalities, capturing the correlations between different modalities, and dynamically adjusting weights based on the importance of information from different modalities to achieve effective feature fusion. Compared to traditional multimodal fusion methods (such as simple feature concatenation or weighted averaging), dynamic weight allocation can better utilize information from each modality, avoid information loss or redundancy, and improve the accuracy of psychological state assessment.

[0096] In an optional embodiment, step S1 includes the following steps:

[0097] S11: Provides a variety of assessment tools, including psychological assessment questionnaires.

[0098] In this step, subjects can fill out a psychological assessment questionnaire online. This embodiment provides subjects with three mental health assessment tools: the Mental Health Diagnostic Test for Adolescents (MHT), the PHQ-9 Health Questionnaire, and the Generalized Anxiety Disorder Scale (GAD-7). Subjects can choose one of them to answer the questions.

[0099] S12: Acquire multimodal data of the subjects, including video and audio data during the answering process, as well as text data extracted from the audio data.

[0100] In this step, multimodal data, including video, audio, and text data, are collected. The video data refers to facial videos of the subjects answering questions.

[0101] The camera and microphone can be activated to collect facial video and voice data from the subjects when they begin answering questions.

[0102] The Mental Health Diagnostic Test for Adolescents (MHT), the PHQ-9 Health Questionnaire, and the Generalized Anxiety Disorder Scale (GAD-7) are multiple-choice questionnaires. To collect both verbal and textual data from the participants, this embodiment prompts them to describe their reasons for choosing answers, as well as their feelings and experiences, after completing the questions. Simultaneously, facial video and verbal data are collected. This embodiment combines multiple-choice psychological assessment questionnaires with open-ended questions, which helps to collect more emotional details and background information reflecting the participants' psychological state. Furthermore, allowing participants to share their unique experiences and feelings enhances the accuracy and personalization of the psychological state assessment.

[0103] S13: After completing the questions, calculate the score of the psychological assessment questionnaire as the psychological state label of the subject's multimodal data.

[0104] In this step, the scores of the psychological assessment questionnaires are calculated based on the subjects' questionnaire responses and the scoring criteria for each questionnaire.

[0105] Specifically, the Mental Health Diagnostic Test for Adolescents (MHT) scores the mental health status of primary and secondary school students in three levels: scores between 1 and 55 indicate normal mental health, scores between 56 and 64 indicate poor mental health, and scores above 65 indicate serious mental health problems. The PHQ-9 Health Questionnaire scores depression in five levels: scores between 0 and 4 indicate mild depression, scores between 5 and 9 indicate mild depression, scores between 10 and 14 indicate moderate depression, scores between 15 and 19 indicate moderate to severe depression, and scores between 20 and 27 indicate severe depression. The Generalized Anxiety Disorder Scale (GAD-7) scores anxiety in four levels: scores between 0 and 4 indicate mild anxiety, scores between 5 and 9 indicate mild anxiety, scores between 10 and 14 indicate moderate anxiety, and scores between 15 and 21 indicate severe anxiety.

[0106] S14: Construct a training sample set for the psychological state assessment model based on the multimodal data in S12 and the psychological state labels in S13.

[0107] In this optional embodiment, by combining selective psychological assessment questionnaires and open-ended questions, more emotional details and background information reflecting the psychological state of the subjects are collected, thereby enhancing the personalization and accuracy of the assessment.

[0108] In an optional embodiment, step S2 includes the following steps:

[0109] S21: Preprocessing of video data includes cropping video frames to focus on facial regions, adjusting image sizes to meet model input requirements, and normalizing images.

[0110] S22: Preprocessing of speech data includes removing background noise, normalizing the speech signal, and segmenting long audio files into shorter segments.

[0111] S23: Preprocessing of text data includes data cleaning to remove irrelevant characters, punctuation marks, and special symbols; using the jieba Chinese word segmentation tool to segment the text into words or phrases and remove stop words; converting all text to a uniform uppercase and lowercase form; and standardizing numbers, dates, etc.

[0112] In an optional embodiment, step S2 further includes the following steps:

[0113] S24: Input the preprocessed video data from S21 into the EMO-AffectNetModel and use this pre-trained model to extract deep facial features.

[0114] In this step, the open-source EMO-AffectNetModel is used to extract deep features from video modal data. This pre-trained model has been pre-trained on multiple facial expression recognition datasets and can capture subtle facial changes of subjects during psychological testing, thereby improving the ability to express deep facial features of subjects and enhancing the robustness of visual information in the psychological assessment process.

[0115] S25: Input the preprocessed speech data from S22 into emotion2vec, and obtain deep speech features with a sampling rate of 50Hz through the output of the last layer of the model.

[0116] In this step, the open-source speech emotion analysis pre-trained large model emotion2vec is used to extract deep representations of the subject's speech modality data. After being pre-trained and fine-tuned on a large-scale emotion dataset, emotion2vec has the ability to express highly general speech emotion features. It can capture subtle emotional changes and non-verbal cues such as tone, rhythm, and intensity from the speech data of the subject during the psychological assessment process, thereby improving the expressive ability of speech features in the psychological state assessment process.

[0117] S26: Input the preprocessed text data from S23 into Piccolo2 and output the embedding vector of the subject's text expression.

[0118] In this step, the Piccolo2 Chinese text embedding model is used to embed the text data. Piccolo2 is currently the best embedding model for Chinese text embedding, and its multi-granularity text expression capability can capture the meaning and contextual information of the text from different perspectives.

[0119] In an optional embodiment, step S3 includes the following steps:

[0120] S31: Global information is captured from video data, audio data, and text data through a multi-head self-attention mechanism; among them, a cross-attention mechanism is constructed based on the structure of the Transformer network.

[0121] In this step, a multi-head self-attention mechanism is used to capture global information for each single-modal data, enhancing the expressive and generalization capabilities of each modality. Each single-modal data represents a deep feature extracted by S2.

[0122] S32: Construct a text-aligned video representation cross-attention mechanism and a text-aligned speech representation cross-attention mechanism.

[0123] S33: Construct a cross-attention mechanism for video and speech-aligned text representations to obtain multimodal fusion features. .

[0124] In an optional embodiment, such as Figure 2-4 As shown, step S31 includes the following steps:

[0125] S311: Represent the single-modal vector as X, and generate the query (Q), key (K), and value (V) matrix of the modality data through a linear layer, in the following form:

[0126] ;

[0127] ;

[0128] ;

[0129] in, , and This is the weight matrix.

[0130] In this step, the single-modal vector X is the deep feature extracted by S2.

[0131] S312: Divide Q, K, and V into multiple heads, calculate the attention weights for each head independently, and then perform a weighted sum. The formula for calculating the attention weights for each head is as follows:

[0132] ;

[0133] in, is the dimension of the key vector, used to scale the dot product attention, and i is the label of the segmentation head.

[0134] In this step, i is the label of the segment head, that is, i represents the i-th segment head.

[0135] S313: Concatenate the outputs of all heads and pass them through a linear layer to obtain the output of the multi-head self-attention mechanism:

[0136] ;

[0137] Where h is the number of segment heads. It is the output weight matrix.

[0138] S314: By performing a residual join, Z is added to the original single-mode vector X to obtain... .

[0139] S315: The application layer normalization yields the output of each modality after multi-head self-attention, calculated using the following formula:

[0140] ;

[0141] The outputs of video, audio, and text modalities are respectively represented as follows: , and .

[0142] In this step, the single-modal vectors of video, audio, and text, represented as X, are obtained by performing steps S311-S315 respectively. , and This stabilizes the training process and improves model performance.

[0143] In an optional embodiment, such as Figure 2-3 As shown, step S32 includes the following steps:

[0144] S321: Based on video features As a query (Q), with text features As keys (K) and values ​​(V), they are obtained through multi-head self-attention mechanism, residual linking, and layer normalization. .

[0145] In this step, the process of multi-head self-attention mechanism, residual linking, and layer normalization is the same as the cross-attention calculation process in S31. The cross-attention mechanism is used to align the video features with the text features to obtain the desired result. .

[0146] S322: Represent the aligned video The output is fed into the feedforward network FFN to obtain The format is as follows:

[0147] .

[0148] S323: Yes Residual connections and layer normalization are performed to obtain the final video representation aligned with text. .

[0149] S324: Based on speech features As a query (Q), with text features As keys (K) and values ​​(V), they are obtained through multi-head self-attention mechanism, residual linking, and layer normalization. .

[0150] In this step, the process of multi-head self-attention mechanism, residual linking, and layer normalization is the same as the cross-attention calculation process in S31. The cross-attention mechanism is used to align the video features with the text features to obtain the desired result. .

[0151] S325: Align the speech representation The output is fed into the feedforward network FFN to obtain The format is as follows:

[0152] .

[0153] S326: Yes Residual connections and layer normalization are performed to obtain the final text-aligned speech representation. .

[0154] In an optional embodiment, such as Figure 4 As shown, step S33 includes the following steps:

[0155] S331: Using text features As the overall query (Q), based on video features As a key in the video modality ( ) and value ( Text representations aligned with video are obtained through a multi-head attention layer. .

[0156] S332: Using text features As the total query (Q), with speech features As a key in the video modality ( ) and value ( Text representations aligned with video are obtained through a multi-head attention layer. .

[0157] S333: Will and The text representation is obtained through a fusion layer, and then input into a layer normalization layer, a feedforward network layer, and another layer normalization layer to obtain the final text representation aligned with video and speech. .

[0158] S334: The final method will utilize text-aligned video representation. Finally, text-aligned speech representation is used. And ultimately, text representations that utilize video and audio alignment. Multimodal fusion features for psychological state analysis are obtained through fusion layer fusion. .

[0159] In the above optional embodiments, text modality alignment to video and audio modalities, and video and audio modal alignment to text modalities, are achieved through modal fusion technology and cross-attention mechanism. This bidirectional alignment can more comprehensively capture the interactions and complementary information between different modalities, accurately match the correspondence between different modalities, and more accurately capture the psychological state of the subjects. Furthermore, the attention mechanism from the deep Transformer architecture is employed during the alignment process, enabling the model to learn the weights and importance of different modalities more meticulously.

[0160] In an optional embodiment, step S5 includes the following steps:

[0161] S51: The mental state analysis model consists of two fully connected layers. The input is multimodal fusion features. The first layer uses the ReLU activation function, and the second fully connected layer uses the Softmax activation function. The final output is the mental state analysis result. The calculation formula of the mental state analysis model is as follows:

[0162] ;

[0163] in, These are the weight matrix and bias vector of the first fully connected layer, respectively. These are the weight matrix and bias vector of the second fully connected layer, respectively.

[0164] The model's loss function is the cross-entropy loss function used in classification tasks, as shown in the following formula:

[0165] ,

[0166] Where c is the total number of categories, This represents the true label of the i-th class. This represents the predicted probability of the i-th class; during training, the model parameters are updated using the backpropagation algorithm. , To minimize the loss function L.

[0167] Example 2:

[0168] An intelligent psychological state assessment system based on multimodal alignment, such as Figure 6 As shown, it includes:

[0169] The data acquisition module is used to collect video, audio, and text data from subjects using psychological testing and assessment tools to build a training sample set for the psychological state assessment model.

[0170] The preprocessing module is used to preprocess video data, speech data, and text data in the training data sample set, and to extract deep features from the preprocessed video data, speech data, and text data using a pre-trained large model.

[0171] The alignment and fusion module utilizes multimodal cross-attention to align and fuse text data as the primary modality, with video and audio data as auxiliary text data, to obtain multimodal fusion features for psychological state analysis.

[0172] The model training module is used to train a psychological state analysis model using aligned and fused multimodal fusion features.

[0173] The psychological analysis module is used to obtain multi-dimensional psychological analysis results of test subjects through a trained psychological state analysis model.

[0174] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for assessing intelligent psychological states based on multimodal alignment, characterized in that, Includes the following steps: S1: Collect video, audio, and text data of subjects using psychological testing and assessment tools to construct a training sample set for the psychological state assessment model; S2: Preprocess the video data, speech data, and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video data, speech data, and text data. S3: Utilizing multimodal cross-attention, with text data as the primary modality and video and audio data as auxiliary text data, alignment and fusion are performed to obtain multimodal fusion features for psychological state analysis; S4: A mental state analysis model is obtained by training the aligned and fused multimodal fusion features; S5: Obtain multi-dimensional psychological analysis results of test subjects through the psychological state analysis model obtained through training; S3 includes the following steps: S31: Global information is captured from video data, audio data, and text data respectively through a multi-head self-attention mechanism; among which, a cross-attention mechanism is constructed based on the structure of the Transformer network. S32: Construct a text-aligned video representation cross-attention mechanism and a text-aligned speech representation cross-attention mechanism; S33: Construct a cross-attention mechanism for video and speech-aligned text representations to obtain multimodal fusion features. ; S32 includes the following steps: S321: Based on video features As query Q, with text features As the key K and value V, they are obtained through a multi-head self-attention mechanism, residual linking, and layer normalization. ; S322: Represent the aligned video The output is fed into the feedforward network FFN to obtain The format is as follows: ； S323: Yes Residual connections and layer normalization are performed to obtain the final video representation aligned with text. ; S324: Based on speech features As query Q, with text features As the key K and value V, they are obtained through a multi-head self-attention mechanism, residual linking, and layer normalization. ; S325: Align the speech representation The output is fed into the feedforward network FFN to obtain The format is as follows: ； S326: Yes Residual connections and layer normalization are performed to obtain the final text-aligned speech representation. ; S33 includes the following steps: S331: Using text features As the overall query Q, based on video features As a key in the video modal component Sum Text representations aligned with video are obtained through a multi-head attention layer. ; S332: Using text features As the total query Q, based on speech features As a key in the video modal component Sum Text representations aligned with video are obtained through a multi-head attention layer. ; S333: Will and The text representation is obtained through a fusion layer, and then input into a layer normalization layer, a feedforward network layer, and another layer normalization layer to obtain the final text representation aligned with video and speech. ; S334: The final method will utilize text-aligned video representation. Finally, text-aligned speech representation is used. And ultimately, text representations that utilize video and audio alignment. Multimodal fusion features for psychological state analysis are obtained through fusion layer fusion. .

2. The intelligent psychological state assessment method based on multimodal alignment according to claim 1, characterized in that, S1 includes the following steps: S11: Provides a variety of assessment tools, including psychological assessment questionnaires; S12: Acquire the subject's multimodal data, which includes video and audio data during the answering process, as well as text data extracted from the audio data; S13: After completing the questions, calculate the score of the psychological assessment questionnaire as the psychological state label of the subject's multimodal data; S14: Construct a training sample set for the psychological state assessment model based on the multimodal data in S12 and the psychological state labels in S13.

3. The intelligent psychological state assessment method based on multimodal alignment according to claim 1, characterized in that, S2 includes the following steps: S21: Preprocessing of video data includes cropping video frames to focus on facial regions, adjusting image sizes to meet model input requirements, and normalizing images; S22: Preprocessing of speech data includes removing background noise, normalizing the speech signal, and segmenting long audio files into shorter segments; S23: Preprocessing of text data includes data cleaning to remove irrelevant characters, punctuation marks, and special symbols; using the jieba Chinese word segmentation tool to segment the text into words or phrases and remove stop words; converting all text to a uniform uppercase and lowercase form; and standardizing numbers and dates.

4. The intelligent psychological state assessment method based on multimodal alignment according to claim 3, characterized in that, S2 further includes the following steps: S24: Input the pre-processed video data from S21 into the pre-trained model EMO-AffectNetModel and use the pre-trained model to extract deep facial features; S25: Input the preprocessed speech data from S22 into emotion2vec, and obtain deep speech features with a sampling rate of 50Hz through the output of the last layer of the model; S26: Input the preprocessed text data from S23 into Piccolo2 and output the embedding vector of the subject's text expression.

5. The intelligent psychological state assessment method based on multimodal alignment according to claim 1, characterized in that, S31 includes the following steps: S311: Represent the single-modal vector as X, and generate the query Q, key K, and value V matrix of this modality data through a linear layer, in the following form: ； ； ； in, , and This is the weight matrix; S312: Divide Q, K, and V into multiple heads, calculate the attention weights for each head independently, and then perform a weighted sum. The formula for calculating the attention weights for each head is as follows: ； in, is the dimension of the key vector, used to scale the dot product attention, and i is the label of the segmentation head; S313: Concatenate the outputs of all heads and pass them through a linear layer to obtain the output of the multi-head self-attention mechanism: ； Where h is the number of segment heads. It is the output weight matrix; S314: By performing a residual join, Z is added to the original single-mode vector X to obtain... ; S315: The application layer normalization yields the output of each modality after multi-head self-attention, calculated using the following formula: ； The outputs of video, audio, and text modalities are respectively represented as follows: , and .

6. The intelligent psychological state assessment method based on multimodal alignment according to claim 1, characterized in that, S5 includes the following steps: S51: The mental state analysis model consists of two fully connected layers. The input is multimodal fusion features. The first layer uses the ReLU activation function, and the second fully connected layer uses the Softmax activation function. The final output is the mental state analysis result. The calculation formula of the mental state analysis model is as follows: ； in, These are the weight matrix and bias vector of the first fully connected layer, respectively. These are the weight matrix and bias vector of the second fully connected layer, respectively. The model's loss function is the cross-entropy loss function used in classification tasks, as shown in the following formula: ， Where c is the total number of categories, This represents the true label of the i-th class. This represents the predicted probability of the i-th class; during training, the model parameters are updated using the backpropagation algorithm. , To minimize the loss function L.

7. An intelligent psychological state assessment system based on multimodal alignment, characterized in that, The method for assessing intelligent mental states based on multimodal alignment, as described in any one of claims 1-6, includes: The data acquisition module is used to collect video, audio, and text data from subjects using psychological testing and assessment tools to build a training sample set for the psychological state assessment model. The preprocessing module is used to preprocess video data, speech data, and text data in the training data sample set, and to extract deep features from the preprocessed video data, speech data, and text data using a pre-trained large model. The alignment and fusion module is used to utilize multimodal cross-attention, with text data as the primary modality and video and audio data as auxiliary text data, to perform alignment and fusion, thereby obtaining multimodal fusion features for psychological state analysis. The model training module is used to train a mental state analysis model using the aligned and fused multimodal fusion features; The psychological analysis module is used to obtain multi-dimensional psychological analysis results of test subjects through a trained psychological state analysis model.

Images