Intelligent psychological state assessment method and system based on multi-modal alignment

Abstract

The invention discloses an intelligent psychological state assessment method and system based on multi-modal alignment, and relates to the technical field of psychological assessment, and the method comprises the following steps: S1, constructing a training sample set of a psychological state assessment model; s2, preprocessing video data, voice data and text data in the training data sample set; s3, performing alignment and fusion by using multi-modal cross attention, taking the text data as a main modal, and taking the video data and the voice data as assistance of the text data to obtain multi-modal fusion features for psychological state analysis; s4, training by using the aligned and fused multi-modal fusion features to obtain a psychological state analysis model; and S5, obtaining a multi-dimensional psychological analysis result of the testee through the psychological state analysis model obtained through training. According to the method, the multi-modal deep features of the teenagers are extracted by pre-training the large model, and the feature expression ability of the data is improved by using the modal alignment technology.

Description

Technical Field

[0001] The present invention relates to the field of psychological assessment technology, and in particular to an intelligent psychological state assessment method and system based on multimodal alignment. Background Art

[0002] According to the data of the "China National Mental Health Development Report (2021-2022)", 14.8% of the adolescent population is at risk of depression to varying degrees, and severe depression reaches 4%. However, there is a serious shortage of psychological teachers in my country, a shortage of psychological professionals, and a serious imbalance between supply and demand, which has led to a serious lack of coverage of adolescent psychological screening and intervention.

[0003] In this context, artificial intelligence-based psychological state assessment methods have gradually become an important means to solve this problem. Traditional psychological assessment methods mainly rely on questionnaires and interviews, which are difficult to fully and objectively reflect the psychological state of adolescents. Multimodal data fusion technology, by comprehensively utilizing multiple data sources such as video, voice, and text, can more comprehensively capture and analyze the psychological characteristics of adolescents and achieve more accurate psychological state assessment.

[0004] Traditional deep learning methods often use traditional small-scale models to extract features of each modality when extracting features. Their feature expression ability is limited and it is difficult to capture complex changes in psychological states and high-level expressions. In addition, because they are trained on small-scale and specific domain data, their generalization ability is poor and they are difficult to adapt to different scenarios and individual data, resulting in unstable performance in practical applications. In addition, current multimodal fusion methods, such as CN116936098A, are usually simple feature concatenation or weighted averaging, which cannot fully capture the interactions and correspondences between different modalities, and therefore have average accuracy.

[0005] In view of this, this invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings existing in the prior art, and to propose an intelligent psychological state assessment method and system based on multimodal alignment. By pre-training a large model, the multimodal deep features of adolescents are extracted, and the feature expression ability of the data is improved by using modal alignment technology, so as to realize the analysis of the psychological state of adolescents and meet the requirements of accuracy and robustness of adolescent mental health assessment.

[0007] In order to achieve the above object, the present invention also adopts the following technical solutions: An intelligent psychological state assessment method based on multimodal alignment includes the following steps: S1: Collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, and construct a training sample set for the psychological state assessment model.

[0008] S2: Preprocess the video data, voice data and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video data, voice data and text data.

[0009] S3: Using multimodal cross attention, text data is used as the main modality, and video data and voice data are used as auxiliary to align and fuse text data to obtain multimodal fusion features for psychological state analysis.

[0010] S4: The mental state analysis model is trained using the aligned and fused multimodal fusion features.

[0011] S5: The psychological state analysis model obtained through training obtains the multi-dimensional psychological analysis results of the subjects.

[0012] Further, the S1 comprises the following steps: S11: Provides a variety of assessment tools, including psychological assessment questionnaires.

[0013] S12: Obtain multimodal data of the subject, where the multimodal data includes video data and voice data when answering questions, and text data extracted from the voice data.

[0014] S13: After the questions are answered, the scores of the psychological assessment questionnaire are calculated as the psychological state labels of the subject's multimodal data.

[0015] S14: Construct a training sample set for the mental state assessment model based on the multimodal data of S12 and the mental state labels of S13.

[0016] Further, the S2 comprises the following steps: S21: Preprocessing of video data includes cropping video frames to focus on the facial area, adjusting the image size to meet the model input requirements, and normalizing the image.

[0017] S22: Preprocessing of speech data includes removing background noise, normalizing speech signals, and segmenting long audio files into shorter segments.

[0018] S23: The preprocessing of text data includes data cleaning to remove irrelevant characters, punctuation marks, and special symbols, using the jieba Chinese word segmentation tool to segment the text into words or phrases and remove stop words, converting all text to a uniform case, and standardizing numbers, dates, etc.

[0019] Furthermore, the S2 further comprises the following steps: S24: Input the video data preprocessed in S21 into EMO-AffectNetModel, and use the pre-trained model to extract deep facial features.

[0020] S25: Input the speech data preprocessed by S22 into emotion2vec, and obtain the deep speech features with a sampling rate of 50Hz through the last layer output of the model.

[0021] S26: Input the text data preprocessed in S23 into Piccolo2, and output the embedding vector of the subject's text expression.

[0022] Further, the S3 comprises the following steps: S31: The global information of video data, voice data and text data is captured respectively through the multi-head self-attention mechanism; among them, the cross-attention mechanism is constructed based on the structure of the Transformer network.

[0023] S32: Construct a cross-attention mechanism for text-aligned video representations and a cross-attention mechanism for text-aligned speech representations.

[0024] S33: Construct a cross-attention mechanism for video and speech alignment text representation to obtain multimodal fusion features .

[0025] Further, the S31 comprises the following steps: S311: The single modal vector is represented as X, and the query (Q), key (K), and value (V) matrices of the modal data are generated through a linear layer. The specific form is as follows: ; ; ; in, , and is the weight matrix.

[0026] S312: Divide Q, K and V into multiple heads, calculate the attention weight for each head independently, and perform weighted summation. The calculation formula for the attention weight of each head is as follows: ; in, is the dimension of the key vector, used to scale the dot product attention, and i is the label of the segmentation head.

[0027] S313: Concatenate the outputs of all heads and pass them through a linear layer to get the output of the multi-head self-attention mechanism: ; Where h is the number of segmentation heads, is the output weight matrix.

[0028] S314: By performing a residual connection, Z is added to the original unimodal vector X to obtain .

[0029] S315: Apply layer normalization to obtain the output of each modality after multi-head self-attention. The calculation formula is as follows: ; Among them, the outputs of video, speech and text modalities are represented as , and .

[0030] Further, the S32 comprises the following steps: S321: Video Features As query (Q), with text features As the key (K) and value (V), obtained through multi-head self-attention mechanism, residual connection and layer normalization .

[0031] S322: Representing the aligned video The output is fed into the feed-forward network FFN to obtain , of the following form:

[0032] .

[0033] S323: Yes Perform residual connection and layer normalization to obtain the final video representation using text alignment .

[0034] S324: Based on voice features As query (Q), with text features As the key (K) and value (V), obtained through multi-head self-attention mechanism, residual connection and layer normalization .

[0035] S325: Represent the aligned speech The output is fed into the feed-forward network FFN to obtain , of the following form:

[0036] .

[0037] S326: Yes Perform residual connection and layer normalization to obtain the final text-aligned speech representation .

[0038] Further, the S33 comprises the following steps: S331: Text features As the total query (Q), with video features As a key for the video modal part ( ) and value ( ), through the multi-head attention layer to obtain the text representation using video alignment .

[0039] S332: Text features As the total query (Q), with the phonetic features As a key for the video modal part ( ) and value ( ), through the multi-head attention layer to obtain the text representation using video alignment .

[0040] S333: and The text representation is obtained through the fusion layer, and then input into the layer normalization, feedforward network layer and layer normalization to obtain the final text representation using video and speech alignment .

[0041] S334: Finally, the video representation will be aligned using text , and finally use text to align speech representation And finally use the text representation of video and speech alignment Multimodal fusion features for psychological state analysis are obtained through fusion layer fusion .

[0042] Further, the S5 comprises the following steps: S51: The psychological state analysis model consists of two fully connected layers. The input is multimodal fusion features. The first layer uses the ReLU activation function, and the second fully connected layer uses the Softmax activation function. The final output is the psychological state analysis result. The calculation formula of the psychological state analysis model is as follows: ; in, are the weight matrix and bias vector of the first fully connected layer, respectively. are the weight matrix and bias vector of the second fully connected layer respectively.

[0043] Among them, the loss function of the model adopts the cross entropy loss function of the classification task, and the formula is as follows: , Where c is the total number of categories, represents the true label of the i-th category, Represents the predicted probability of the i-th category; during the training process, the parameters of the model are updated through the back propagation algorithm , To minimize the loss function L.

[0044] In order to achieve the above object, the present invention also adopts the following technical solutions: An intelligent mental state assessment system based on multimodal alignment, comprising: The data collection module is used to collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, and to construct a training sample set for the psychological state assessment model.

[0045] The preprocessing module is used to preprocess the video data, voice data and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video data, voice data and text data.

[0046] The alignment and fusion module is used to utilize multimodal cross-attention, with text data as the main modality and video data and voice data as auxiliary to text data for alignment and fusion, to obtain multimodal fusion features for psychological state analysis.

[0047] The model training module is used to train the mental state analysis model using the aligned and fused multimodal fusion features.

[0048] The psychological analysis module is used to obtain multi-dimensional psychological analysis results of the subjects through the trained psychological state analysis model.

[0049] Compared with the prior art, the present invention has the following beneficial effects: 1. Use large models to extract deep features of video data, voice data, and text data. Use pre-trained models of each modality to pre-train on a large amount of data in the corresponding field to capture more subtle and abstract psychological features. Large-scale pre-trained models can also improve the robustness of the evaluation process to noise and abnormal data, and reduce the impact of these factors on the evaluation results.

[0050] 2. With text data as the main modality, video data and voice data as auxiliary enhanced text representation modalities of text data, the alignment of video data, voice data and text data is achieved through modal fusion technology, the correspondence between different modalities is accurately matched, the correlation between different modalities is captured, and the weights are dynamically adjusted according to the importance of different modal information to achieve effective feature fusion. Compared with traditional multimodal fusion methods (such as simple feature splicing or weighted averaging), dynamic weight allocation can better utilize the information of each modality, avoid information loss or redundancy, and improve the accuracy of psychological state assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of the intelligent mental state assessment method based on multimodal alignment.

[0052] Figure 2 Schematic diagram of the criss-cross attention mechanism for text-aligned video representation.

[0053] Figure 3 Schematic diagram of the criss-cross attention mechanism for aligning speech representations to text.

[0054] Figure 4 Schematic diagram of the cross-attention mechanism for aligning text representations for video and speech.

[0055] Figure 5 This is a framework diagram of the intelligent mental state assessment method based on multimodal alignment.

[0056] Figure 6 This is a structural diagram of the intelligent mental state assessment method based on multimodal alignment. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0058] Embodiment 1: An intelligent mental state assessment method based on multimodal alignment, such as Figure 1-5 As shown, including: S1: Collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, and construct a training sample set for the psychological state assessment model.

[0059] S2: Preprocess the video data, voice data and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video data, voice data and text data.

[0060] S3: Using multimodal cross attention, text data is used as the main modality, and video data and voice data are used as auxiliary to align and fuse text data to obtain multimodal fusion features for psychological state analysis.

[0061] In this embodiment, among the video, audio and text data, the subjects will answer the reasons for choosing the answers in the psychological test, so the text data will contain key clues for evaluating the individual's psychological state. Therefore, in the multimodal cross-attention alignment and fusion process, text data is used as the main modality, and video data and voice data are used as auxiliary enhanced text representation modalities of text data.

[0062] S4: The mental state analysis model is trained using the aligned and fused multimodal fusion features.

[0063] In this embodiment, the method for acquiring multimodal fusion features is mainly improved; in addition, for the method of training the model, the existing method can be used to train the model, and the model can be continuously optimized to obtain the optimal psychological state analysis model.

[0064] S5: The psychological state analysis model obtained through training obtains the multi-dimensional psychological analysis results of the subjects.

[0065] The method of this embodiment uses a large model to extract deep features of video data, voice data, and text data, and uses a pre-trained model of each modality to pre-train on a large amount of data in the corresponding field, which can capture more subtle and abstract psychological characteristics; and the large-scale pre-trained model can improve the robustness of the evaluation process to deal with noise and abnormal data, and reduce the impact of these factors on the evaluation results.

[0066] The method of this embodiment uses modal fusion technology to align video data, voice data, and text data, accurately matches the correspondence between different modalities, captures the correlation between different modalities, and dynamically adjusts the weights according to the importance of different modal information to achieve effective feature fusion. Compared with traditional multimodal fusion methods (such as simple feature concatenation or weighted averaging), dynamic weight allocation can better utilize the information of each modality, avoid information loss or redundancy, and improve the accuracy of psychological state assessment.

[0067] In an optional embodiment, the S1 comprises the following steps: S11: Provides a variety of assessment tools, including psychological assessment questionnaires.

[0068] In this step, the subjects can fill out a psychological assessment questionnaire online. This embodiment provides the subjects with three psychological health assessment tools, namely, the Mental Health Diagnostic Test (MHT) for Adolescents and Young Students, the PHQ-9 Health Questionnaire, and the Generalized Anxiety Disorder Scale (GAD-7). The subjects can choose one of them to answer the questions.

[0069] S12: Obtain multimodal data of the subject, where the multimodal data includes video data and voice data when answering questions, and text data extracted from the voice data.

[0070] In this step, the collection of multimodal data including video data, voice data and text data is completed, wherein the video data refers to the facial video of the subject when answering questions.

[0071] When the subject starts answering questions, the camera and microphone can be activated to collect the subject's facial video data and voice data.

[0072] The Mental Health Diagnostic Test (MHT), PHQ-9 Health Questionnaire, and Generalized Anxiety Disorder Scale (GAD-7) are multiple-choice scales. In order to collect the voice data and text data of the subjects, this embodiment prompts the subjects to describe the reasons for their choice of answers and their feelings and experiences after they complete the questions, and simultaneously collects the facial video data and voice data of the subjects. In this embodiment, the combination of multiple-choice psychological assessment questionnaires and open-ended questions helps to collect more emotional details and background information reflecting the psychological state of the subjects; secondly, the subjects share their unique experiences and feelings, which can enhance the accuracy and personalization of psychological state assessment.

[0073] S13: After the questions are answered, the scores of the psychological assessment questionnaire are calculated as the psychological state labels of the subject's multimodal data.

[0074] In this step, the scores of the psychological assessment questionnaires are calculated based on the subjects' questionnaire conditions and the scoring criteria for each questionnaire.

[0075] Specifically, the Mental Health Diagnostic Test (MHT) for adolescent students divides the mental health status of primary and secondary school students into three levels: those with scores between 1 and 55 are normal, those with scores between 56 and 64 are in poor mental state, and those with scores above 65 are in a serious tendency of mental problems. The depression status of the PHQ-9 health questionnaire is divided into five levels: those with scores between 0 and 4 are mild depression, those with scores between 5 and 9 are mild depression, those with scores between 10 and 14 are moderate depression, those with scores between 15 and 19 are moderate to severe depression, and those with scores between 20 and 27 are severe depression. The anxiety status of the Generalized Anxiety Disorder Scale (GAD-7) is divided into four levels: those with scores between 0 and 4 are mild anxiety, those with scores between 5 and 9 are mild anxiety, those with scores between 10 and 14 are moderate anxiety, and those with scores between 15 and 21 are severe anxiety.

[0076] S14: Construct a training sample set for the mental state assessment model based on the multimodal data of S12 and the mental state labels of S13.

[0077] In this optional embodiment, by combining a selective psychological assessment questionnaire with open-ended questions and answers, more emotional details and background information reflecting the subject's psychological state are collected, thereby enhancing the personalization and accuracy of the assessment.

[0078] In an optional embodiment, the S2 comprises the following steps: S21: Preprocessing of video data includes cropping video frames to focus on the facial area, adjusting the image size to meet the model input requirements, and normalizing the image.

[0079] S22: Preprocessing of speech data includes removing background noise, normalizing speech signals, and segmenting long audio files into shorter segments.

[0080] S23: The preprocessing of text data includes data cleaning to remove irrelevant characters, punctuation marks, and special symbols, using the jieba Chinese word segmentation tool to segment the text into words or phrases and remove stop words, converting all text to a uniform case, and standardizing numbers, dates, etc.

[0081] In an optional embodiment, the S2 further includes the following steps: S24: Input the video data preprocessed in S21 into EMO-AffectNetModel, and use the pre-trained model to extract deep facial features.

[0082] In this step, the open source EMO-AffectNetModel is used to extract deep features of video modality data. This pre-trained model has been pre-trained on multiple facial expression recognition datasets and can capture subtle facial changes of subjects during psychological tests, thereby improving the expressiveness of deep facial features of subjects and enhancing the robustness of visual information in the psychological assessment process.

[0083] S25: Input the speech data preprocessed by S22 into emotion2vec, and obtain the deep speech features with a sampling rate of 50Hz through the last layer output of the model.

[0084] In this step, the open source speech emotion analysis pre-trained large model emotion2vec is used to extract the deep representation of the subject's speech modal data. After pre-training and fine-tuning on a large-scale emotion dataset, emotion2vec has the ability to express highly general speech emotion features. It can capture subtle emotional changes and non-verbal clues such as intonation, rhythm, and intensity from the speech data of the subjects during the psychological evaluation process, thereby improving the expressiveness of speech features in the process of psychological state assessment.

[0085] S26: Input the text data preprocessed in S23 into Piccolo2, and output the embedding vector of the subject's text expression.

[0086] In this step, the Piccolo2 Chinese text embedding model is used to embed the text data. Piccolo2 is currently the best embedding model for Chinese text embedding. Its multi-granularity text expression capability can capture the meaning and contextual information of the text from different perspectives.

[0087] In an optional embodiment, S3 includes the following steps: S31: The global information of video data, voice data and text data is captured respectively through the multi-head self-attention mechanism; among them, the cross-attention mechanism is constructed based on the structure of the Transformer network.

[0088] In this step, the multi-head self-attention mechanism is used to capture the global information of each unimodal data, thereby enhancing the expression and generalization capabilities of each modal data. Each unimodal data is the deep feature extracted by S2.

[0089] S32: Construct a cross-attention mechanism for text-aligned video representations and a cross-attention mechanism for text-aligned speech representations.

[0090] S33: Construct a cross-attention mechanism for video and speech alignment text representation to obtain multimodal fusion features .

[0091] In an optional embodiment, if Figure 2-4 As shown, the S31 includes the following steps: S311: The single modal vector is represented as X, and the query (Q), key (K), and value (V) matrices of the modal data are generated through a linear layer. The specific form is as follows: ; ; ; in, , and is the weight matrix.

[0092] In this step, the unimodal vector X is the deep feature extracted by S2.

[0093] S312: Divide Q, K and V into multiple heads, calculate the attention weight for each head independently, and perform weighted summation. The calculation formula for the attention weight of each head is as follows: ; in, is the dimension of the key vector, used to scale the dot product attention, and i is the label of the segmentation head.

[0094] In this step, i is the label of the segmentation head, that is, i represents the i-th segmentation head.

[0095] S313: Concatenate the outputs of all heads and pass them through a linear layer to get the output of the multi-head self-attention mechanism: ; Where h is the number of segmentation heads, is the output weight matrix.

[0096] S314: By performing a residual connection, Z is added to the original unimodal vector X to obtain .

[0097] S315: Apply layer normalization to obtain the output of each modality after multi-head self-attention. The calculation formula is as follows: ; Among them, the outputs of video, speech and text modalities are represented as , and .

[0098] In this step, the unimodal vectors of video, speech and text are represented as X and S311-S315 are performed respectively to obtain , and , thereby stabilizing the training process and improving model performance.

[0099] In an optional embodiment, as Figure 2-3 As shown, the S32 includes the following steps: S321: Video Features As query (Q), with text features As the key (K) and value (V), obtained through multi-head self-attention mechanism, residual connection and layer normalization .

[0100] In this step, the process of multi-head self-attention mechanism, residual link and layer normalization is the same as the cross attention calculation process of S31. The cross attention mechanism is used to align the video features with the text features to obtain .

[0101] S322: Representing the aligned video The output is fed into the feed-forward network FFN to obtain , of the following form: .

[0102] S323: Yes Perform residual connection and layer normalization to obtain the final video representation using text alignment .

[0103] S324: Based on voice features As query (Q), with text features As the key (K) and value (V), obtained through multi-head self-attention mechanism, residual connection and layer normalization .

[0104] In this step, the process of multi-head self-attention mechanism, residual link and layer normalization is the same as the cross attention calculation process of S31. The cross attention mechanism is used to align the video features with the text features to obtain .

[0105] S325: Represent the aligned speech The output is fed into the feed-forward network FFN to obtain , of the following form: .

[0106] S326: Yes Perform residual connection and layer normalization to obtain the final text-aligned speech representation .

[0107] In an optional embodiment, if Figure 4 As shown, the S33 includes the following steps: S331: Text features As the total query (Q), with video features As a key for the video modal part ( ) and value ( ), through the multi-head attention layer to obtain the text representation using video alignment .

[0108] S332: Text features As the total query (Q), with the phonetic features As a key for the video modal part ( ) and value ( ), through the multi-head attention layer to obtain the text representation using video alignment .

[0109] S333: and The text representation is obtained through the fusion layer, and then input into the layer normalization, feedforward network layer and layer normalization to obtain the final text representation using video and speech alignment .

[0110] S334: Finally, the video representation will be aligned using text , and finally use text to align speech representation And finally use the text representation of video and speech alignment Multimodal fusion features for psychological state analysis are obtained through fusion layer fusion .

[0111] In the above optional embodiment, the alignment of the text modality to the video modality and the audio modality is achieved through the modality fusion technology and the cross attention mechanism, and the alignment of the video modality and the audio modality to the text modality is achieved. This two-way alignment can more comprehensively capture the interaction and complementary information between different modalities, accurately match the correspondence between different modalities, and more accurately capture the psychological state of the subject. In the alignment process, the attention mechanism in the deep Transformer architecture is used, so that the model can learn the weights and importance between different modalities more carefully during the alignment process.

[0112] In an optional embodiment, the S5 comprises the following steps: S51: The psychological state analysis model consists of two fully connected layers. The input is multimodal fusion features. The first layer uses the ReLU activation function, and the second fully connected layer uses the Softmax activation function. The final output is the psychological state analysis result. The calculation formula of the psychological state analysis model is as follows: ; in, are the weight matrix and bias vector of the first fully connected layer, respectively. are the weight matrix and bias vector of the second fully connected layer respectively.

[0113] Among them, the loss function of the model adopts the cross entropy loss function of the classification task, and the formula is as follows: , Where c is the total number of categories, represents the true label of the i-th category, Represents the predicted probability of the i-th category; during the training process, the parameters of the model are updated through the back propagation algorithm , To minimize the loss function L.

[0114] Embodiment 2: An intelligent mental state assessment system based on multimodal alignment, such as Figure 6 As shown, including:

[0115] The data collection module is used to collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, and to construct a training sample set for the psychological state assessment model.

[0116] The preprocessing module is used to preprocess the video data, voice data and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video data, voice data and text data.

[0117] The alignment and fusion module is used to utilize multimodal cross-attention, with text data as the main modality and video data and voice data as auxiliary to text data for alignment and fusion, to obtain multimodal fusion features for psychological state analysis.

[0118] The model training module is used to train the mental state analysis model using the aligned and fused multimodal fusion features.

[0119] The psychological analysis module is used to obtain multi-dimensional psychological analysis results of the subjects through the trained psychological state analysis model.

[0120] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent psychological state assessment method based on multimodal alignment, characterized in that: The steps include: S1: Collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, and build a training sample set for the psychological state assessment model; S2: Preprocess the video data, voice data and text data in the training data sample set, and use the pre-trained large model to extract deep features from the pre-processed video data, voice data and text data; S3: Using multimodal cross attention, we use text data as the main modality, and use video data and voice data as auxiliary data for alignment and fusion to obtain multimodal fusion features for psychological state analysis; S4: The mental state analysis model is trained using the aligned and fused multimodal fusion features; S5: The psychological state analysis model obtained through training obtains the multi-dimensional psychological analysis results of the subjects.

2. According to claim 1, the intelligent psychological state assessment method based on multimodal alignment is characterized in that: The S1 comprises the following steps: S11: Provide a variety of assessment tools, the assessment tool is a psychological assessment questionnaire; S12: obtaining multimodal data of the subject, where the multimodal data includes video data and voice data of the subject answering questions, and text data extracted from the voice data; S13: After the questions are answered, the scores of the psychological assessment questionnaire are calculated as the psychological state labels of the subject's multimodal data; S14: Construct a training sample set for the mental state assessment model based on the multimodal data of S12 and the mental state labels of S13.

3. The intelligent psychological state assessment method based on multimodal alignment according to claim 1, characterized in that: The S2 comprises the following steps: S21: Preprocessing of video data includes cropping video frames to focus on the facial area, adjusting the image size to meet the model input requirements, and normalizing the image; S22: Preprocessing of speech data includes removing background noise, normalizing speech signals, and segmenting long audio files into shorter segments; S23: The preprocessing of text data includes data cleaning to remove irrelevant characters, punctuation marks, and special symbols, using the jieba Chinese word segmentation tool to segment the text into words or phrases and remove stop words, converting all text to a uniform case, and standardizing numbers, dates, etc.

4. The intelligent psychological state assessment method based on multimodal alignment according to claim 3 is characterized in that: The S2 further comprises the following steps: S24: Input the video data pre-processed in S21 into EMO-AffectNetModel, and use the pre-trained model to extract deep facial features; S25: Input the speech data preprocessed by S22 into emotion2vec, and obtain the deep speech features with a sampling rate of 50Hz through the last layer output of the model; S26: Input the text data preprocessed in S23 into Piccolo2, and output the embedding vector of the subject's text expression.

5. The intelligent psychological state assessment method based on multimodal alignment according to claim 1, characterized in that: The S3 comprises the following steps: S31: The global information of video data, voice data and text data is captured through the multi-head self-attention mechanism. The cross-attention mechanism is constructed based on the structure of the Transformer network. S32: Construct cross-attention mechanism for text-aligned video representation and cross-attention mechanism for text-aligned speech representation; S33: Construct a cross-attention mechanism for video and speech alignment text representation to obtain multimodal fusion features .

6. The intelligent psychological state assessment method based on multimodal alignment according to claim 5, characterized in that: The S31 comprises the following steps: S311: The single modal vector is represented as X, and the query (Q), key (K), and value (V) matrices of the modal data are generated through a linear layer. The specific form is as follows: ； ； ； in, , and is the weight matrix; S312: Divide Q, K and V into multiple heads, calculate the attention weight for each head independently, and perform weighted summation. The calculation formula for the attention weight of each head is as follows: ； in, is the dimension of the key vector, used to scale the dot product attention, and i is the label of the segmentation head; S313: Concatenate the outputs of all heads and pass them through a linear layer to get the output of the multi-head self-attention mechanism: ； Where h is the number of segmentation heads, is the output weight matrix; S314: By performing a residual connection, Z is added to the original unimodal vector X to obtain ; S315: Apply layer normalization to obtain the output of each modality after multi-head self-attention. The calculation formula is as follows: ； Among them, the outputs of video, speech and text modalities are represented as , and .

7. The intelligent psychological state assessment method based on multimodal alignment according to claim 6, characterized in that: The S32 comprises the following steps: S321: Video Features As query (Q), with text features As the key (K) and value (V), obtained through multi-head self-attention mechanism, residual connection and layer normalization ; S322: Representing the aligned video The output is fed into the feed-forward network FFN to obtain , of the following form: ； S323: Yes Perform residual connection and layer normalization to obtain the final video representation using text alignment ; S324: Based on voice features As query (Q), with text features As the key (K) and value (V), obtained through multi-head self-attention mechanism, residual connection and layer normalization ; S325: Represent the aligned speech The output is fed into the feed-forward network FFN to obtain , of the following form: ； S326: Yes Perform residual connection and layer normalization to obtain the final text-aligned speech representation .

8. The intelligent psychological state assessment method based on multimodal alignment according to claim 7, characterized in that: The S33 comprises the following steps: S331: Text features As the total query (Q), with video features As a key for the video modal part ( ) and value ( ), through the multi-head attention layer to obtain the text representation using video alignment ; S332: Text features As the total query (Q), with the phonetic features As a key for the video modal part ( ) and value ( ), through the multi-head attention layer to obtain the text representation using video alignment ; S333: and The text representation is obtained through the fusion layer, and then input into the layer normalization, feedforward network layer and layer normalization to obtain the final text representation using video and speech alignment ; S334: Finally, the video representation will be aligned using text Finally, we use text to align speech representation And finally use the text representation of video and speech alignment Multimodal fusion features for psychological state analysis are obtained through fusion layer fusion .

9. The intelligent psychological state assessment method based on multimodal alignment according to claim 8, characterized in that: The S5 comprises the following steps: S51: The psychological state analysis model consists of two fully connected layers. The input is multimodal fusion features. The first layer uses the ReLU activation function, and the second fully connected layer uses the Softmax activation function. The final output is the psychological state analysis result. The calculation formula of the psychological state analysis model is as follows: ； Among them, among them, are the weight matrix and bias vector of the first fully connected layer, respectively. They are the weight matrix and bias vector of the second fully connected layer respectively; Among them, the loss function of the model adopts the cross entropy loss function of the classification task, and the formula is as follows: ， Where c is the total number of categories, represents the true label of the i-th category, Represents the predicted probability of the i-th category; during the training process, the parameters of the model are updated through the back propagation algorithm , To minimize the loss function L.

10. An intelligent psychological state assessment system based on multimodal alignment, characterized in that: include: The data collection module is used to collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, and to construct a training sample set for the psychological state evaluation model; A preprocessing module is used to preprocess the video data, voice data and text data in the training data sample set, and use the pre-trained large model to extract deep features from the preprocessed video data, voice data and text data; The alignment and fusion module is used to utilize multimodal cross attention, with text data as the main modality, and video data and voice data as auxiliary to align and fuse text data, to obtain multimodal fusion features for psychological state analysis; A model training module is used to train a psychological state analysis model using the aligned and fused multimodal fusion features; The psychological analysis module is used to obtain multi-dimensional psychological analysis results of the subjects through the trained psychological state analysis model.

Images