Joint representation psychological state assessment method and system based on PMI text enhancement

Abstract

The invention discloses a joint representation psychological state assessment method and system based on PMI text enhancement, and relates to the technical field of psychological assessment, and the method comprises the following steps: S1, constructing a training sample set of a psychological state assessment model; s2, preprocessing the collected video data, voice data and text data, extracting deep features of each mode, and constructing a PMI matrix expressed by a text; s3, optimizing an attention mechanism of a text mode by using the text PMI matrix, and constructing a triple attention fusion mechanism to realize fusion of multi-mode deep features of the testee; s4, training a psychological state evaluator through the multi-modal fusion features, and obtaining a psychological state evaluation model by adjusting parameters; and S5, performing psychological assessment through the psychological state assessment model. According to the method provided by the invention, high evaluation accuracy can still be kept under the condition that some modal data are lacked.

Description

Technical Field

[0001] The present invention relates to the field of psychological assessment technology, and in particular to a joint representation psychological state assessment method and system based on PMI text enhancement. Background Art

[0002] Currently, there are 300 million people suffering from depression, about 60 million people suffering from bipolar disorder, and 23 million people suffering from schizophrenia in the world. According to data from the Mental Health Center of the Chinese Center for Disease Control and Prevention, the development trend of mental illness in my country has shown a significant upward trend in recent years. The number of patients with various mental illnesses in China has exceeded 100 million, of which the number of patients with schizophrenia exceeds 6.4 million and the number of patients with bipolar disorder reaches 1.1 million. These data show that mental illness has become a social health problem that cannot be ignored, and it has placed a heavy burden on both individuals and society.

[0003] Traditional methods of assessing mental status usually include face-to-face clinical interviews and the use of standardized psychometric tools, such as questionnaires. However, these methods are subjective, may be affected by biases of both the assessor and the subject, and are usually time-consuming and resource-intensive. Although questionnaires are a common assessment tool, their limitation lies in the subjective self-report of the subjects, which may be inaccurate or incomplete.

[0004] In contrast, artificial intelligence algorithms have shown significant advantages in terms of accuracy, real-time, efficiency and scale of psychological state assessment. However, current deep learning methods still have insufficient modal fusion and lack a universally applicable optimal fusion strategy. In practical applications, they often face the challenge of missing modalities, that is, data of certain modalities cannot be obtained due to various reasons (such as sensor failure, data loss, etc.), resulting in a significant decrease in the accuracy of the assessment results when certain modalities are missing.

[0005] In view of this, this invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to address the shortcomings existing in the prior art, and to propose a joint representation psychological state assessment method and system based on PMI text enhancement, which can still maintain a high assessment accuracy in the absence of certain modal data, thereby enhancing the robustness and practicality of the psychological assessment system.

[0007] In order to achieve the above object, the present invention also adopts the following technical solutions: A joint representation psychological state assessment method based on PMI text enhancement includes the following steps: S1: Collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, as well as the psychological labels corresponding to the three, and construct a training sample set for the psychological state assessment model.

[0008] S2: Preprocess the collected video data, voice data and text data, extract the deep features of each modality, and construct the PMI matrix of text expression.

[0009] S3: Use the text PMI matrix to optimize the attention mechanism of the text modality and construct a triple attention fusion mechanism to achieve the fusion of the multimodal deep features of the subjects.

[0010] S4: The mental state evaluator is trained through multimodal fusion features, and the mental state evaluation model is obtained by adjusting parameters.

[0011] S5: Conduct psychological assessment using the Mental State Assessment Model.

[0012] Further, the S2 comprises the following steps: S201: Convert the facial image of each frame in the video data into a grayscale image.

[0013] S202: Use a median filter to replace the central pixel value by selecting the median of the pixel values ​​in the neighborhood to remove noise from each frame of the facial image.

[0014] S203: Adjust the brightness of the image through linear transformation.

[0015] S204: Use histogram equalization to enhance the contrast of the image.

[0016] S205: Normalize the pixel values ​​of the image to the interval [0, 1]. The normalization formula is: ; In the formula, I is the original pixel value, min(I) is the minimum value among the original pixel values, max(I) is the maximum value among the original pixel values, and I' is the normalized pixel value.

[0017] S206: Using the model VideoMAE pre-trained on the MER2023 facial expression recognition database as a video modality encoder, the deep visual features of the subject's video modality are extracted. The deep features of the video modality are in the following form: ; In the formula, N is the number of frames in the video data.

[0018] Further, the S2 comprises the following steps: S207: Using the FFmpeg audio processing tool, the sampling rates of all audio data are unified to 16 kHz, and the channels are unified to mono.

[0019] S208: Using Deep Speaker to ensure that only the audio data to be tested is in the audio according to the similarity of the speakers in the audio.

[0020] S209: Using the wav2vec-Large pre-trained model trained on the Chinese audio dataset as the encoder of the subject's audio dataset, extract the deep features of the subject's audio modality. The deep features of the audio modality are in the following form: ; In the formula, S is the number of frames in the audio data.

[0021] Further, the S2 comprises the following steps: S210: Remove irrelevant characters in the text data.

[0022] S211: Use LAC word segmentation technology to segment the text into multiple words or phrases.

[0023] S212: Remove stop words from the text.

[0024] S213: Convert the text data into vector form.

[0025] S214: The deep features of the text modality of the subject are extracted using the BLOOM-7B pre-trained model pre-trained on a large number of Chinese corpora. The deep features of the text modality are in the following form: ; In the formula, T is the number of words in the text data.

[0026] Further, the S2 comprises the following steps: S215: Obtain a vocabulary list of the text data, the vocabulary list includes all the words that have appeared, and the vocabulary list is assumed to be Y, which includes M different words.

[0027] S216: For each word , count the frequency of each word or phrase in all text data , and the co-occurrence frequency matrix between them ,The co-occurrence frequency matrix is ​​a two-dimensional matrix, where the rows and columns are words or phrases, and the elements in the matrix represent the co-occurrence frequencies between the corresponding words or phrases.

[0028] S217: Construct vocabulary pairs for all text expressions of the test subject The PMI matrix, the calculation formula of the elements in the PMI matrix is ​​as follows: ; ; ; ; In the formula, For word pairs and The probability of co-occurrence, D is the total number of text expressions, Words to express The probability of occurrence, For vocabulary Probability of occurrence.

[0029] Further, the S3 comprises the following steps: S31: Construct self-attention modules for video and audio modalities.

[0030] S32: Construct a self-attention module for the text modality and use the constructed PMI matrix to optimize the self-attention mechanism for the text modality.

[0031] S33: Construct a multimodal joint representation module C, whose calculation formula is as follows: ; In the formula, k represents the number of stages of execution attention. In this embodiment, k=2. and The context vectors representing text, audio, and video, respectively, come from the output of the attention modules of each modality at each stage.

[0032] S34: After a two-stage multimodal joint representation module, the fusion features of text modality, audio modality and video modality are obtained.

[0033] Further, the S32 comprises the following steps: S321: Jointly represent multiple modalities and deep representation of textual modality Fusion is performed via a linear layer.

[0034] S322: Get the query (Query, Q), key (Key, K) and value (Value, V) of the text self-attention module through the linear layer.

[0035] S323: Then calculate the attention score, normalize the attention score by Softmax to get the attention weight of the text, and use the attention weight score to perform weighted summation to get the text context vector of the kth stage of the visual attention modality , and its calculation formula is as follows: ; ; ; ; In the formula, is the learnable parameter of the k-th stage attention module, and d is Dimension.

[0036] Further, in said S33: Initialization state of the multimodal joint representation module The calculation formula is as follows: ; ; ; ; A global average pooling method is used to obtain the initial values ​​of the context vectors for the text modality, audio modality, and video modality.

[0037] Further, the S4 comprises the following steps: S41: Constructing a multilayer perceptron as a mental state evaluator.

[0038] S42: receiving the fused features as input, wherein the neurons of each hidden layer of the multilayer perceptron take the output of the previous layer as input.

[0039] S43: Calculated by weighted sum plus bias term.

[0040] S44: The output of the last hidden layer is sent to the output layer through the nonlinear activation function ReLU, and the output layer outputs the psychological state classification results through the Softmax function.

[0041] Among them, for each layer l of the multilayer perceptron, the output of the network and activation , the weighted sum and bias formula are as follows: ; In the formula, and are the weight and bias of the lth layer respectively, is the activation output of the previous layer.

[0042] The loss function calculation formula is as follows: ; In the formula, M is the number of categories, is the real class of mental states, The category of the predicted psychological state.

[0043] In order to achieve the above object, the present invention also adopts the following technical solution: A joint representation psychological state assessment system based on PMI text enhancement, including: The data collection module is used to collect the video data, voice data and text data of the subject for the psychological testing and evaluation tool, as well as the psychological labels corresponding to the three, to construct a training sample set for the psychological state evaluation model.

[0044] The PMI matrix construction module is used to preprocess the collected video data, voice data and text data, extract deep features of each modality, and construct a PMI matrix of text expression.

[0045] The feature fusion module uses the text PMI matrix to optimize the attention mechanism of the text modality and constructs a triple attention fusion mechanism to achieve the fusion of the multi-modal deep features of the subject.

[0046] A model training module is used to train the mental state evaluator through multimodal fusion features and obtain a mental state evaluation model by adjusting parameters.

[0047] A psychological assessment module is used to perform psychological assessment through a psychological state assessment model.

[0048] Compared with the prior art, the present invention has the following beneficial effects: 1. By using multimodal alignment technology to fuse the subject's video, audio and text multimodal data, the subject's PMI (Pointwise Mutual Information) matrix is ​​constructed, which can enhance the expression of the subject's text features and the fusion of modalities. In the absence of certain modal data, it can still maintain a high level of assessment accuracy, thereby enhancing the robustness and practicality of the psychological assessment system and improving the assessment efficiency and scale.

[0049] 2. Utilize the self-attention mechanism and multimodal alignment technology after PMI matrix optimization to achieve effective fusion of data of different modalities and enhance the comprehensiveness and accuracy of multimodal data in psychology to improve the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the joint representation mental state assessment method based on PMI text enhancement.

[0051] Figure 2 This is the structure diagram of the video / audio feature self-attention module.

[0052] Figure 3 Optimizing the structure of the text self-attention module for PMI.

[0053] Figure 4 This is a framework diagram for the psychological state assessment method.

[0054] Figure 5 This is the structural diagram of the joint representation mental state assessment method based on PMI text enhancement. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0056] Embodiment 1: A joint representation psychological state assessment method based on PMI text enhancement, such as Figure 1-4 As shown, the following steps are included: S1: Collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, as well as the psychological labels corresponding to the three, and construct a training sample set for the psychological state assessment model.

[0057] In this embodiment, the psychological testing and evaluation tools can be an integrated self-rating depression scale (Self-Rating Depression Scale, SDS) and a self-rating anxiety scale (Self-Rating Anxiety Scale, SAS), which are used to evaluate the depressive symptoms and anxiety symptoms of the subject respectively. The SDS self-rating depression scale and the SAS self-rating anxiety scale are standardized psychological evaluation scale tools, which are used to measure the psychological characteristics or mental health status of the subject. They have been scientifically verified and standardized and can provide objective and reliable data.

[0058] In this embodiment, a web-based platform can be provided for the test subject, on which both scales present questions, options, and necessary instructions. The test subject fills in the SDS and SAS scales by accessing the web-based platform. During the test, the system collects the test subject's facial video data and voice data in real time through a camera and a microphone.

[0059] In this embodiment, data can be obtained through the following specific implementation steps: First, after the test subject completes the answer selection for each item in the scale, the platform can gently guide the test subject to describe in detail the specific reasons for their choice of each item through prompts; the description can be a further explanation of the options, or the test subject's current feelings, experiences or thoughts, thereby providing more comprehensive and detailed information on the test subject's psychological state. Both scales are in the form of a series of multiple-choice questions. The above steps can collect the test subject's voice and text expressions more deeply, and further understand their psychological state and the reasons behind their choices.

[0060] Then, after the subjects complete the SDS self-rating depression scale and the SAS self-rating anxiety scale, according to the subjects' answers, based on the subjects' choices in the scale to reflect the severity of their depression and anxiety symptoms, the subjects' psychological labels are constructed according to the answers, which serve as an important feature for the subsequent psychological state assessment model training to improve the accuracy and reliability of the model. Specifically, according to the SDS self-rating depression scale and the SAS self-rating anxiety scale, the subjects' psychological labels (depression state labels) can be divided into four levels: scores below 53 are normal, 53-62 are mild depression, 63-72 are moderate depression, and 72 points or more are severe depression. The subjects' psychological labels (anxiety state labels) can also be divided into four levels: scores below 50 are normal, 50-59 points are mild anxiety, 60-69 points are moderate anxiety, and 69 points or more are severe anxiety.

[0061] Finally, the facial video data and voice data of the subjects collected in real time by the camera and microphone are combined, and the text is extracted from the voice data as text data to construct a training sample set for the psychological state assessment model.

[0062] It should be noted that in the process of constructing the training sample set, it is necessary to strictly abide by the relevant regulations on privacy protection and data security to ensure that the sensitive information of the subjects is properly handled and protected, and the training samples need to be de-identified and anonymized to avoid leaking the personal information of the subjects.

[0063] S2: Preprocess the collected video data, voice data and text data, extract the deep features of each modality, and construct the PMI matrix of text expression.

[0064] S3: Use the text PMI matrix to optimize the attention mechanism of the text modality and construct a triple attention fusion mechanism to achieve the fusion of the multimodal deep features of the subjects.

[0065] S4: The mental state evaluator is trained through multimodal fusion features, and the mental state evaluation model is obtained by adjusting parameters.

[0066] S5: Conduct psychological assessment using the Mental State Assessment Model.

[0067] In this embodiment, S5 is evaluated through the model constructed by S1-S4.

[0068] The method of this embodiment, by utilizing multimodal alignment technology to fuse the subject's video, audio and text multimodal data, constructs the subject's PMI (Pointwise Mutual Information) matrix, which can enhance the expression of the subject's text features and the fusion of modalities, and thus can maintain a high evaluation accuracy in the absence of certain modal data, thereby enhancing the robustness and practicality of the psychological assessment system and improving the evaluation efficiency and scale.

[0069] The method of this embodiment utilizes the self-attention mechanism and multimodal alignment technology after PMI matrix optimization to achieve effective fusion between different modal data, enhance the comprehensiveness and accuracy of multimodal data in psychology and improve the evaluation results.

[0070] In an optional embodiment, the video data of the subject is preprocessed through the following steps, and the deep features of the video modality are extracted from the preprocessed data; specifically, S2 includes the following steps: S201: Convert the facial image of each frame in the video data into a grayscale image.

[0071] In this optional embodiment, this step can reduce the data dimension and avoid interference.

[0072] In addition, the OpenCV DNN facial detection tool can be used to ensure that there is only one face in each frame of the video, and combined with the InsightFace facial recognition tool to ensure that each frame of the video belongs to the same subject, that is, other faces other than the subject are deleted to ensure that the subject's facial information is recorded completely and accurately during the evaluation process, and to avoid evaluation errors caused by identity confusion.

[0073] S202: Use a median filter to replace the central pixel value by selecting the median of the pixel values ​​in the neighborhood to remove noise from each frame of the facial image.

[0074] In this optional embodiment, by selecting the median of the pixel values ​​in the neighborhood to replace the central pixel value, noise can be effectively removed while retaining the edge features of the facial image.

[0075] S203: Adjust the brightness of the image through linear transformation.

[0076] In this optional embodiment, the overall brightness level of the image can be improved by this step.

[0077] S204: Use histogram equalization to enhance the contrast of the image.

[0078] In this optional embodiment, this step can expand the grayscale range of the image to improve the contrast of the image and make the facial details more obvious.

[0079] S205: Normalize the pixel values ​​of the image to the interval [0, 1]. The normalization formula is: ; In the formula, I is the original pixel value, min(I) is the minimum value among the original pixel values, max(I) is the maximum value among the original pixel values, and I' is the normalized pixel value.

[0080] In this optional embodiment, this step ensures that the pixel values ​​of all images are within the same range, thereby improving the performance of the subsequent facial video feature analysis model.

[0081] S206: Using the model VideoMAE pre-trained on the MER2023 facial expression recognition database as a video modality encoder, the deep visual features of the subject's video modality are extracted. The deep features of the video modality are in the following form: ; In the formula, N is the number of frames in the video data.

[0082] In an optional embodiment, the audio data of the subject is preprocessed through the following steps, and the deep features of the audio modality are extracted from the preprocessed data; specifically, S2 includes the following steps: S207: Using the FFmpeg audio processing tool, the sampling rates of all audio data are unified to 16 kHz, and the channels are unified to mono.

[0083] S208: Using Deep Speaker to ensure that only the audio data to be tested is in the audio according to the similarity of the speakers in the audio.

[0084] S209: Using the wav2vec-Large pre-trained model trained on the Chinese audio dataset as the encoder of the subject's audio dataset, extract the deep features of the subject's audio modality. The deep features of the audio modality are in the following form: ; In the formula, S is the number of frames in the audio data.

[0085] In this optional embodiment, Deep Speaker is used to ensure that only the audio data of the person being tested is included in the audio according to the similarity of the speakers in the audio, that is, the audio data other than the audio data of the person being tested is deleted.

[0086] In an optional embodiment, the text data of the subject is preprocessed through the following steps, and the deep features of the text modality are extracted from the preprocessed data; specifically, S2 includes the following steps: S210: Remove irrelevant characters from the text data.

[0087] In this optional embodiment, remove the irrelevant characters from the text data to retain useful text information; in addition, the irrelevant characters include punctuation marks, numbers, special symbols, etc.

[0088] S211: Use the LAC word segmentation technology to split the text into multiple words or phrases.

[0089] In this optional embodiment, split the text into multiple words or phrases to facilitate subsequent sentiment analysis and PMI matrix construction; in addition, LAC refers to Lexical Analysis of Chinese.

[0090] S212: Remove stop words from the text.

[0091] In this optional embodiment, remove the stop words from the text, that is, remove the words that have no practical significance for the sentiment analysis of the text; in addition, the stop words include "de", "le", "zai", etc.

[0092] S213: Convert the text data into vector form.

[0093] In this optional embodiment, convert the text data into vector form to facilitate subsequent machine learning algorithm processing; in addition, the text data can be converted into vector form by methods such as bag-of-words model, TF-IDF, word embedding, etc.

[0094] S214: Use the pre-trained BLOOM-7B pre-trained model on a large number of Chinese corpora to extract the deep features of the text modality of the被测者文本模态. The form of the deep features of the text modality is as follows: ; In the formula, T is the number of words in the text data.

[0095] In the above optional embodiment, by using the pre-training of Chinese corpora and large language models as the encoders of each modality, convert data of different modalities (text data, speech data, video data, and video data is facial expression) into a general representation that the model can understand, which not only contains rich semantic information but also can reveal the psychological state and behavior patterns of individuals in different situations, improving the accuracy and generalization of subsequent psychological assessment models.

[0096] In an optional embodiment, construct a PMI matrix for text expression through the following steps; specifically, the S2 includes the following steps: S215: Obtain the vocabulary of the text data. The vocabulary contains all the words that have appeared. Let the vocabulary be Y, which contains M different words.

[0097] S216: For each word , count the frequency of each word or phrase in all text data , and the co-occurrence frequency matrix between them ,The co-occurrence frequency matrix is ​​a two-dimensional matrix, where the rows and columns are words or phrases, and the elements in the matrix represent the co-occurrence frequencies between the corresponding words or phrases.

[0098] S217: Construct vocabulary pairs for all text expressions of the test subject The PMI matrix, the calculation formula of the elements in the PMI matrix is ​​as follows: ; ; ; ; In the formula, For word pairs and The probability of co-occurrence, D is the total number of text expressions, Words to express The probability of occurrence, For vocabulary Probability of occurrence.

[0099] In an optional embodiment, S3 includes the following steps: S31: Construct self-attention modules for video and audio modalities.

[0100] In this optional embodiment, the specific structure of the self-attention module of the video modality and the audio modality is shown in Figure 2 , both are traditional self-attention mechanisms. Specifically, taking the video modality as an example, in the kth stage, the multimodal joint representation is first and deep representation of video modalities Fusion is performed through a linear layer; the query (Query, Q), key (Key, K) and value (Value, V) of the visual self-attention module are then obtained through a linear layer; the attention score is then calculated, and the attention score is normalized through Softmax to obtain the visual attention weight, and the attention weight score is used to perform a weighted summation on V to obtain the video context vector of the kth stage of the visual attention modality , and its calculation formula is as follows: ; ; ; In the formula, is the learnable parameter of the k-th stage attention module, and d is Dimension.

[0101] S32: Construct a self-attention module for the text modality and use the constructed PMI matrix to optimize the self-attention mechanism for the text modality.

[0102] In this optional embodiment, the constructed PMI matrix is ​​used to optimize the self-attention mechanism of the text modality to enhance the expression of text features in the psychological assessment process.

[0103] S33: Construct a multimodal joint representation module C, whose calculation formula is as follows: ; In the formula, k represents the number of stages of execution attention. In this embodiment, k=2. and The context vectors representing text, audio, and video, respectively, come from the output of the attention modules of each modality at each stage.

[0104] S34: After a two-stage multimodal joint representation module, the fusion features of text modality, audio modality and video modality are obtained.

[0105] In this optional embodiment, the multimodal joint representation of the previous stage guides the text, audio and video self-attention modules of the next stage to focus on key information for psychological state assessment.

[0106] In an optional embodiment, the specific structure of the text self-attention optimization module is shown in Figure 3 , in the kth stage, a self-attention module of the text modality is constructed through the following steps; specifically, the S32 includes the following steps: S321: Jointly represent multiple modalities and deep representation of textual modality Fusion is performed via a linear layer.

[0107] S322: Get the query (Query, Q), key (Key, K) and value (Value, V) of the text self-attention module through the linear layer.

[0108] S323: Then calculate the attention score, normalize the attention score by Softmax to get the attention weight of the text, and use the attention weight score to perform weighted summation to get the text context vector of the kth stage of the visual attention modality , and its calculation formula is as follows: ; ; ; ; In the formula, is the learnable parameter of the k-th stage attention module, and d is Dimension.

[0109] In an optional embodiment, in S33: Initialization state of the multimodal joint representation module The calculation formula is as follows: ; ;

[0110] ; A global average pooling method is used to obtain the initial values ​​of the context vectors for the text modality, audio modality, and video modality.

[0111] In this optional embodiment, each layer of information of each modality can be compactly represented, thereby effectively encoding important features of the input data.

[0112] In an optional embodiment, the final mental state assessment is a classification task, so a multi-layer perceptron is constructed as a mental state evaluator; specifically, S4 includes the following steps: S41: Constructing a multilayer perceptron as a mental state evaluator.

[0113] S42: receiving the fused features as input, wherein the neurons of each hidden layer of the multilayer perceptron take the output of the previous layer as input.

[0114] S43: Calculated by weighted sum plus bias term.

[0115] S44: The output of the last hidden layer is sent to the output layer through the nonlinear activation function ReLU, and the output layer outputs the psychological state classification results through the Softmax function.

[0116] Among them, for each layer l of the multilayer perceptron, the output of the network and activation , the weighted sum and bias formula are as follows: ; In the formula, and are the weight and bias of the lth layer respectively, is the activation output of the previous layer.

[0117] The loss function calculation formula is as follows: ; In the formula, M is the number of categories, is the real class of mental states, The category of the predicted psychological state.

[0118] Embodiment 2: A joint representation psychological state assessment system based on PMI text enhancement, including:

[0119] The data collection module is used to collect the video data, voice data and text data of the subject for the psychological testing and evaluation tool, as well as the psychological labels corresponding to the three, to construct a training sample set for the psychological state evaluation model.

[0120] The PMI matrix construction module is used to preprocess the collected video data, voice data and text data, extract deep features of each modality, and construct a PMI matrix of text expression.

[0121] The feature fusion module uses the text PMI matrix to optimize the attention mechanism of the text modality and constructs a triple attention fusion mechanism to achieve the fusion of the multi-modal deep features of the subject.

[0122] A model training module is used to train the mental state evaluator through multimodal fusion features and obtain a mental state evaluation model by adjusting parameters.

[0123] A psychological assessment module is used to perform psychological assessment through a psychological state assessment model.

[0124] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A joint representation psychological state assessment method based on PMI text enhancement, characterized in that: The steps include: S1: Collect the video data, voice data and text data of the subjects for the psychological testing and evaluation tools, as well as the psychological labels corresponding to the three, and construct a training sample set for the psychological state evaluation model; S2: Preprocess the collected video data, voice data and text data, extract the deep features of each modality, and construct the PMI matrix of text expression; S3: Use the text PMI matrix to optimize the attention mechanism of the text modality and construct a triple attention fusion mechanism to achieve the fusion of the multi-modal deep features of the subjects; S4: Train the mental state evaluator through multimodal fusion features and obtain the mental state evaluation model by adjusting parameters; S5: Conduct psychological assessment using the Mental State Assessment Model.

2. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 1 is characterized in that: The S2 comprises the following steps: S201: Convert the facial image of each frame in the video data into a grayscale image; S202: Using a median filter to replace the central pixel value by selecting the median of the pixel values ​​in the neighborhood, so as to remove noise from each frame of the facial image; S203: adjusting the brightness of the image through linear transformation; S204: using histogram equalization to enhance the contrast of the image; S205: Normalize the pixel values ​​of the image to the interval [0, 1]. The normalization formula is: ； In the formula, I is the original pixel value, min(I) is the minimum value among the original pixel values, max(I) is the maximum value among the original pixel values, and I' is the normalized pixel value. S206: Using the model VideoMAE pre-trained on the MER2023 facial expression recognition database as a video modality encoder, the deep visual features of the subject's video modality are extracted. The deep features of the video modality are in the following form: ； In the formula, N is the number of frames in the video data.

3. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 1 is characterized in that: The S2 comprises the following steps: S207: Using the FFmpeg audio processing tool, the sampling rate of all audio data is unified to 16kHz, and the channels are unified to mono; S208: Using Deep Speaker to ensure that only the audio data being tested is in the audio according to the similarity of the speakers in the audio; S209: Using the wav2vec-Large pre-trained model trained on the Chinese audio dataset as the encoder of the subject's audio dataset, extract the deep features of the subject's audio modality. The deep features of the audio modality are in the following form: ； In the formula, S is the number of frames in the audio data.

4. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 1 is characterized in that: The S2 comprises the following steps: S210: removing irrelevant characters in the text data; S211: Use LAC word segmentation technology to segment the text into multiple words or phrases; S212: Remove stop words from the text; S213: converting text data into vector form; S214: The deep features of the text modality of the subject are extracted using the BLOOM-7B pre-trained model pre-trained on a large number of Chinese corpora. The deep features of the text modality are in the following form: ； In the formula, T is the number of words in the text data.

5. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 1 is characterized in that: The S2 comprises the following steps: S215: Obtain a vocabulary of the text data, the vocabulary includes all the words that have appeared, and the vocabulary is denoted by Y, which includes M different words; S216: For each word , count the frequency of each word or phrase in all text data , and the co-occurrence frequency matrix between them ,The co-occurrence frequency matrix is ​​a two-dimensional matrix, where the rows and columns are words or phrases, and the elements in the matrix represent the co-occurrence frequencies between the corresponding words or phrases; S217: Construct vocabulary pairs for all text expressions of the test subject The PMI matrix, the calculation formula of the elements in the PMI matrix is ​​as follows: ； ； ； ； In the formula, For word pairs and The probability of co-occurrence, D is the total number of text expressions, Words to express The probability of occurrence, For vocabulary Probability of occurrence.

6. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 1, characterized in that: The S3 comprises the following steps: S31: Construct self-attention modules for video and audio modalities; S32: Construct a self-attention module for the text modality, and use the constructed PMI matrix to optimize the self-attention mechanism for the text modality; S33: Construct a multimodal joint representation module C, whose calculation formula is as follows: ； In the formula, k represents the number of stages of execution attention. In this embodiment, k=2. and The context vectors representing text, audio, and video come from the output of the attention modules of each modality at each stage; S34: After a two-stage multimodal joint representation module, the fusion features of text modality, audio modality and video modality are obtained.

7. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 6 is characterized in that: The S32 comprises the following steps: S321: Jointly represent multiple modalities and deep representation of textual modality Fusion is performed via linear layers; S322: Get the query (Query, Q), key (Key, K) and value (Value, V) of the text self-attention module through the linear layer; S323: Then calculate the attention score, normalize the attention score by Softmax to get the attention weight of the text, and use the attention weight score to perform weighted summation to get the text context vector of the kth stage of the visual attention modality , and its calculation formula is as follows: ； ； ； ； In the formula, is the learnable parameter of the k-th stage attention module, and d is Dimension.

8. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 6 is characterized in that: In said S33: Initialization state of the multimodal joint representation module The calculation formula is as follows: ； ； ； ； A global average pooling method is used to obtain the initial values ​​of the context vectors of the text modality, audio modality, and video modality.

9. The method for joint representation of psychological state assessment based on PMI text enhancement according to claim 1, characterized in that: The S4 comprises the following steps: S41: Constructing a multilayer perceptron as a mental state evaluator; S42: receiving the fused features as input, wherein the neurons of each hidden layer of the multilayer perceptron take the output of the previous layer as input; S43: calculated by weighted sum plus bias term; S44: The output of the last hidden layer is sent to the output layer through the activation of the nonlinear activation function ReLU, and the output layer outputs the psychological state classification results through the Softmax function; Among them, for each layer l of the multilayer perceptron, the output of the network and activation , the weighted sum and bias formula are as follows: ； In the formula, and are the weight and bias of the lth layer respectively, is the activation output of the previous layer; The loss function calculation formula is as follows: ； In the formula, M is the number of categories, is the real class of mental states, The category of the predicted psychological state.

10. A joint representation psychological state assessment system based on PMI text enhancement, characterized in that: include: A data collection module, which is used to collect the video data, voice data and text data of the subject for the psychological testing and evaluation tool and the psychological labels corresponding to the three, and construct a training sample set for the psychological state evaluation model; A PMI matrix construction module, which is used to preprocess the collected video data, voice data and text data, extract deep features of each modality, and construct a PMI matrix of text expression; A feature fusion module, which uses the text PMI matrix to optimize the attention mechanism of the text modality and constructs a triple attention fusion mechanism to achieve the fusion of the multi-modal deep features of the subject; A model training module, wherein the model training module is used to train the mental state evaluator through multimodal fusion features and obtain a mental state evaluation model by adjusting parameters; A psychological assessment module is used to perform psychological assessment through a psychological state assessment model.

Images