Psychological condition identification method based on multi-modal data

Abstract

The invention discloses a psychological condition recognition method based on multi-modal data. The method comprises the steps that S1, video and audio feature quality is improved through a linear space-time detector; s2, ensuring time synchronization between modals by using a cross-modal time aligner; and S3, detecting the key segment by using Transformers, and obtaining a final prediction result through a linear layer. And inputting a prediction result. The problem that an existing deep learning method is not high in psychological condition recognition precision is solved.

Description

Technical Field

[0001] The present invention belongs to the field of computer artificial intelligence technology, and in particular relates to a psychological condition recognition method based on multimodal data. Background Art

[0002] Studies have shown that individuals with mental problems and healthy individuals have significant differences in speech, facial expressions, and their respective physiological signals. Compared with using single-modal data, integrating data from multiple modalities can improve diagnostic accuracy. Therefore, research on automatic diagnosis of mental conditions using multimodal data is in the ascendant, and feature extraction and modality fusion are the two core steps in multimodal automatic diagnosis of mental conditions research, and also the challenges.

[0003] For feature extraction, the core purpose of this step is to better explore the information contained in the data. Early feature extraction work usually extracted manual features from audio and video modalities, and then used traditional classification models to detect psychological conditions. However, the experimental results of related studies show that manual features ignore more information contained in the original data and cannot make a more accurate diagnosis of psychological conditions; using 2D convolutional neural networks (2DCNN) to obtain spatial information from speech spectrograms treats continuous data as discrete data, completely ignoring the temporal information in the data; using 3D convolutional neural networks (3D CNN) for feature extraction is limited by the size of the convolution kernel, and the range of temporal information that 3D convolutional networks can capture is also very small.

[0004] For modal fusion, in order to obtain additional information generated by the interaction between different modalities, the study of modal fusion using modal cross attention has gradually emerged. However, the traditional Transformer assumes that the importance of all spatial dimensions on the same time dimension is always the same, and then uses different spatial dimensions on the same time dimension as a minimum unit (token) to calculate temporal attention. This assumption is applicable to unimodal data, but in multimodal data, there may be more complex relationships between modalities. This problem is more obvious in psychological condition recognition. Due to the special data collection method, we believe that the importance of information provided by features of different modalities is not completely aligned in time. For example, when conducting some stimulation experiments such as reading aloud, the sound modality can provide a lot of information. Although the visual modality can also provide information, it cannot provide useful information for most of the time in this experimental task like the sound modality. If useful sound information and useless visual information are forced to be fused at many timestamps, this may weaken the strength of the information provided by the sound. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method for identifying psychological conditions based on multimodal data to solve the problem that the existing deep learning methods have low accuracy in identifying psychological conditions.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is a method for identifying psychological conditions based on multimodal data, comprising the following steps:

[0007] Step S1, using a linear spatiotemporal detector to improve the quality of video and audio features;

[0008] Step S2: using a cross-modal time aligner to ensure time synchronization between modalities;

[0009] Step S3: Use Transformers to detect key segments, and then pass through the linear layer to obtain the final prediction result; input the prediction result.

[0010] Beneficial effects of the present invention:

[0011] (1) In order to emphasize the spatiotemporal information in features and provide high-quality features for subsequent modules, the present invention introduces a linear spatiotemporal detector (LSTD) to enhance features after the feature extraction step, thereby improving the quality of multimodal features.

[0012] (2) In order to align the information importance of multimodal features in the temporal dimension, the cross-modal temporal alignment module (CMTA) was obtained by improving the traditional attention mechanism. The traditional temporal attention mechanism was also applied to the psychological condition recognition task to better discover the key information in the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 It is a schematic diagram of a spatiotemporal information detector based on a linear model according to an embodiment of the present invention.

[0015] Figure 2 It is a schematic diagram of an alignment model based on multimodal temporal attention according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The embodiment of the present invention discloses a method for identifying a psychological condition based on multimodal data, comprising the following steps:

[0018] Step S1, using a linear spatiotemporal detector (LSTD) to improve feature quality;

[0019] Step S2: using a cross-modal time aligner (CMTA) to ensure time synchronization between modalities;

[0020] Step S3: Use Transformers to detect key segments, and then pass the linear layer to get the final prediction result.

[0021] Step S4: input prediction results.

[0022] Furthermore, the step S1 includes the following steps:

[0023] Step S11, pre-training supervised feature extractors of video and audio modalities;

[0024] Step S12: extracting audio features and video image features using a feature extractor;

[0025] Step S13, using the proposed linear spatiotemporal detector to perform feature enhancement;

[0026] Furthermore, the step S13 includes the following steps:

[0027] Step S131, performing sequence decomposition on the multimodal features;

[0028] Step S132, respectively capturing time information and space information of the subsequences decomposed by the sequence to obtain time enhancement features and space enhancement features;

[0029] Step S133, adding the time enhancement feature, the space enhancement feature and the original feature to obtain the final output of the linear spatiotemporal detector;

[0030] Furthermore, the step S2 comprises the following steps:

[0031] Step S21, performing modal splitting on the multimodal features to restore single-modal data;

[0032] Step S22, normalizing each single modal data to eliminate the dimension effect;

[0033] Step S23: inputting the single-modal data of audio and video into the cross-modal time alignment module at the same time to perform modal alignment;

[0034] Step S24, performing inverse normalization on each single-modal data and then splicing it into multi-modal data again;

[0035] Furthermore, step S3 includes the following steps:

[0036] Step S31: Use the Transformer encoder to encode the features and discover key segments;

[0037] Step S32: Send the encoding result to the linear layer for regression or classification prediction.

[0038] Example 1

[0039] By learning a large amount of psychological data, this method can quickly extract discriminative features and quickly and objectively determine the psychological condition of the subject. The present invention is based on a linear spatiotemporal detector and a cross-modal time aligner. The core algorithm can be widely used in medical and health institutions to provide rapid, accurate and convenient testing services for the general public. By providing accurate auxiliary testing, it can promote the early screening of mental illness and promote the development of my country's mental health field.

[0040] The specific steps are as follows:

[0041] Step S1, using a linear spatiotemporal detector (LSTD) to improve feature quality;

[0042] Step S2: using a cross-modal time aligner (CMTA) to ensure time synchronization between modalities;

[0043] Step S3: Use Transformers to detect key segments, and then pass the linear layer to get the final prediction result.

[0044] Furthermore, the step S1 includes the following steps:

[0045] Step S11, pre-training a supervised feature extractor for each modality, and using the feature extractor to extract audio features and video image features;

[0046] Step S12: using the proposed linear spatiotemporal detector to perform feature enhancement;

[0047] S11 is as follows:

[0048] For the video modality, we first used the OpenFace toolkit to perform face alignment and background removal on the original videos in the dataset, with the goal of converting the original videos into frames with clearer facial information. Next, we used the IResNet50 framework, which has been pre-trained on a large amount of face data, as a benchmark, and then used the video frames obtained from the two datasets to fine-tune it, respectively, to obtain two video feature extractors, and finally input each frame into the corresponding video feature extractor for feature extraction.

[0049] For the audio modality, we first use the MoviePy toolkit to extract audio files from the original video and perform preprocessing and sliding window segmentation. Then, we use the PANN_16K framework, which has been pre-trained on a large amount of audio data, as a benchmark. We then use the audio segments obtained after preprocessing and segmentation to fine-tune them to obtain an audio feature extractor. Finally, each processed audio segment is input into the corresponding audio feature extractor for feature extraction.

[0050] After the above operations, each sample of the video is converted into Each sample in the audio modality is converted to Where T represents the time step. For the video modality, T is the total number of frames in the sample. For the audio modality, T is the total number of segments into which the original audio in the sample is divided. D is the feature dimension of the mapping, and the default value is 128. In addition, multimodal features are obtained by concatenating single-modal features on the feature dimension. is the set of real numbers.

[0051] S12 is specifically:

[0052] The overall architecture of the linear space-time detector is as follows Figure 1 As shown in the figure, the LSTD architecture diagram shows that each feature is first divided into two subsequences, and then processed by different linear modules respectively, and finally all the processing results are added to obtain the final enhanced feature. Since the LSTD module always operates on the first dimension (the time dimension in the original feature), when capturing spatial information, the original features need to be transposed and sent to the module so that the module can operate on the feature mapping dimension.

[0053] The temporal information capture branch of a unimodal feature is used as an example to illustrate the operation process of the linear spatiotemporal detector. The multimodal feature is calculated in the same way as the unimodal feature. The spatial information capture branch only transposes the feature at input, and the rest of the calculation is the same as the temporal information capture branch.

[0054] First, the basic features of feature extraction are sequenced and decomposed into two subsequences: average moving distance (Moving) and overall change trend (Trend). The specific processing process is as follows:

[0055] X m =[AvgPool1d(Padding(X) T )] T #(1)

[0056] X t =XX m #(2)

[0057] In the above formula, X m is the average moving distance, X t is the overall trend of change, X is the single-modal basic feature for feature extraction; AvgPool1d represents the average pooling operation applied to one-dimensional data, and T represents the transposition operation;

[0058] Assuming that the pooling window size is K, since the first and last lines of the input are copied [(K-1) / / 2] times and then spliced ​​to the beginning and end of the original input respectively, the padded features are obtained. Next, use a one-dimensional average pooling layer to get the average moving distance X m , because of the head and tail padding, Then use the unimodal basic features X and X m Subtract to get the overall trend In the following, these three steps are summarized as follows: m , X t = SeriesDecomp(X). For ease of description, SeriesDecomp() is used below to uniformly represent the decomposition operation of time series.

[0059] Next, a linear module (a linear spatiotemporal detector contains sequence decomposition and linear modules for capturing temporal and spatial information) is used to capture spatiotemporal information. Each module contains two fully connected layers with the same input and output shapes to process the two subsequences obtained by sequence decomposition. For each additional modality, two more such linear modules are set up, one for capturing temporal information and the other for capturing spatial information of the modality.

[0060] After completing the capture of spatiotemporal information, the enhancement results of the two subsequences are added together to obtain the enhanced features of time and space. Finally, the enhanced results of time and space and the original features are added together to obtain the final enhanced features. The specific processing process is as follows:

[0061] X Tm , XTt =SeriesDecomp(X)#(3)

[0062] X Sm , X St = SeriesDecomp(X T )#(4)

[0063]

[0064]

[0065]

[0066] In the above formula, the superscript T indicates the transposition operation, and the subscript T is the identifier of the time feature. They represent the two subsequences of the average moving distance (Moving) and the overall change trend (Trend) obtained by decomposing the time series. The two subsequences are the average moving distance (Moving) and the overall change trend (Trend) obtained by decomposing the spatial sequence. is the weight of the fully connected layer, Represents the bias term of the fully connected layer. Every two fully connected layers form a linear module, i is the number of modes, and N is the total number of modes. They are time-enhanced features and space-enhanced features. In order to avoid the loss of other information in the original features during the process of capturing spatiotemporal information, the original basic features are added to the final features. The final enhanced feature.

[0067] S2 is specifically:

[0068] The overall architecture of the cross-modal time aligner is as follows Figure 2 As shown in , the cross-modal time aligner is only used when the input features are multimodal, so we will take the audio and video bimodal features as an example to illustrate the operation process of the cross-modal time aligner. For the convenience of description, whether it is a variable or a subscript, V in the following text represents the relevant features of the video modality, and A represents the relevant features of the audio modality.

[0069] The cross-modal time aligner consists of modal decomposition, modal normalization, cross-modal time alignment attention layer, and inverse normalization.

[0070] First, the multimodal data is split into multiple unimodal vectors and normalized to eliminate the dimension and scale effects between different modalities. Then, the normalized unimodal data is embedded and sent to the cross-modal attention layer of the cross-modal time aligner for modal alignment. Finally, only one fully connected layer is used as the decoder. Its processing before entering the attention layer is shown in formulas (8) to (12):

[0071] V, A = ModalSplit (M) # (8)

[0072]

[0073]

[0074] V e =Embedding((V n ) T )#(11)

[0075] A e =Embedding((A n ) T )#(12)

[0076] From multimodal features The audio and video unimodal features are directly decomposed in . It is the result of normalizing the unimodal features in the time dimension. Var(V) and Var(A) represent the variance of video features and audio features respectively, and mean(V) and mean(A) represent the mean of video features and audio features respectively.

[0077] mean v , stdev v , They are the mean and standard deviation of the video mode (V) and audio mode (A) in the time dimension. To facilitate the subsequent inverse normalization operation, these four vectors will be copied in the time dimension to make them the same as the original input shape, that is: It is the result of embedding the unimodal features in the time dimension, and d_model is a hyperparameter.

[0078] After completing modality normalization and vector embedding, the features are fed into the encoder, which contains L layers of cross-modal time alignment attention modules. The calculation process in a single module is shown in formulas (13) to (18):

[0079] V 0 =W hv Ve +b hv #(13)

[0080] A 0 =W ha A e +b ha #(14)

[0081]

[0082]

[0083]

[0084] FFN(x)=max(0,xW1+b1)W2+b2#(18)

[0085] Among them, W hv , W ha 、b hv , b ha A represents the weight and bias of the linear layer used for multi-head mapping of video and audio modalities, respectively. l 、V l Respectively represent the intermediate results of the audio modality and video modality features after being operated by the previous attention layer. A ,d V Indicates the dimension of the query and key vectors in the attention mechanism, which is used to scale the dot product result to prevent the value from being too large. L represents the total number of attention layers, which is a hyperparameter. In the formula This is the result of dividing the input into multiple heads using a fully connected layer. They are the cross-modal attention results of video paying attention to audio and audio paying attention to video, respectively. l represents the current attention module layer. Formulas (17) and (18) are the same as the calculation methods of layer normalization and feedforward network in traditional Transformer. Mean(x) and Var(x) represent the mean and variance of the input. LayerNorm(x) represents the layer normalization operation, which is used to stabilize the training of the model and help the model converge faster at different levels. FFN(x) represents the feedforward network in the classic Transformer. This network includes two fully connected layers and a nonlinear activation function (usually ReLU). Its function is to further extract and transform the representation of each position and increase the nonlinear feature representation capability of the model.

[0086] After the encoding operation is completed, for the encoding results of each modality, the previously divided multi-head attention is first merged using a fully connected layer, and then the decoder is used to transform its dimension so that it has the same shape as the original input unimodal feature. Next, the unimodality is inverse normalized to restore the original data distribution. Finally, the unimodal decoding results are spliced ​​according to the mapping dimension to restore the multimodal form. The specific processing of the above process is as follows:

[0087] V eer =w ev (V L-1 )+b ev #(19)

[0088] A enc =w ea (A L-1 )+b ea #(20)

[0089] V dec =(W dv *V enc +b dv ) T #(twenty one)

[0090] A dec =(W da *A enc +b da ) T #(twenty two)

[0091] X aliqn =Concat(stdev v *V dec +mean v , stdev a *A dec +mean a )#(twenty three)

[0092] In the formula, w ev 、w ea , b ev , b ea Respectively represent the weight and bias of the linear layer used for multi-head merging of video and audio features. L-1 , A L-1 represents the output of the last attention layer in the cross-modal attention encoder, W dv , W da , b dv , b da Represent the weight and bias of the linear layer for decoding video and audio respectively. To complete the encoder output of multi-head merging, is the decoder output, The multimodal features are the final output of the cross-modal time aligner.

[0093] S3 is specifically:

[0094] When processing multimodal data, the cross-modal time alignment module is used to align the importance of information in different modes before detecting key information. After the Transformer encoder completes the key information detection, it still uses only one fully connected layer (Projection) to perform the final regression prediction on the detection results. The specific processing process is as follows (the operation process for single-mode input is the same):

[0095] X enc = TransformerEncoder(X)#(24)

[0096]

[0097] is the encoding result of the Transformer encoder. Before regression prediction, X enc The average is performed in the feature map dimension, that is: Then it is sent to Projection for regression in the time dimension. TransformerEncoder represents the encoder in the classic Transformer, and Projection represents the fully connected layer used for regression prediction. Represents the final predicted psychological status rating score.

[0098] As shown in Table 1 and Table 2, in order to illustrate the effectiveness of this embodiment, the method based on the present invention is compared with the advanced methods in other studies in the field in terms of performance in various modalities. According to the number of modalities used, these comparisons can be divided into three categories: psychological condition detection based on video modality, psychological condition detection based on audio modality, and psychological condition detection based on "audio + video" multi-modality.

[0099] Comparison based on video modality: On AVEC2013, among the two most advanced methods in the listed studies, the first method proposed a new time pool to capture the spatiotemporal information in the video and encode it into the image map, which was then fed into the two-stream network together with the original image information. The second method proposed the use of multi-scale 3D-CNN to model spatiotemporal information. The final performance of the method based on the present invention is slightly better than the above two works, thanks to the effective modeling of spatiotemporal information by the LSTM module. On AVEC2014, among the two most advanced methods in the listed studies, the second method also models the spatiotemporal information in the video through the spatiotemporal attention mechanism (STA). Compared with AVEC2013, the performance of the method based on the present invention on AVEC2014 is far better than these two most advanced methods.

[0100] Comparison based on audio modality: At AVEC2013, among the three most advanced methods, the first method used a variety of audio-based manual features and used machine learning methods for prediction. The second method used spectrograms and proposed attention blocks and vector blocks on the time-frequency channels for prediction. The third method used raw audio as input and made predictions through a multi-convolution architecture. At AVEC2014, more excellent works appeared, among which the first method used both raw audio and spectrograms as inputs to capture speaker identification features and emotional features, respectively. The second method input motion features, spectra, and MFCC into a three-stream model. The method based on the present invention is slightly better than the above-mentioned most advanced methods in audio single modality, mainly due to two aspects. First, the fine-tuned feature extractor is more suitable for the audio characteristics of people with psychological problems. Second, the linear spatiotemporal detector effectively highlights the spatiotemporal features, providing better features for the prediction of subsequent models.

[0101] Comparison based on multimodality: When using multimodal data, the most advanced methods on the two datasets use spatiotemporal attention networks (STA) and feature fusion methods (EEP) to fuse multimodal features. The performance of the method based on the present invention comprehensively surpasses all multimodal methods listed in Table 2, thanks to the CMTA module that readjusts the relationship between modalities and helps Transformer better detect key information. The method based on the present invention can outperform other modal fusion methods that directly calculate attention in the time dimension, and also confirms the effectiveness of aligning multimodal information to a certain extent.

[0102] Table 1 Comparison with the most advanced methods on AVEC2013

[0103]

[0104] Table 2 Comparison with the most advanced methods on AVEC2014

[0105]

[0106] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for identifying psychological conditions based on multimodal data, characterized in that: The following steps are involved: Step S1, using a linear spatiotemporal detector to improve the quality of video and audio features; Step S2: using a cross-modal time aligner to ensure time synchronization between modalities; Step S3: Use Transformers to detect key segments, and then pass through the linear layer to obtain the final prediction result; input the prediction result.

2. The method for identifying psychological conditions based on multimodal data according to claim 1, characterized in that: The step S1 comprises the following steps: Step S11, pre-training supervised feature extractors of video and audio modalities; Step S12: extracting audio features and video image features using a feature extractor; Step S13: Use a linear spatiotemporal detector to perform feature enhancement.

3. The method for identifying psychological conditions based on multimodal data according to claim 2, characterized in that: The step S13 comprises the following steps: Step S131, performing sequence decomposition on the multimodal features; Step S132, respectively capturing time information and space information of the subsequences decomposed by the sequence to obtain time enhancement features and space enhancement features; Step S133: Add the temporal enhancement feature, the spatial enhancement feature and the original feature to obtain the final output of the linear spatiotemporal detector.

4. The method for identifying psychological conditions based on multimodal data according to claim 1, characterized in that: The step S2 comprises the following steps: Step S21, performing modal splitting on the multimodal features to restore single-modal data; Step S22, normalizing each single modal data to eliminate the dimension effect; Step S23: inputting the single-modal data of audio and video into the cross-modal time alignment module at the same time to perform modal alignment; Step S24: Denormalize each unimodal data and then splice it into multimodal data again.

5. The method for identifying psychological conditions based on multimodal data according to claim 1, characterized in that: The step S3 comprises the following steps: Step S31: Use the Transformer encoder to encode the features and discover key segments; Step S32: Send the encoding result to the linear layer for regression or classification prediction.

6. The method for identifying psychological conditions based on multimodal data according to claim 2, characterized in that: In S11, for the video modality, the OpenFace toolkit is first used to perform face alignment and background removal on the original video in the dataset, and the IResNet50 framework is used as a benchmark. Then, the video frames obtained from the two datasets are used to fine-tune them respectively to obtain two video feature extractors, and finally each frame is input into the corresponding video feature extractor for feature extraction; For the audio modality, we first use the MoviePy toolkit to extract audio files from the original video and perform preprocessing and sliding window segmentation. Then, we use the PANN_16K framework as a benchmark, and then use the audio segments obtained after preprocessing and segmentation to fine-tune them to obtain an audio feature extractor. Finally, each processed audio segment is input into the corresponding audio feature extractor for feature extraction. Each sample in the video is converted into Each sample in the audio modality is converted to Among them, T represents the time step, D is the feature dimension of the mapping, and multimodal features are obtained by splicing single-modal features on the feature dimension. is the set of real numbers.

7. The method for identifying psychological conditions based on multimodal data according to claim 2, characterized in that: In S12, first, the basic features after feature extraction are sequenced and decomposed into two subsequences: average moving distance and overall change trend. The specific processing process is as follows: X m =[AvgPool1d(Padding(X) T )] T #(1) X t =X-X m #(2) In the above formula, X m is the average moving distance, X t is the overall trend of change, X is the single-modal basic feature for feature extraction; AvgPool1d represents the average pooling operation applied to one-dimensional data, and T represents the transposition operation; The pooling window size is K. First, the first and last lines of the input are copied [(K-1) / / 2] times and then spliced ​​to the beginning and end of the original input line by line to obtain the filled features. Next, use a one-dimensional average pooling layer to get the average moving distance X m , because of the head and tail padding, Then use the unimodal basic features X and X m Subtract to get the overall trend Next, a linear module is used to capture spatiotemporal information. Each module contains two fully connected layers with the same input and output shapes to process the two subsequences obtained by sequence decomposition, which are used to capture the temporal information and spatial information of the modality respectively. After completing the capture of spatiotemporal information, the enhancement results of the two subsequences are added together to obtain the enhanced features of time and space. Finally, the enhanced results of time and space and the original features are added together to obtain the final enhanced features. The specific processing process is as follows: X Tm ,X Tt =SeriesDecomp(X)#(3) X Sm ,X St =SeriesDecomp(X T )#(4) In the above formula, the superscript T indicates the transposition operation, and the subscript T is the identifier of the time feature. They represent the average moving distance obtained by decomposing the time series, and the two subsequences of the overall change trend. The average moving distance obtained by decomposing the spatial sequence and the two subsequences of the overall change trend are: is the weight of the fully connected layer, represents the bias term of the fully connected layer. Every two fully connected layers form a linear module. i is the number of modes, and N represents the total number of modes. They are time-enhanced features and space-enhanced features respectively; The final enhanced feature.

8. The method for identifying psychological conditions based on multimodal data according to claim 1 or 4, characterized in that: First, the multimodal data is split into multiple unimodal vectors and normalized respectively. The normalized unimodal data is embedded and sent to the cross-modal attention layer of the cross-modal time aligner for modal alignment. Finally, a fully connected layer is used as a decoder to align the multimodal features. The processing before entering the attention layer is shown in formulas (8) to (12): V,A=ModalSplit(M)#(8) V e =Embedding((V n ) T )#(11) A e =Embedding((A n ) T )#(12) From multimodal features The audio and video unimodal features directly decomposed in is the result of normalizing the unimodal features in the time dimension. Var(V) and Var(A) represent the variance of video features and audio features respectively. Mean(V) and mean(A) represent the mean of video features and audio features respectively. v , stdev v , are the mean and standard deviation of the video and audio modalities in the time dimension, It is the result of embedding the unimodal feature in the time dimension, and d_model is a hyperparameter; After completing modality normalization and vector embedding, the features are fed into the encoder, which contains L layers of cross-modal time alignment attention modules. The calculation process in a single module is shown in formulas (13) to (18): V 0 =W hv V e +b hv #(13) A 0 =W ha A e +b ha #(14) FFN(x)=max(0,xW1+b1)W2+b2#(18) Among them, W hv , W ha 、b hv 、b ha Respectively represent the weight and bias of the linear layer used for multi-head mapping of video modality and audio modality; A l 、V l They represent the intermediate results of the audio and video modality features after being calculated by the previous attention layer; d A d V represents the dimension of query and key vector in the attention mechanism; L represents the total number of attention layers, which is a hyperparameter; V in the formula 0 ,A 0 This is the result of dividing the input into multiple heads using a fully connected layer. They are the cross-modal attention results of video paying attention to audio and audio paying attention to video, respectively. l represents the current attention module layer. Mean(x) and Var(x) represent the mean and variance of the input. LayerNorm(x) represents the layer normalization operation. FFN(x) represents the feed-forward network in the classic Transformer. After the encoding operation is completed, for the encoding results of each modality, the previously divided multi-head attention is first merged using a fully connected layer, and then the decoder is used to transform its dimension so that it has the same shape as the original input unimodal feature, and the unimodality is inversely normalized to restore the original data distribution. Finally, the unimodal decoding results are spliced ​​according to the mapping dimension to restore the multimodal form. The specific processing of the above process is as follows: V enc =w ev (V L-1 )+b ev #(19) A enc =w ea (A L-1 )+b ea #(20) V dec =(W dv *V enc +b dv ) T #(21) A dec =(W da *A enc +b da ) T #(22) X align =Concat(stdev v *V dec +mean v ,stdev a *A dec +mean a )#(23) In the formula, w ev 、w ea 、b ev 、b ea Respectively represent the weight and bias of the linear layer used for multi-head merging of video and audio features, V L-1 , A L-1 represents the output of the last attention layer in the cross-modal attention encoder, W dv , W da 、b dv 、b da Represent the weights and biases of the linear layer for decoding video and audio respectively, To complete the encoder output of multi-head merging, is the decoder output, The multimodal features are the final output of the cross-modal time aligner.

9. The method for identifying psychological conditions based on multimodal data according to claim 1 or 5, characterized in that: In S3, the specific processing is as follows: X enc =TransformerEncoder(X)#(24) is the encoding result of the Transformer encoder. Before regression prediction, X enc The average is performed in the feature map dimension, that is: Then it is sent to Projection for regression in the time dimension. TransformerEncoder represents the encoder in the classic Transformer, and Projection represents the fully connected layer for regression prediction. Represents the final predicted psychological status rating score.

Images