Psychological condition recognition method based on multi-modal data

Abstract

The application discloses a psychological condition recognition method based on multi-modal data, comprising the following steps: S1, using a linear space-time detector to improve the quality of video and audio features; S2, using a cross-modal time aligner to ensure time synchronization between modes; S3, using Transformers to detect key segments, and then obtaining the final prediction result through a linear layer; and inputting the prediction result. The problems that the existing deep learning method has low recognition accuracy for psychological conditions are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer artificial intelligence technology, and in particular relates to a method for recognizing psychological states based on multimodal data. Background Technology

[0002] Studies have shown significant differences between individuals with mental health issues and healthy individuals in speech, facial expressions, and their respective physiological signals. Integrating multimodal data can improve diagnostic accuracy compared to using single-modal data. Therefore, research on automated mental health diagnosis using multimodal data is in its early stages, and feature extraction and modality fusion are two core steps and also the challenges in this research.

[0003] For feature extraction, the core purpose of this step is to better uncover the information contained in the data. Early feature extraction work typically extracted handcrafted features from audio and video modalities and then used traditional classification models for psychological state detection. However, experimental results from related studies show that handcrafted features ignore a lot of information contained in the original data and cannot make a relatively accurate diagnosis of psychological state. Using 2D convolutional neural networks (2DCNNs) to obtain spatial information from speech spectrograms treats continuous data as discrete data, completely ignoring the temporal information in the data. When using 3D convolutional neural networks (3DCNNs) for feature extraction, the range of temporal information that 3D convolutional networks can capture is limited by the size of the convolutional kernel.

[0004] For modality fusion, research using modality cross-attention to capture additional information generated by interactions between different modalities is gaining traction. However, the traditional Transformer assumes that all spatial dimensions within the same temporal dimension are of equal importance, and then treats different spatial dimensions within the same temporal dimension as a single unit (token) to calculate temporal attention. This assumption applies to unimodal data, but in multimodal data, the relationships between modalities may be more complex. This problem is more pronounced in psychological state recognition. Due to the specific data acquisition methods, we believe that the importance of information provided by features from different modalities is not perfectly aligned temporally. For example, in certain reading-based stimulus experiments, the auditory modality can provide a wealth of information. While the visual modality can also provide information, it cannot provide useful information for most of the experimental task like the auditory modality. Forcing the fusion of useful auditory information and useless visual information at many times may actually weaken the information intensity provided by the auditory modality. Summary of the Invention

[0005] The purpose of this invention is to provide a psychological state recognition method based on multimodal data to solve the problem of low accuracy in psychological state recognition of existing deep learning methods.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a method for identifying psychological states based on multimodal data, comprising the following steps:

[0007] Step S1: Use a linear spatiotemporal detector to improve the quality of video and audio features;

[0008] Step S2: Use a cross-modal time aligner to ensure time synchronization between modes;

[0009] Step S3: Use Transformers to detect key segments, and then pass them through a linear layer to obtain the final prediction result; input the prediction result.

[0010] The beneficial effects of this invention are:

[0011] (1) In order to emphasize the spatiotemporal information in the features and provide high-quality features for subsequent modules, this invention introduces a linear spatiotemporal detector (LSTD) to enhance the features after the feature extraction step, thereby improving the quality of multimodal features.

[0012] (2) In order to align the information importance of multimodal features in the time dimension, a cross-modal time alignment module (CMTA) was obtained by improving the traditional attention mechanism. The traditional time attention mechanism was also applied to the psychological state recognition task in order to better discover the key information in the data. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of a spatiotemporal information detector based on a linear model according to an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of an alignment model based on multimodal temporal attention according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention discloses a method for identifying psychological states based on multimodal data, comprising the following steps:

[0018] Step S1: Use a linear spatiotemporal detector (LSTD) to improve feature quality;

[0019] Step S2: Use the cross-modal time aligner (CMTA) to ensure time synchronization between modes;

[0020] Step S3: Use Transformers to detect key segments, and then pass them through a linear layer to obtain the final prediction result.

[0021] Step S4: Input the prediction results.

[0022] Further, step S1 includes the following steps:

[0023] Step S11: Pre-trained supervised feature extractor for video and audio modalities;

[0024] Step S12: Extract audio features and video image features using a feature extractor;

[0025] Step S13: Use the proposed linear spatiotemporal detector to perform feature enhancement;

[0026] Further, step S13 includes the following steps:

[0027] Step S131: Perform sequence decomposition on the multimodal features;

[0028] Step S132: Capture temporal and spatial information of the subsequences from sequence decomposition to obtain temporal enhancement features and spatial enhancement features;

[0029] Step S133: Add the temporal enhancement features, spatial enhancement features, and original features to obtain the final output of the linear spatiotemporal detector;

[0030] Further, step S2 includes the following steps:

[0031] Step S21: Perform modal splitting on the multimodal features to recover the single-modal data;

[0032] Step S22: Normalize each single-mode data to eliminate the influence of dimensions;

[0033] Step S23: Simultaneously input the single-modal data of audio and video into the cross-modal time alignment module for modal alignment;

[0034] Step S24: After inverse normalizing each single-modal data, the data are spliced ​​together again to form multimodal data;

[0035] Further, step S3 includes the following steps:

[0036] Step S31: Use the Transformer encoder to encode the features and discover key segments;

[0037] Step S32: Input the encoding results into the linear layer for regression or classification prediction.

[0038] Example 1

[0039] By learning from large amounts of psychological data, this method can quickly extract discriminative features and rapidly and objectively determine the psychological state of subjects. Based on a linear spatiotemporal detector and a cross-modal time aligner, the core algorithm of this invention can be widely applied in medical and health institutions, providing the public with rapid, accurate, and convenient testing services. By providing accurate auxiliary testing, it promotes early screening for mental illnesses and drives the development of the field of mental health in my country.

[0040] The specific steps are as follows:

[0041] Step S1: Use a linear spatiotemporal detector (LSTD) to improve feature quality;

[0042] Step S2: Use the cross-modal time aligner (CMTA) to ensure time synchronization between modes;

[0043] Step S3: Use Transformers to detect key segments, and then pass them through a linear layer to obtain the final prediction result.

[0044] Further, step S1 includes the following steps:

[0045] Step S11: Pre-train supervised feature extractors for each modality, and use the feature extractors to extract audio features and video image features;

[0046] Step S12: Use the proposed linear spatiotemporal detector to perform feature enhancement;

[0047] S11 specifically refers to:

[0048] For the video modality, the OpenFace toolkit was first used to perform face alignment and background removal on the original videos in the dataset, aiming to transform the original videos into frames with clearer facial information. Next, the IResNet50 framework, which had been pre-trained on a large amount of face data, was used as a baseline. Then, video frames obtained from two datasets were used to fine-tune it, resulting in two video feature extractors. Finally, each frame was input into the corresponding video feature extractor for feature extraction.

[0049] For audio modalities, the MoviePy toolkit is first used to extract audio files from the original video and perform preprocessing and sliding window segmentation. Then, the PANN_16K framework, which has been pre-trained on a large amount of audio data, is used as a benchmark. The pre-processed and segmented audio segments are then used to fine-tune the framework to obtain an audio feature extractor. Finally, each processed audio segment is input into the corresponding audio feature extractor for feature extraction.

[0050] After the above operations, each sample of the video is converted into... Each sample in the audio modality is converted into Where T represents the time step, for video modality T is the total number of frames in the sample, and for audio modality T is the total number of segments into which the original audio in the sample is divided. D is the feature dimension of the mapping, with a default value of 128. Furthermore, multimodal features are obtained by concatenating single-modal features along their feature dimensions. It is the set of real numbers.

[0051] S12 specifically refers to:

[0052] The overall architecture of the linear spatiotemporal detector is as follows Figure 1 As shown in the LSTD architecture diagram, each feature is first divided into two subsequences, which are then processed by different linear modules. Finally, all processing results are summed to obtain the final enhanced feature. Since the LSTD module always operates on the first dimension (the time dimension in the original feature), when capturing spatial information, the original feature needs to be transposed and fed into the module so that the module operates on the feature mapping dimension.

[0053] The computation process of a linear spatiotemporal detector will be illustrated using the temporal information capture branch of a single-modal feature as an example. The calculation method for multimodal features is the same as that for single-modal features. The spatial information capture branch only transposes the features upon input; the remaining calculations are the same as those for the temporal information capture branch.

[0054] First, the basic features extracted are decomposed into two subsequences: the average moving distance (Moving) and the overall trend (Trend). For the input basic features... The specific processing procedure is as follows:

[0055] X m =[AvgPool1d(Padding(X)] T )] T #(1)

[0056] X t =XX m #(2)

[0057] In the above formula, X m X is the average distance traveled. t To represent the overall trend, X represents the basic single-modal feature for completing feature extraction; AvgPool1d indicates the application of average pooling operation on one-dimensional data, and T indicates the transpose operation;

[0058] If we assume the pooling window size is K, then by copying the first and last lines of the input [(K-1) / / 2] times and then concatenating them line by line to the beginning and end of the original input, we obtain the padded features. Next, a one-dimensional average pooling layer is used to obtain the average moving distance X. m Because of the padding at the beginning and end, Then, using the single-modal fundamental features X and X... m Subtraction yields the overall trend of change. The following text summarizes these three steps as: X m X t =SeriesDecomp(X). For ease of description, SeriesDecomp() will be used in the following text to uniformly represent the decomposition operation of time series.

[0059] Next, a linear module (containing sequence decomposition and a linear module for capturing temporal and spatial information) is used to capture spatiotemporal information. Each module contains two fully connected layers with identical input and output shapes to process the two subsequences obtained from sequence decomposition, respectively. For each additional mode, two more such linear modules are set up, one for capturing temporal information and the other for capturing spatial information for that mode.

[0060] After capturing spatiotemporal information, the enhancement results of the two subsequences are added together to obtain the temporal and spatial enhancement features. Finally, the temporal and spatial enhancement results are added together with the original features to obtain the final enhancement features. The specific processing procedure is as follows:

[0061] X Tm XTt =SeriesDecomp(X)#(3)

[0062] X Sm X St =SeriesDecomp(X T )#(4)

[0063]

[0064]

[0065]

[0066] In the above formula, the superscript T denotes the transpose operation, and the subscript T indicates the time feature. Original Feature These represent the average moving distance (Moving) and the overall trend (Trend) subsequences obtained from time series decomposition, respectively. The spatial sequence is decomposed into two subsequences: the average moving distance and the overall trend. The weights of the fully connected layer, This represents the bias term of the fully connected layer. Every two fully connected layers form a linear module, where i is the number of modes and N represents the total number of modes. These are temporal enhancement features and spatial enhancement features. To avoid losing other information in the original features during the capture of spatiotemporal information, the original basic features are added to the final features. This is the final enhancement feature.

[0067] S2 specifically refers to:

[0068] The overall architecture of the cross-modal time aligner is as follows: Figure 2 As shown, the cross-modal time aligner is only used when the input features are multimodal. Therefore, we will use audio and video bimodal features as an example to illustrate the operation process of the cross-modal time aligner. For ease of explanation, V in the following text refers to the relevant features of the video modality, and A refers to the relevant features of the audio modality, whether as variables or subscripts.

[0069] The cross-modal time aligner consists of several parts: modality decomposition, modality normalization, cross-modal time alignment attention layer, and inverse normalization.

[0070] First, the multimodal data is split into multiple unimodal vectors, and each vector is normalized to eliminate the influence of dimensions and scale between different modalities. Then, the normalized unimodal data is embedded and simultaneously fed into the cross-modal attention layer of the cross-modal temporal aligner for modal alignment. Finally, only a single fully connected layer is used as the decoder. For multimodal features... Its processing before entering the attention layer is shown in equations (8) to (12):

[0071] V, A = ModalSplit(M)#(8)

[0072]

[0073]

[0074] V e =Embedding((V n ) T )#(11)

[0075] A e =Embedding((A n ) T )#(12)

[0076] To extract multimodal features The audio and video single-modal features obtained by direct decomposition. This represents the result of normalizing the single-modal features in the time dimension. Var(V) and Var(A) represent the variances of the video and audio features, respectively, while mean(V) and mean(A) represent the mean values ​​of the video and audio features, respectively.

[0077] mean v stdev v , These are the mean and standard deviation of the video modality (V) and audio modality (A) in the time dimension, respectively. For ease of use in subsequent inverse normalization operations, these four vectors will be copied in the time dimension to make them identical in shape to the original input, i.e.: This is the result of embedding single-modal features in the time dimension, where d_model is a hyperparameter.

[0078] After modality normalization and vector embedding are completed, the features are fed into the encoder, which contains L layers of cross-modal temporal alignment attention modules. The calculation process in a single module is shown in formulas (13) to (18):

[0079] V 0 =W hv Ve +b hv #(13)

[0080] A 0 =W ha A e +b ha #(14)

[0081]

[0082]

[0083]

[0084] FFN(x)=max(0,xW1+b1)W2+b2#(18)

[0085] Among them, W hv W ha b hv b ha These represent the weights and biases of the linear layers used for multi-head mapping of the video and audio modalities, respectively. l V l These represent the intermediate results of audio and video modal features after processing by the previous attention layer, respectively. A d V This represents the dimension of the query and key vectors in the attention mechanism, used to scale the dot product result and prevent the value from becoming too large. L represents the total number of attention layers and is a hyperparameter. In the formula... This is the result of dividing the input into multiple heads using a fully connected layer. These are the cross-modal attention results for video paying attention to audio and audio paying attention to video, respectively, where l represents the current attention module layer number. Formulas (17) and (18) are calculated using the same methods as the layer normalization and feedback network in the traditional Transformer. Mean(x) and Var(x) represent the mean and variance of the input. LayerNorm(x) represents the layer normalization operation, used to stabilize the training of the model and help the model converge faster at different layers. FFN(x) represents the feedback network in the classic Transformer, which includes two fully connected layers and a non-linear activation function (usually ReLU). Its function is to further extract and transform the representation at each position, increasing the non-linear feature representation capability of the model.

[0086] After the encoding operation is completed, for the encoding result of each modality, a fully connected layer is first used to merge the previously divided multi-head attention, and then a decoder is used to transform its dimensions so that it has the same shape as the original input unimodal feature. Next, inverse normalization is performed on the unimodal to restore the original data distribution. Finally, the unimodal decoding results are concatenated according to the mapped dimensions to restore the multimodal form. The specific processing of the above process is as follows:

[0087] V eer =w ev (V L-1 )+b ev #(19)

[0088] A enc =w ea (A L-1 )+b ea #(20)

[0089] V dec =(W dv *V enc +b dv ) T #(twenty one)

[0090] A dec =(W da *A enc +b da ) T #(twenty two)

[0091] X aliqn =Concat(stdev) v *V dec +mean v stdev a *A dec +mean a )#(twenty three)

[0092] In the formula, w ev w ea b ev b ea V represents the weights and biases of the linear layer used for multi-head merging of video and audio features, respectively. L-1 A L-1 W represents the output of the last attention layer in a cross-modal attention encoder. dv W da b dv b da These represent the weights and biases of the linear layers used for decoding video and audio, respectively. To achieve multi-head merging encoder output, For decoder output, This refers to the multimodal features that are ultimately output by the cross-modal time aligner.

[0093] S3 specifically refers to:

[0094] When processing multimodal data, a cross-modal time alignment module is first used to align the information importance of different modalities before key information detection is performed. After the Transformer encoder completes key information detection, only a single fully connected layer (projection) is used to perform final regression prediction on the detection results for multimodal input. The specific processing procedure is as follows (the calculation procedure for single-modal input is the same):

[0095] X enc =TransformerEncoder(X)#(24)

[0096]

[0097] The X is the encoding result of the Transformer encoder, before performing regression prediction. enc The average is calculated along the feature map dimension, i.e.: Then, the data is fed into Projection for regression over time. TransformerEncoder represents the encoder in the classic Transformer, and Projection represents the fully connected layer used for regression prediction. This indicates the final predicted psychological state level score.

[0098] As shown in Tables 1 and 2, to illustrate the effectiveness of this embodiment, the performance of the method based on this invention was compared with that of other advanced methods in the art across various modalities. Based on the number of modalities used, these comparisons can be categorized into three main types: video-based psychological state detection, audio-based psychological state detection, and "audio + video" multimodal psychological state detection.

[0099] Comparison based on video modalities: At AVEC2013, of the two state-of-the-art methods listed in the research, the first proposed a novel temporal pooling technique to capture spatiotemporal information in the video and encode it into an image map, which was then fed into a two-stream network along with the original image information. The second method proposed using a multi-scale 3D-CNN to model the spatiotemporal information. The method based on this invention achieves slightly better performance than the two aforementioned works, thanks to the effectiveness of the LSTM module in modeling spatiotemporal information. At AVEC2014, of the two state-of-the-art methods listed in the research, the second method also models the spatiotemporal information in the video using a spatiotemporal attention mechanism (STA). Compared to AVEC2013, the method based on this invention significantly outperforms these two state-of-the-art methods at AVEC2014.

[0100] Comparison based on audio modality: At AVEC2013, among the three state-of-the-art methods, the first method used multiple basic handcrafted audio features and machine learning methods for prediction. The second method used spectrograms and proposed attention blocks and vector blocks on the time-frequency channels for prediction. The third method used raw audio as input and made predictions through a multi-convolutional architecture. At AVEC2014, more excellent work emerged, with the first method using both raw audio and spectrograms as input to capture speaker recognition features and emotional features, respectively. The second method input motion features, spectrograms, and MFCCs into a three-stream model. The method based on this invention is slightly better than the aforementioned state-of-the-art methods in audio single-modality, mainly due to two aspects. First, the fine-tuned feature extractor is more suitable for the audio characteristics of individuals with psychological problems. Second, the linear spatiotemporal detector effectively highlights spatiotemporal features, providing higher-quality features for subsequent model predictions.

[0101] Based on multimodal comparison: When using multimodal data, the state-of-the-art methods on both datasets employed a Spatiotemporal Attention Network (STA) and an Energy Evolution (EEP) method to fuse multimodal features. The method based on this invention comprehensively outperforms all multimodal methods listed in Table 2. This is attributed to the CMTA module's readjustment of the relationships between modalities, which helps the Transformer better detect key information. The method based on this invention also surpasses other modality fusion methods that directly compute attention in the temporal dimension, thus demonstrating the effectiveness of aligning multimodal information.

[0102] Table 1 Comparison with state-of-the-art methods at AVEC 2013

[0103]

[0104] Table 2 Comparison with state-of-the-art methods at AVEC2014

[0105]

[0106] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for mental state recognition based on multi-modal data, characterized in that, Comprising the following steps: Step S1, using a linear space-time detector to improve the quality of video, audio features; The step S1 comprises the following steps: Step S11, pre-training the supervised feature extractor of video and audio modal; Step S12, using the feature extractor to extract audio features and video image features; Step S13, using a linear space-time detector to strengthen the features; The step S13 comprises the following steps: Step S131, sequence decomposition is performed on the multi-modal features; Step S132, time information capture and spatial information capture are performed on the sequence-decomposed sub-sequences respectively to obtain time-enhanced features and spatial-enhanced features; Step S133, the time-enhanced features, the spatial-enhanced features and the original features are added to obtain the final output of the linear space-time detector; Step S2, using a cross-modal time aligner to ensure time synchronization between modalities; Step S3, using Transformers to detect key segments, and then passing through a linear layer to obtain the final prediction result; output the prediction result; In the S3, the specific processing is as follows: The encoding result of the Transformer encoder, before regression prediction is performed, is Average in the feature mapping dimension, i.e. Then fed into Projection to perform regression in the time dimension, denotes the encoder in the classic Transformer, denotes the fully connected layer for regression prediction, denotes the final predicted mental state rating, is the multi-modal feature output by the cross-modal time aligner. 2.The method for recognizing a psychological condition based on multi-modal data according to claim 1, characterized in that, The step S2 comprises the following steps: Step S21, modal splitting is performed on the multi-modal features to restore single-modal data; Step S22, each single-modal data is normalized respectively to eliminate the dimension effect; Step S23, the single-modal data of audio and video are simultaneously input into the cross-modal time alignment module for modal alignment; Step S24, each single-modal data is inversely normalized and then spliced into multi-modal data again. 3.The method of claim 1, wherein, In the S11, for the video modality, first, the original video in the data set is aligned and the background is removed using the OpenFace tool package, and the IResNet50 framework is used as the benchmark, then the video frames obtained from two data sets are used for fine tuning respectively to obtain two video feature extractors, and finally each frame is input into the corresponding video feature extractor for feature extraction; For the audio modality, first, the audio file is extracted from the original video using the MoviePy tool package and preprocessed and segmented by sliding window, then the PANN_16K framework is used as the benchmark, and the audio segments obtained after preprocessing and segmentation are used for fine tuning respectively to obtain an audio feature extractor, and finally each processed audio segment is input into the corresponding audio feature extractor for feature extraction; Each sample of the video is transformed into Each sample of the audio modality is transformed into where T denotes the time step, D is the dimension of the mapped features, and the multimodal features are obtained by concatenating the single-modality features in the feature dimension is a set of real numbers. 4.The method of claim 1, wherein, In the S12, firstly, the basic features after feature extraction are subjected to sequence decomposition, and the basic features are decomposed into two subsequences of average moving distance and overall change trend. The specific processing process is as follows: In the above formula, is the average moving distance, is the overall change trend, is a single-modal base feature for completing feature extraction; represents an average pooling operation applied on one-dimensional data, T represents a transpose operation; The pooling window size is K, and the first and last rows of the input are copied first After that, the first and last rows of the original input are spliced respectively to obtain the filled feature , and then a one-dimensional average pooling layer is used to obtain the average moving distance Since the first and last rows are filled, the last row of the input is ; then subtract the single-modal basic feature from to obtain the overall trend ; Next, a linear module is used to capture space-time information, and each module includes two fully connected layers with consistent input and output shapes for processing the two sub-sequences obtained by sequence decomposition respectively, which are used for time information capture and spatial information capture of the modality respectively; After completing the space-time information capture, the enhanced results of the two sub-sequences are added to obtain the time and spatial enhanced features, and finally the time and spatial enhanced results and the original features are added to obtain the final enhanced features, and the specific processing process is as follows: In the above formula, the upper subscript represents the transpose operation, and the lower subscript T is the identification of the time feature, and the original feature respectively represent the average moving distance and the overall change trend of the two sub-sequences obtained by time series decomposition, respectively represent the average moving distance and the overall change trend of the two sub-sequences obtained by space series decomposition, is the weight of the fully connected layer, is the bias term of the fully connected layer, and each two fully connected layers form a linear module, is the number of each modality, and N represents the total number of modalities, respectively represent the time-enhanced feature and the space-enhanced feature; is the final enhanced feature.

5. The psychological condition recognition method based on multi-modal data according to claim 1, characterized in that, Firstly, the multi-modal data is split into multiple single-modal vectors, and each of them is normalized. The normalized single-modal data is embedded and sent into the cross-modal attention layer of the cross-modal time aligner for modal alignment. Finally, a fully connected layer is used as a decoder for the multi-modal features The processing before entering the attention layer is shown in equations (8) to (12): audio and video features, audio and video features, audio and video features, audio and video features, audio and video features; audio and video features, audio and video features, hyperparameters; After the completion of modal normalization and vector embedding, the features are sent into the encoder, which contains L layers of cross-modal time alignment attention modules. The calculation process in a single module is shown in equations (13) to (18): wherein, respectively represent the weights and biases of the linear layer for multi-head mapping of the video modality and the audio modality; respectively represent the intermediate results of the audio modality and the video modality features after being operated by the previous attention layer; represents the dimension of the query and key vectors in the attention mechanism; represents the total number of attention layers, which is a hyperparameter; in the formula is the result of dividing the input into multiple heads by a fully connected layer, respectively represent the cross-modal attention results of the video paying attention to the audio and the audio paying attention to the video, represents the number of the attention module currently located in; Mean(x), Var(x) represent the mean and variance of the input, and LayerNorm(x) represents the layer normalization operation; FFN(x) represents the pre-feedback network in the classic Transformer. After the end of the encoding operation, for the encoding result of each modality, the previously divided multi-head attention is first combined using a fully connected layer, and then the decoder is used to convert its dimensions so that it is the same as the original input single-modal feature shape. The single modality is inversely normalized to restore the original data distribution. Finally, the single-modal decoding result is spliced according to the mapping dimension to restore the multi-modal form. The specific processing of the above process is as follows: In the formula, These represent the weights and biases of the linear layer used for multi-head merging of video and audio features, respectively. This represents the output of the last attention layer in a cross-modal attention encoder. These represent the weights and biases of the linear layers used for decoding video and audio, respectively. To achieve multi-head merging encoder output, For decoder output, This refers to the multimodal features that are ultimately output by the cross-modal time aligner.

Images