Voice and video fusion emotional state recognition method based on IP3S model

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a voice and video fusion emotional state recognition method based on an IP3S model. Comprising the following steps of: 1, establishing an improved shared private subspace separation model IP3S (Imported Private-Shared Subspace); and 2, voice and video fused emotional state recognition based on the IP3S model. According to the method, the problem of voice and video feature fusion in multi-modal emotional state recognition is discussed, and a multi-modal emotional state recognition model is generated. The method comprises the following steps: firstly, establishing a voice and video feature extraction model, respectively extracting voice and video features, then based on an improved shared private subspace separation model IP3S model, fusing the two features, and establishing an emotional state recognition classifier to perform emotional state recognition. The method can effectively solve the problem that the difference between modes is too large in the voice and video fusion process, thereby improving the accuracy of emotion state recognition.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence and relates to an emotion state recognition method based on voice and video fusion of IP3S model. Background Art

[0002] Emotion is one of the psychological activities of human beings, with different emotional states such as happiness, disgust, resentment, liking, depression, etc. These emotional states will affect people's daily lives. Since emotions are contagious, a person's emotional state can have a certain impact on the people around them. In today's society, the fast-paced life has led to a gradual increase in psychological pressure on some people. Many people have experienced emotional instability and long-term depression, which has seriously affected their physical and mental health. Therefore, effectively identifying emotional states can help people self-regulate and reduce or avoid the impact of bad emotional states. At the same time, with the development of artificial intelligence technology, household companion robots will gradually enter people's lives. Effectively identifying the user's emotional state will help the robot better provide users with more emotionally valuable services. Emotional state recognition is also one of the important research contents of emotional computing today.

[0003] A large number of studies have shown that people's speech will produce obvious differences in different emotional states, and some studies have proved the feasibility and effectiveness of using sound to identify emotional states. In addition to sound differences, a series of related studies have also determined that people have differences in video data performance in different emotional states, such as facial expressions, the curvature of the mouth corners, and whether the movements are slow. In the past period of time, deep learning technology has achieved excellent performance in emotional state recognition, and emotional state recognition methods based on multimodal fusion of voice and video data have also become a research hotspot. Summary of the invention

[0004] The invention provides an emotion state recognition method based on speech and video fusion of IP3S model. Aiming at the heterogeneity problem between two modes in the speech and video fusion process, an improved shared private subspace separation model IP3S (Improved Private-Shared Subspaces Separation) is proposed. Then the definition of relevant feature sequence is given. On this basis, an emotion state recognition process based on speech and video fusion of IP3S model is constructed, so that the emotion state recognition can be effectively performed based on speech and video data.

[0005] The technical solution of the present invention to solve the problem is as follows:

[0006] A method for emotional state recognition based on voice and video fusion of IP3S model, comprising:

[0007] Step 1: Establish an improved shared private subspace separation model IP3S (Improved Private–Shared Subspaces)

[0008] Step 2: Emotional state recognition based on voice and video fusion based on IP3S model

[0009] In the first step, the improved shared private subspace separation model IP3S is as follows:

[0010] In order to solve the problem of inconsistent distribution and representation of different modalities in the process of multimodal fusion, a shared private subspace separation model P3S (Private–Shared Subspaces Separation) is proposed. This model proposes to build a common subspace to generate a common representation of different modalities (public representation) to reduce heterogeneity differences, and set a private subspace to separate the interference information of each modality (private representation), so as to make the representation of different modalities in the shared subspace more effective. In view of the characteristics of the emotional state recognition problem based on speech and video fusion, an improved shared private subspace separation model IP3S is proposed based on the P3S model. This model makes the following improvements to the P3S model:

[0011] (1) The P3S model innovatively uses private and shared representations as subspace representations of the original modality features. However, since it considers the private representation as interference information, only the public representation of the two modalities is used as the input of the classifier in the classification process. However, this private representation not only contains interference information, but also contains the content that each modality pays attention to in the process of identifying emotional states (such as the vocal characteristics in the speech data and the facial expressions in the video data involved in the classification). This part of information should not be simply discarded. At the same time, there will be information loss in the process of mapping the two subspaces. Therefore, the proposed IP3S model uses the original features, private representations and public representations of the two modalities as the input of the final classifier (a total of 6 representations for the two modalities). Here, the public representation plays a role in reducing the differences between different modalities as a similar representation of the two modalities. The private representation, as the unique information of each modality, provides the content that each modality pays attention to in the process of identifying emotional states (such as the vocal characteristics in the speech data and the facial expressions in the video data). The original features of the modality are used as a supplement to the above two representations to make up for the information loss problem in the subspace mapping process.

[0012] (2) The P3S model uses two shared encoders to learn the common representations of different modalities respectively, and uses the method of minimizing similarity loss to make the extracted representations of the two as close as possible. In this case, the model can accept features of two modalities with different feature lengths. In the targeted scenario, the extracted speech and video features have the same length. Therefore, the proposed IP3S model uses the same shared encoder (actually two encoders share parameters) to extract the common representations of different modalities. Using the same encoder can ensure that different modalities are mapped to the same subspace through the same mapping function. At the same time, the learning method that originally required the optimization of two shared encoders is changed to the optimization of the same encoder, which reduces the number of parameters to be optimized and speeds up the parameter optimization.

[0013] (3) In the P3S model, the difference in the public representations of different modalities is measured by introducing the cross-modal correlation similarity method, and the public representations obtained between different modalities are made similar by minimizing the similarity loss. The proposed IP3S model introduces an adversarial method to map the public representation to the normal distribution space, and speeds up the search for the public representation by specifying the subspace. At the same time, mapping the public representation to the normal distribution space can ensure that the output of the shared encoder is evenly distributed in the latent space, and provide a stable training process, reducing the learning difficulty of the emotional state recognition classifier.

[0014] Through the above improvements, the learning process of the common representation is accelerated, and more information input is given to the classification process, which can help the model achieve better results. The proposed IP3S model structure is shown in the figure Figure 1 As shown in the figure, the model consists of three core modules: a public subspace learning module called a shared encoder, and two private subspace learning modules called speech private encoder and video private encoder. For speech, the speech public representation and speech private representation are obtained through the shared encoder and the speech private encoder. Similarly, the video data obtains the video public representation and video private representation through the shared encoder and the video private encoder. For each modality, they are designed as encoder-decoder structures (Encoder-Decoder structure). The original modality features obtain the public representation and private representation through the shared encoder and its private encoder respectively, and then the two are spliced ​​and input into the decoder of the corresponding modality to restore the original modality features. In the figure, these encoders and decoders are composed of simple multi-layer perceptrons.

[0015] To solve the heterogeneity problem between multiple modalities, the first thing is to make the two as close as possible. To achieve this goal, in the proposed IP3S model, the voice and video modalities use the same shared encoder to extract similar features of the two. At the same time, a discriminator is introduced to determine whether the common representation is normally distributed, and the common representation is reversely guided to the normal distribution space to reduce the learning difficulty of the model. For the common representation generated by the two modalities, the cross-modal correlation similarity is used to determine the distance between the two, and the extracted common representation is made to be the similar part of the two by minimizing the Euclidean distance.

[0016] The IP3S model uses two private encoders to extract private representations of speech and video data respectively. The private encoders are used to learn the content unique to each modality (information outside the public representation) learned by the data of each modality during the process of emotional state recognition. In order to effectively extract this representation, orthogonal constraints are introduced in the IP3S model to ensure that the public subspace and the private subspace are independent of each other.

[0017] After the two modalities extract subspace representations through the shared encoder and the private encoder, the two representations are concatenated and restored using the decoder. Through this special autoencoder, the two subspace representations can obtain sufficient information in the original modality.

[0018] The IP3S model extracts multimodal public and private representations. The public representation represents the similar information between the two modalities (used to alleviate the heterogeneity problem between different modalities), and the private representation represents the information content unique to the two modalities (used to refine the content that different modalities focus on when identifying emotional states). In addition to these two representations, the original speech and video modal features (containing the richest information in each modality, used to supplement the content lost in the mapping subspace process) will be used together with the public and private representations of each modality to identify emotional states.

[0019] In the second step, the emotional state recognition process based on voice and video fusion of the IP3S model includes the following steps:

[0020] Step 1: Build an emotional state recognition model based on voice and video fusion of IP3S model

[0021] Step 1.1: Build a speech feature extraction model

[0022] A model based on M-DFE and LSTM is used as a speech feature extraction model. The model can effectively extract the deep features of the audio feature sequence (AFS). The audio feature sequence (Audio Feature Sequence, AFS) represents the features extracted from the DAIC-WOZ speech data.

[0023] Step 1.2: Build a video feature extraction model

[0024] The Dilated Residual Temporal Convolutional Network (DepArt-Net) is used as the video feature extraction model. This model can effectively extract the deep features of the Video Feature Sequence (VFS). The Video Feature Sequence (VFS) represents the features extracted from the video data of DAIC-WOZ.

[0025] Step 2: Build the IP3S model and train it

[0026] Step 2.1: Build a shared encoder

[0027] Using a multi-layer perceptron as a shared encoder, the encoder can map AFS and VFS into a common subspace and extract similar features of the two modalities;

[0028] Step 2.2: Build a private audio encoder and a private video encoder

[0029] Multilayer perceptrons are used as private encoders for speech and video. These two encoders can map AFS and VFS to their own private subspaces and extract features specific to each modality.

[0030] Step 2.3: Build the audio decoder and video decoder

[0031] Multilayer perceptrons are used as speech decoders and video decoders. These two decoders can concatenate features in the public subspace and private subspace and restore them to AFS and VFS.

[0032] Step 2.4: Build the Discriminator

[0033] A multi-layer perceptron is used as a discriminator, which can determine whether the features in the common subspace conform to the normal distribution, and map the features in the common subspace to the normal distribution space through adversarial training;

[0034] Step 2.5: Build an emotional state recognition classifier

[0035] A multi-layer perceptron is used as the emotional state recognition classifier, which can fuse AFS, public subspace features, private subspace features and original modal features, and finally output the emotional state recognition result;

[0036] Step 2.6: Train the IP3S model

[0037] The IP3S model is trained using adversarial training. The training process includes the following three steps:

[0038] Step 2.6.1: Subspace mapping

[0039] In this step, the speech feature sequence (AFS) and video feature sequence (VFS) obtained by feature extraction using speech and video data are used as input. The two sequences are passed through a public encoder to obtain a speech common representation sequence (ACRS) and a video common representation sequence (VCRS). The AFS is passed through a speech private encoder to obtain a speech private representation feature sequence (APRS), and the VFS is passed through a video private encoder to obtain a video private representation feature sequence (VPRS). Among them, the audio common representation sequence (ACRS) represents the representation sequence obtained after the speech feature sequence AFS passes through the public encoder; the audio private representation sequence (APRS) represents the representation sequence obtained after the speech feature sequence AFS passes through the speech private encoder; the audio reconstruction feature sequence (ARFS) represents the speech reconstruction feature sequence obtained by inputting ACRS and APRS into the speech decoder; the video common representation sequence (VCRS) represents the representation sequence obtained after the video feature sequence VFS passes through the public encoder; the video private representation sequence (VPRS) Sequence, VPRS), which means the representation sequence obtained after the video feature sequence VFS passes through the video private encoder; Video Reconstruction Feature Sequence (VRFS), which means the video reconstruction feature sequence obtained by inputting VCRS and VPRS into the video decoder;

[0040] The purpose of this step is to enable the public encoder to have the ability to extract the common representation of the two modalities, enable the speech private encoder and the video private encoder to have the ability to extract the unique information of their respective modalities, and make the subspace mapped by the public encoder a normal distribution space. To this end, the following loss is optimized to achieve the above purpose.

[0041] 1) Inter-modality similarity loss

[0042] For AFS and VFS, their public representations ACRS and VCRS are obtained through the same shared encoder, as shown in formulas 1 and 2. In this process, the mapped ACRS and VCRS are required to be as similar as possible, which is crucial to narrowing the differences between the two different modalities. Cross-modal correlation similarity (CMCS) is a commonly used standard that uses the Euclidean distance to measure the differences between different modalities. Therefore, the process of solving the similarity loss between the public representations of speech and video in this chapter is shown in 1-3. Among them, ACRS i and VCRS i The common representation of the speech and video representing the i-th data is obtained by minimizing L AV The difference of the voice-video common representation pair of the same data can be reduced.

[0043] ACRS=Encoder Share (AFS) (1)

[0044] VCRS=Encoder Share (VFS) (2)

[0045]

[0046] 2) Public-private loss within the modality

[0047] For a single modality AFS / VFS, it passes through a shared encoder and a voice / video private encoder to obtain the corresponding voice / video public representation ACRS / VCRS and voice / video private representation APRS / VPRS. In this process, the CRS and PRS of the same modality need to represent different content information. Among them, CRS is used to represent content similar to that related to other modalities, while PRS is used to represent content unique to this modality (different from other modalities, the current modality has unique feature information when identifying emotional states, such as vocal characteristics in voice data and facial expressions in video data). Orthogonal constraints are usually used in mathematics to ensure that two vectors or matrices can remain as independent as possible in geometric relationships, which is expressed in mathematical relationships as the dot product between the two approaches zero.

[0048] In the field of deep learning and machine learning, orthogonal constraints are a common optimization method. It improves the stability and generalization of the model by making the parameters of the neural network conform to orthogonal constraints. Using the mathematical definition of orthogonal constraints, orthogonal constraints are used between public representations and private representations to ensure the difference between private and public subspaces, as shown in Formula 6. Where M ·s and M ·pis a matrix, representing the public and private representation sequences of each modality, and the data in each row is m ·s and m ·p Represents the public and private representations of each input sample. It is used to represent the squared Frobenius norm, and its calculation process is shown in Formula 7.

[0049]

[0050] L diff =L A +L V (6)

[0051]

[0052] 3) Normal distribution fitting loss

[0053] The speech and video modality features are mapped to a common subspace through a shared encoder. To speed up the shared encoder in finding a common representation, the subspace is assigned to a normal distribution space. Mapping the common representation to a normal distribution space ensures that the output of the shared encoder is evenly distributed in the latent space, provides a stable training process, and reduces the learning difficulty of the emotional state recognition classifier.

[0054] The purpose of specifying the subspace is achieved by adding adversarial learning. The discriminator is used to determine whether the public representation is in the normal distribution space, and the shared encoder is used to output the public representation that confuses the discriminator as much as possible. The speech public representation ACRS, the video public representation VCRS and the data generated using the normal distribution are input into the discriminator, and the ACRS and VCRS labels 0 and the normal distribution data label 1 are assigned respectively. At this time, the discriminator parameters are optimized to enhance its identification ability. Then AFS and VFS are sent to the shared encoder to generate ACRS and VCRS and sent to the discriminator to be assigned label 1. At this time, the shared encoder parameters are optimized to enhance its ability to generate normal distribution data. The cross entropy loss function is used as the fitting loss throughout the process, as shown in Formula 8. Where p is the probability that the model predicts that the sample is 1.

[0055] L fit =-(ylog(p)+(1-y)log(1-p)) (8)

[0056] Step 2.6.2: Reconstruction

[0057] For each modality, the voice / video feature sequence (AFS / VFS) is used to pass through the public encoder and the corresponding private encoder to obtain the voice / video public representation sequence (ACRS / VCRS) and the voice / video private representation sequence (APRS / VPRS). The public representation sequence CRS and the private representation sequence PRS are concatenated and sent to the voice / video decoder to obtain the reconstructed AFS and VFS. The purpose of this step is to strengthen the extraction ability of the public encoder and the private encoder. In the mapping process, it is difficult to avoid the transmission loss of the original modal features, which leads to the inability of each subspace representation to fully represent the original modal data. The reconstruction process is added to enhance the ability of each encoder to extract information, and the above purpose is achieved by optimizing the reconstruction loss.

[0058] Reconstruction loss. For a single modality, FS is passed through a shared encoder and a private encoder to obtain a public representation sequence CRS and a private representation sequence PRS. CRS and PRS are concatenated and sent to the decoder for reconstructing FS. In this process, the reconstructed FS is required to be as close to the original FS as possible, so that CRS and PRS can contain enough information of the original FS. The mean square error (MSE) is used as the loss function to measure the difference between the reconstructed FS and the original FS. The reconstruction loss is shown in formulas (9)-(11), where · is the original FS and ·′ is the reconstructed FS.

[0059]

[0060] L Rec =L RecA +L RecV (11)

[0061] Step 2.6.3: Emotional state identification

[0062] In this step, for each modality, the speech / video feature sequence (AFS / VFS) is used to pass through the public encoder and the corresponding private encoder to obtain the speech / video public representation sequence (ACRS / VCRS) and the speech / video private representation sequence (APRS / VPRS). The emotional state prediction label is obtained by concatenating AFS, ACRS, APRS, VFS, VCRS and VPRS and sending them into the emotional state recognition classifier.

[0063] The purpose of this step of learning is to optimize all encoders and emotional state recognition classifiers so that all encoders can extract the emotional state features of each subspace and the classifiers can integrate multiple representations and recognize emotional states. This section achieves the above goals by optimizing the emotional state recognition loss.

[0064] Emotional state recognition loss. The ACRS, APRS, VCRS and VPRS obtained by using AFS and VFS through the shared encoder and the corresponding private encoder are used as subspace representations of different modalities. At the same time, AFS and VFS are also sent to the emotional state recognition classifier as supplementary information to obtain the emotional state prediction label. In this process, the cross entropy loss function is used as the emotional state recognition loss. By minimizing this loss, the emotional state feature extraction capabilities of each subspace of all encoders and the fusion and emotional state prediction capabilities of the emotional state recognition classifier are enhanced. The emotional state recognition loss function is shown in Formula 12, where p is the probability that the model predicts that the sample is 1.

[0065] L classifier =-(ylog(p)+(1-y)log(1-p)) (12)

[0066] Step 3: Use the trained IP3S model to identify emotional states

[0067] The voice and video data uploaded by the user are respectively input into the voice feature extraction model and the video feature extraction model to obtain AFS and VFS. Then AFS, VFS, public subspace features, private subspace features and original modal features are input into the emotion state recognition classifier to obtain the final emotion state recognition result.

[0068] The algorithm 1 involved in the generation process of the IP3S model is as follows:

[0069] Algorithm input: voice dataset, video dataset

[0070] Algorithm output: IP3S model

[0071] Step 1: Divide the dataset into training set Tr, validation set V and test set Te

[0072] Step 2: Extract speech features AFS for each of the three datasets Tr 、AFS V 、AFS Te

[0073] Step 3: Extract video features VFS for the three data sets Tr 、VFS V 、VFS Te

[0074] Step 4: Set IP3S model related parameters (including setting the size of training rounds, learning rate, etc.)

[0075] Step 5: In each training round, execute steps 6 to 24 for all the data in the training set Tr. When the number of training rounds exceeds the set value, stop training and go to step 25.

[0076] Step 6: Install AFS Tr Send to the shared encoder to get ACRS

[0077] Step 7: VFS Tr Send to the shared encoder to get VCRS

[0078] Step 8: Install AFS Tr Send it to the private speech encoder to get APRS

[0079] Step 9: VFS Tr Send it to the private speech encoder to get VPRS

[0080] Step 10: Use formulas 1 to 7 to solve and minimize the inter-modal similarity loss and the intra-modal public-private loss respectively.

[0081] Step 11: Minimize the sum of the two losses and optimize multiple encoder parameters

[0082] Step 12: Regenerate data using the optimized encoder

[0083] Step 13: Generate data z through normal distribution

[0084] Step 14: Assign ACRS and VCRS the label 0, z the label 1, and input them into the discriminator to get the result

[0085] Step 15: Solve the loss and optimize the discriminator parameters by formula 12

[0086] Step 16: Lock the discriminator parameters, assign ACRS and VCRS to label 1, and input them into the discriminator to get the result

[0087] Step 17: Solve the loss and optimize the shared encoder parameters using Equation 12

[0088] Step 18: Regenerate data using the optimized encoder

[0089] Step 19: Input ACRS and APRS into the voice decoder to get the restored AFS Tr , namely AFS Tr '

[0090] Step 20: Input VCRS and VPRS into the video decoder to get the restored VFS Tr , namely VFS Tr '

[0091] Step 21: Solve the reconstruction loss according to formula 9-11 and optimize the encoder and decoder

[0092] Step 22: Regenerate data using the optimized encoder

[0093] Step 23: Use the emotional state recognition classifier to obtain the emotional state recognition result y

[0094] Step 24: Solve the loss according to formula 12 and optimize the encoder and emotional state recognition classifier

[0095] Step 25: Get the IP3S model and use AFS on the validation set V V 、VFS V Verification of IP3S model

[0096] Step 26: If the verification result is as expected, use the AFS of the test set Te Te 、VFS Te Evaluate the IP3S model;

[0097] If the verification result does not meet expectations, go to step 4 to retrain the model.

[0098] Step 27: Export the IP3S model.

[0099] Beneficial effects of the present invention:

[0100] (1) Construct an emotion state recognition method based on speech and video fusion of IP3S model, explore the problem of speech and video feature fusion in multimodal emotion state recognition, and generate a multimodal emotion state recognition model.

[0101] (2) The emotional state recognition method based on speech and video fusion of the IP3S model is introduced. The problem definition, problem solving process and algorithm in this method are introduced, and the effectiveness of this method is proved through experimental comparative analysis.

[0102] (3) First, a speech and video feature extraction model was established to extract speech and video features respectively. Then, based on the improved shared private subspace separation model IP3S model, the two features were fused, and an emotion state recognition classifier was built to perform emotion state recognition. This can effectively solve the problem of excessive differences between modalities in the process of speech and video fusion, thereby improving the accuracy of emotion state recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 Schematic diagram of the overall design of the present invention.

[0104] Figure 2 P3S structure diagram.

[0105] Figure 3 IP3S model structure diagram.

[0106] Figure 4(a) is the confusion matrix of IP3S_2C.

[0107] Figure 4(b) is the ROC diagram of IP3S_2C.

[0108] Figure 5(a) is the confusion matrix of IP3S_2C_2P_2O.

[0109] Figure 5(b) is the ROC diagram of IP3S_2C_2P_2O.

[0110] Figure 6(a) is the confusion matrix of the IP3S model.

[0111] Figure 6(b) is the ROC graph of the IP3S model.

[0112] Specific implementation content

[0113] Embodiment 1: Emotional state recognition method based on voice and video fusion of IP3S model

[0114] The first step is to create a data set

[0115] DAIC-WOZ dataset. The depression recognition task released by AVEC2017 (The Audio / Visual Emotion Challenge and Workshop) uses the DAIC-WOZ dataset (Distress Analysis Interview Corpus–Wizard of Oz). The DAIC-WOZ dataset contains 189 clinical interview data, of which the interviews are divided into a training set (containing 107 data), a validation set (containing 35 data), and a test set (containing 47 data). The length of each speech is between 7 and 33 minutes, and provides a depression scale score of PHQ-8 (obtained by answering eight questions). When the score is greater than or equal to 10, the person is considered to have depression, and a score below 10 is normal.

[0116] The DAIC-WOZ dataset consists of three different data types, namely speech (A), text (T), and video (V). Considering the spurious correlation between the scale score labels and the text content of the person (the scale scores are derived from the text data), only the speech and video data are used for depression recognition research.

[0117] The second step is to verify the effectiveness and ablation of the improved IP3S model.

[0118] Step 1: Verify the effectiveness of the improved parts of the IP3S model and verify the ablation of each part of the IP3S model

[0119] (1) Verify the effectiveness of the number of shared encoders for multimodal fusion

[0120] In the P3S model, two shared encoders are used for the two modalities to extract the common subspace representations corresponding to the two modalities, and the similarity loss between the two common representations is solved to achieve the purpose of extracting similar information. Since the corresponding feature dimensions extracted from speech and video are the same, the two shared encoder parameters can be shared (equivalent to using the same shared encoder) to achieve consistency in the mapping function. To verify the effectiveness of this approach, the following three groups are set for analysis:

[0121] 1) Use the features of the two modalities to concatenate and feed them into the MLP model for experiments (build a simple model for comparison);

[0122] 2) Using the common representation of the two modalities as the classifier input, not specifying the shared subspace as a normal distribution space, and using the IP3S model with two shared encoders (i.e., the P3S model without any modification);

[0123] 3) The IP3S model uses the common representation of the two modalities as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder.

[0124] The present invention uses the training set, validation set and test set provided by the data official for training and result comparison. Finally, four evaluation criteria, namely accuracy, precision, recall and F1-score, are used to compare the results.

[0125] (2) Verify the effectiveness of the classifier receiving more inputs for multimodal fusion.

[0126] In the P3S model, the input for the target task is only the public representation of the two modalities, and the private representation of the two modalities and the features of the two modalities themselves are also sent to the depression recognition classifier. This provides more information for the target task. The public representation provides similar information between different modalities, the private representation provides unique information between different modalities, and the features of the modalities themselves provide information lost in the subspace mapping process. To verify the effectiveness of this approach, the following four groups are set for analysis:

[0127] 1) Using the common representation of the two modalities (two pieces of information in total) as the classifier input, not specifying the shared subspace as a normal distribution space, and using an IP3S model with a shared encoder;

[0128] 2) The IP3S model uses the public representation and private representation of the two modalities (a total of four pieces of information) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder;

[0129] 3) The IP3S model uses the common representation of the two modalities and the features of the modalities themselves (a total of four pieces of information) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder;

[0130] 4) The IP3S model uses the public representation, private representation and features of the two modalities (a total of six pieces of information) as classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder.

[0131] (3) Verify the effectiveness of introducing normal distribution for multimodal fusion.

[0132] In the P3S model, there is no guidance for the common subspace, and the model itself is left to learn and explore. The proposed IP3S model maps the common representation to the normal distribution space by introducing adversarial forces, and accelerates the model's search for common representations through this subspace distribution. At the same time, mapping the common representation to the normal distribution space can ensure that the output of the shared encoder is evenly distributed in the latent space, and provide a stable training process, reducing the learning difficulty of the depression recognition classifier. To verify the effectiveness of this approach, the following two groups are set up for analysis:

[0133] 1) The public representation, private representation and features of the two modalities (a total of six pieces of information) are used as classifier inputs, the shared subspace is not specified as a normal distribution space, and the IP3S model with a shared encoder is used;

[0134] 2) The public representation, private representation and features of the two modalities (a total of six pieces of information) are used as classifier inputs, the shared subspace is specified as a normal distribution space and an IP3S model with a shared encoder is used.

[0135] Step 2: Verify the ablation of each part of the IP3S model

[0136] (4) Verify the ablation of each part of the IP3S model

[0137] In the proposed IP3S model, the modal features are mapped to the public subspace and private subspace through shared coding and private encoders, and the information extraction capabilities of shared coding and private encoders are enhanced through the reconstruction process of the decoder. In order to verify the impact of each module on the overall structure of the IP3S model, the following four groups are set for analysis:

[0138] (1) Experiments were conducted using the IP3S model without the reconstruction part;

[0139] (2) Experiments were conducted using the IP3S model without the private encoder;

[0140] (3) Experiments were conducted using the IP3S model without the shared encoder;

[0141] (4) Experiments were conducted using the proposed IP3S model.

[0142] The training set, validation set, and test set provided by the official data source were used for training and result comparison, and the results were compared using four evaluation criteria.

[0143] Example 2: Verifying the emotional state recognition method based on voice and video fusion based on IP3S model

[0144] The following four sets of experiments were used for verification.

[0145] Experiment 1:

[0146] This experiment explores the impact of the number of shared encoders on the multimodal fusion method. By sharing the parameters of the shared encoder, different modalities can be mapped to the common subspace through the same function. To verify its effectiveness, this section sets up three groups of experiments for comparison. To facilitate the collation of results, the IP3S model that uses the common representation of the two modalities as the classifier input, does not specify the shared subspace as a normal distribution space, and uses two shared encoders is referred to as P3S, and the IP3S model that uses the common representation of the two modalities as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S_2C. The experimental results of this experiment are shown in Table 1, where the indicators in () correspond to the normal category.

[0147] Table 1 Experimental results

[0148]

[0149] As can be seen from Table 1, simply concatenating speech features and video features and feeding them into the MLP model does not allow us to learn the key information. This is because the heterogeneity between speech and video modalities makes the fusion process of the two very challenging [60,65,73]. At the same time, IP3S_2C outperforms P3S in the key indicator F1-Score, which also shows that the modification of the number of shared encoders has a certain effect on multimodal fusion. Figure 4(a) and Figure 4(b) are the confusion matrix and ROC graph of IP3S_2C.

[0150] Experiment 2:

[0151] This experiment explores the impact of the amount of information received by the classifier on the multimodal fusion method. By giving the depression recognition classifier more information to improve the effect of multimodal fusion, in order to verify its effectiveness, this section sets up four groups of experiments for comparison. To facilitate the result arrangement, the IP3S model that uses the public representation of the two modalities (two information in total) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S_2C, the IP3S model that uses the public representation and private representation of the two modalities (four information in total) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S_2C_2P, the IP3S model that uses the public representation of the two modalities and the modality's own features (four information in total) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S_2C_2O, and the IP3S model that uses the public representation, private representation, and modality's own features (six information in total) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S_2C_2P_2O. The experimental results of this experiment are shown in Table 1, where the indicators in () are the corresponding indicators of the normal category.

[0152] Table 2 Experimental results

[0153]

[0154] As can be seen from Table 2, by giving more information input, the multimodal fusion model has achieved better indicators in depression recognition. When additional private representation is given, the F1 and Recall of the depression category are improved. When the original modal features are given, the indicators do not get better results. When additional private representation and original modal features are given, the indicators are improved compared with the IP3S_2C (no additional information) model. This shows that providing more information as the input of the classifier has a significant positive effect on multimodal fusion depression recognition. Figure 5 (a) and Figure 5 (b) are the confusion matrix and ROC diagram of IP3S_2C_2P_2O.

[0155] Experiment 3:

[0156] This experiment explores the impact of introducing normal distribution on multimodal fusion. By introducing adversarial means, the common subspace is specified as a normal distribution space, which enables the model to find an effective common representation faster. To verify the effectiveness of this approach, this section sets up two sets of experiments for comparison. To facilitate the collation of results, the IP3S model that uses the common representation, private representation and modality features of the two modalities (a total of six pieces of information) as classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S_2C_2P_2O, and the IP3S model that uses the common representation, private representation and modality features of the two modalities (a total of six pieces of information) as classifier input, specifies the shared subspace as a normal distribution space, and uses a shared encoder is referred to as IP3S. The experimental results of this experiment are shown in Table 3, where the indicators corresponding to the normal category are in ().

[0157] Table 3 Experimental results

[0158]

[0159] As shown in Table 3, the IP3S model has improved both the F1-score and Recall indicators of the depression category, which shows that specifying the common subspace as a normal distribution space can help the IP3S model achieve better results in multimodal depression recognition. Figure 6(a) and Figure 6(b) are the confusion matrix and ROC diagram of the IP3S model.

[0160] Experiment 4: This experiment uses ablation experiments to verify the impact of each module on the IP3S model. This experiment conducts experiments on the reconstruction module, the mapping public subspace module, and the mapping private subspace module, and sets up a total of four groups of experiments for comparative analysis. To facilitate the result arrangement, the IP3S model without the reconstruction part is referred to as IP3S_NoR, the IP3S model without the private encoder is referred to as IP3S_NoP, and the IP3S model without the shared encoder is referred to as IP3S_NoC. The experimental results of this experiment are shown in Table 4, where the indicators in () correspond to the normal category.

[0161] Table 4 Experimental results

[0162]

[0163] As shown in Table 4, removing each module will have a certain negative impact on the model. Among them, removing the reconstruction process has the smallest decline in the indicators of the depression category, and the reconstruction process has little impact on the basic framework of the model. Removing the private representation of the depression category has a large decline in the indicators, and the mapping of the private subspace has a great impact on the overall model. Removing the public representation of the depression category has a significant decline in the indicators. The public representation is the key content of the IP3S model to narrow the differences between different modalities. The mapping of the public subspace directly affects the overall results of the model.

[0164] Experiment 4:

[0165] This experiment uses ablation experiments to verify the impact of each module on the IP3S model. This experiment conducts experiments on the reconstruction module, the mapping public subspace module, and the mapping private subspace module, and sets up four groups of experiments for comparative analysis. To facilitate the result arrangement, the IP3S model without the reconstruction part is referred to as IP3S_NoR, the IP3S model without the private encoder is referred to as IP3S_NoP, and the IP3S model without the shared encoder is referred to as IP3S_NoC. The experimental results of this experiment are shown in Table 5, where the indicators in () correspond to the normal category.

[0166] Table 5 Experimental results

[0167]

[0168] As shown in Table 5, removing each module will have a certain negative impact on the model. Among them, removing the reconstruction process has the smallest decline in the indicators of the depression category, and the reconstruction process has little impact on the basic framework of the model. Removing the private representation of the depression category has a large decline in the indicators, and the mapping of the private subspace has a great impact on the overall model. Removing the public representation of the depression category has a significant decline in the indicators. The public representation is the key content of the IP3S model to narrow the differences between different modalities. The mapping of the public subspace directly affects the overall results of the model.

Claims

1. A method for emotional state recognition based on voice and video fusion of IP3S model, characterized in that: include: Step 1: Establish an improved shared private subspace separation model IP3S (Improved Private–Shared Subspaces); Step 2: Emotional state recognition based on speech and video fusion of IP3S model.

2. The method for emotional state recognition based on voice and video fusion of IP3S model according to claim 1, characterized in that: In the first step, the improved shared private subspace separation model IP3S is as follows: The model consists of three core modules: a public subspace learning module called a shared encoder, and two private subspace learning modules called speech private encoder and video private encoder. For speech, the shared encoder and speech private encoder are used to obtain speech public representation and speech private representation. Similarly, the shared encoder and video private encoder are used to obtain video public representation and video private representation. For each modality, the encoder-decoder structure is designed. The original modality features are respectively obtained through the shared encoder and its private encoder to obtain public representation and private representation, and then the two are spliced ​​and input into the decoder of the corresponding modality to restore the original modality features. These encoders and decoders are composed of simple multi-layer perceptrons. To solve the heterogeneity problem between multiple modalities, the first thing is to make the two modalities as close as possible. To achieve this goal, in the proposed IP3S model, the voice and video modalities use the same shared encoder to extract similar features of the two. At the same time, a discriminator is introduced to determine whether the common representation is normally distributed, and the common representation is reversely guided to the normal distribution space to reduce the learning difficulty of the model. For the common representation generated by the two modalities, the cross-modal correlation similarity is used to determine the distance between the two. By minimizing the Euclidean distance, the extracted common representation is the similar part of the two. The IP3S model uses two private encoders to extract private representations of speech and video data respectively. The private encoders are used to learn the content unique to each modality (information outside the public representation) learned by the data of each modality during the process of emotional state recognition. In order to effectively extract the representation, orthogonal constraints are introduced in the IP3S model to ensure that the public subspace and the private subspace are independent of each other. After the two modalities are extracted from the subspace representation by the shared encoder and the private encoder, the two representations are concatenated and restored using the decoder. Through this special autoencoder, the two subspace representations can obtain enough information in the original modality; The IP3S model extracts multimodal public and private representations. The public representation represents the similar information between the two modalities (used to alleviate the heterogeneity problem between different modalities), and the private representation represents the information content unique to the two modalities (used to refine the content that different modalities focus on when identifying emotional states). In addition to these two representations, the original speech and video modal features (containing the richest information in each modality, used to supplement the content lost in the mapping subspace process) will be used together with the public and private representations of each modality to identify emotional states.

3. The method for emotional state recognition based on voice and video fusion of IP3S model according to claim 1, characterized in that: In the second step, the emotional state recognition process based on voice and video fusion of IP3S model Step 1: Build an emotional state recognition model based on voice and video fusion of IP3S model Step 1.1: Build a speech feature extraction model A model based on M-DFE and LSTM is used as a speech feature extraction model. The model can effectively extract the deep features of the audio feature sequence (AFS). The audio feature sequence (Audio Feature Sequence, AFS) represents the features extracted from the DAIC-WOZ speech data. Step 1.2: Build a video feature extraction model The Dilated Residual Temporal Convolutional Network (DepArt-Net) is used as the video feature extraction model. This model can effectively extract the deep features of the Video Feature Sequence (VFS). The Video Feature Sequence (VFS) represents the features extracted from the video data of DAIC-WOZ. Step 2: Build and train the IP3S model Step 2.1: Build a shared encoder Using a multi-layer perceptron as a shared encoder, the encoder can map AFS and VFS into a common subspace and extract similar features of the two modalities; Step 2.2: Build a private audio encoder and a private video encoder Multilayer perceptrons are used as private encoders for speech and video. These two encoders can map AFS and VFS to their own private subspaces and extract features specific to each modality. Step 2.3: Build the audio decoder and video decoder Multilayer perceptrons are used as speech decoders and video decoders. These two decoders can concatenate features in the public subspace and private subspace and restore them to AFS and VFS. Step 2.4: Build the Discriminator A multi-layer perceptron is used as a discriminator, which can determine whether the features in the common subspace conform to the normal distribution, and map the features in the common subspace to the normal distribution space through adversarial training; Step 2.5: Build an emotional state recognition classifier A multi-layer perceptron is used as the emotional state recognition classifier, which can fuse AFS, public subspace features, private subspace features and original modal features, and finally output the emotional state recognition result; Step 2.6: Train the IP3S model The IP3S model is trained using adversarial training. The training process includes the following three steps: Step 2.6.1: Subspace mapping In this step, the speech feature sequence (AFS) and video feature sequence (VFS) obtained by feature extraction using speech and video data are used as input. The two sequences are passed through a public encoder to obtain a speech common representation sequence (ACRS) and a video common representation sequence (VCRS). The AFS is passed through a speech private encoder to obtain a speech private representation feature sequence (APRS), and the VFS is passed through a video private encoder to obtain a video private representation feature sequence (VPRS). Among them, the audio common representation sequence (Audio Common Representations Sequence, ACRS) represents the representation sequence obtained after the speech feature sequence AFS passes through the public encoder; the audio private representation sequence (Audio Private Representations Sequence, APRS) represents the representation sequence obtained after the speech feature sequence AFS passes through the speech private encoder; the audio reconstruction feature sequence (Audio Reconstruction Feature Sequence, ARFS) represents the speech reconstruction feature sequence obtained by inputting ACRS and APRS into the speech decoder; the video common representation sequence (Video Common Representations Sequence, VCRS) represents the representation sequence obtained after the video feature sequence VFS passes through the public encoder; the video private representation sequence (Video Private Representations Sequence, VPRS) Sequence, VPRS), which means the representation sequence obtained after the video feature sequence VFS passes through the video private encoder; Video Reconstruction Feature Sequence (VRFS), which means the video reconstruction feature sequence obtained by inputting VCRS and VPRS into the video decoder; The purpose of this step is to enable the public encoder to have the ability to extract the common representation of the two modalities, to enable the speech private encoder and the video private encoder to have the ability to extract the unique information of their respective modalities, and to make the subspace mapped by the public encoder a normal distribution space. To this end, the following losses are optimized to achieve the above purpose; 1) Inter-modality similarity loss For AFS and VFS, their common representations ACRS and VCRS are obtained through the same shared encoder, as shown in formulas 1 and 2. In this process, the mapped ACRS and VCRS are required to be as similar as possible, which is crucial to narrowing the differences between the two different modes. Cross-modal correlation similarity (CMCS) is a commonly used standard that uses the Euclidean distance to measure the differences between different modes. Therefore, the process of solving the similarity loss between the common representations of speech and video in this chapter is shown in 1-3. Among them, ACRS i and VCRS i The common representation of the speech and video representing the i-th data is obtained by minimizing L AV It can reduce the differences of the voice-video common representation pairs of the same data; ACRS=Encoder Share (AFS) (1) VCRS=Encoder Share (VFS) (2) 2) Public-private loss within the modality For a single modality AFS / VFS, it passes through a shared encoder and a voice / video private encoder to obtain the corresponding voice / video public representation ACRS / VCRS and voice / video private representation APRS / VPRS respectively; in this process, the CRS and PRS of the same modality need to represent different content information; among them, CRS is used to represent content similar to that related to other modalities, while PRS is used to represent the unique content within this modality (different from other modalities, the unique feature information of the current modality when identifying emotional states, such as the vocal characteristics in voice data and facial expressions in video data); orthogonal constraints are usually used in mathematics to ensure that two vectors or matrices can remain as independent as possible in terms of geometric relationships, which is expressed in mathematical relationships as the dot product between the two approaches zero; In the field of deep learning and machine learning, orthogonal constraints are a common optimization method. It improves the stability and generalization of the model by making the parameters of the neural network conform to orthogonal constraints. Using the mathematical definition of orthogonal constraints, orthogonal constraints are used between public representations and private representations to ensure the difference between private and public subspaces, as shown in Formula 6; where M ·s and M ·p is a matrix, representing the public and private representation sequences of each modality, and the data in each row is m ·s and m ·p Represent the public and private representations of each input sample; It is used to represent the squared Frobenius norm, and its calculation process is shown in Formula 7; L diff =L A +L V (6) 3) Normal distribution fitting loss The speech and video modality features are mapped to a common subspace through a shared encoder. To speed up the shared encoder in finding a common representation, the subspace is designated as a normal distribution space. Mapping the common representation to a normal distribution space ensures that the output of the shared encoder is evenly distributed in the latent space and provides a stable training process, thereby reducing the learning difficulty of the emotional state recognition classifier. The purpose of specifying the subspace is achieved by adding adversarial learning. The discriminator is used to determine whether the public representation is in the normal distribution space, and the shared encoder is used to output the public representation that confuses the discriminator as much as possible; the speech public representation ACRS, the video public representation VCRS and the data generated using the normal distribution are input into the discriminator, and the ACRS and VCRS labels 0 and the normal distribution data label 1 are assigned respectively. At this time, the discriminator parameters are optimized to enhance its identification ability; then AFS and VFS are sent to the shared encoder to generate ACRS and VCRS and sent to the discriminator to be assigned label 1. At this time, the shared encoder parameters are optimized to enhance its ability to generate normal distribution data; the cross entropy loss function is used as the fitting loss in the whole process, as shown in Formula 8; where p is the probability that the model predicts that the sample is 1; L fit =-(ylog(p)+(1-y)log(1-p)) (8) Step 2.6.2: Reconstruction For each modality, the voice / video feature sequence (AFS / VFS) is used to pass through the public encoder and the corresponding private encoder to obtain the voice / video public representation sequence (ACRS / VCRS) and the voice / video private representation sequence (APRS / VPRS). The public representation sequence CRS and the private representation sequence PRS are concatenated and sent to the voice / video decoder to obtain the reconstructed AFS and VFS; the purpose of this step of learning is to strengthen the extraction ability of the public encoder and the private encoder. In the mapping process, it is difficult to avoid the transmission loss of the original modal features, which leads to the inability of each subspace representation to fully represent the original modal data; the reconstruction process is added to enhance the ability of each encoder to extract information, and the above purpose is achieved by optimizing the reconstruction loss; Reconstruction loss; For a single modality, FS is passed through a shared encoder and a private encoder to obtain a public representation sequence CRS and a private representation sequence PRS. CRS and PRS are concatenated and sent to the decoder for reconstructing FS. In this process, the reconstructed FS is required to be as close to the original FS as possible, so that CRS and PRS can contain enough information of the original FS. The mean square error (MSE) is used as the loss function to measure the difference between the reconstructed FS and the original FS. The reconstruction loss is shown in formulas (9)-(11), where · is the original FS and ·′ is the reconstructed FS. L Rec =L RecA +L RecV (11) Step 2.6.3: Emotional state identification In this step, for each modality, the voice / video feature sequence (AFS / VFS) is used to pass through the public encoder and the corresponding private encoder to obtain the voice / video public representation sequence (ACRS / VCRS) and the voice / video private representation sequence (APRS / VPRS). The emotional state prediction label is obtained by splicing AFS, ACRS, APRS, VFS, VCRS and VPRS and sending them to the emotional state recognition classifier; The purpose of this step is to optimize all encoders and emotional state recognition classifiers, so that all encoders can extract the characteristics of emotional states in each subspace, and the classifiers can integrate multiple representations and recognize emotional states. This section optimizes the emotional state recognition loss to achieve the above purpose. Emotional state recognition loss; ACRS, APRS, VCRS and VPRS obtained by AFS and VFS through the shared encoder and the corresponding private encoder are used as subspace representations of different modalities, and AFS and VFS are also sent to the emotional state recognition classifier as supplementary information to obtain the emotional state prediction label; In this process, the cross entropy loss function is used as the emotional state recognition loss, and this loss is minimized to enhance the emotional state feature extraction capabilities of each subspace of all encoders and the fusion and emotional state prediction capabilities of the emotional state recognition classifier; The emotional state recognition loss function is shown in Formula 12, where p is the probability that the model predicts that the sample is 1; L classifier =-(ylog(p)+(1-y)log(1-p)) (12) Step 3: Use the trained IP3S model to identify emotional states The voice and video data uploaded by the user are respectively input into the voice feature extraction model and the video feature extraction model to obtain AFS and VFS. Then AFS, VFS, public subspace features, private subspace features and original modal features are input into the emotion state recognition classifier to obtain the final emotion state recognition result.

4. The emotional state recognition method based on voice and video fusion of IP3S model according to claim 2 or 3, characterized in that: The generation process of the IP3S model involves Algorithm 1 as follows: Algorithm input: voice dataset, video dataset Algorithm output: IP3S model Step 1: Divide the dataset into training set Tr, validation set V and test set Te Step 2: Extract speech features AFS for each of the three datasets Tr 、AFS V 、AFS Te Step 3: Extract video features VFS for the three data sets Tr 、VFS V 、VFS Te Step 4: Set IP3S model related parameters (including setting the size of training rounds, learning rate, etc.) Step 5: In each training round, execute steps 6 to 24 for all the data in the training set Tr. When the number of training rounds exceeds the set value, stop training and go to step 25. Step 6: Install AFS Tr Send to the shared encoder to get ACRS Step 7: VFS Tr Send to the shared encoder to get VCRS Step 8: Install AFS Tr Send it to the private speech encoder to get APRS Step 9: VFS Tr Send it to the private speech encoder to get VPRS Step 10: Use formulas 1 to 7 to solve and minimize the inter-modal similarity loss and the intra-modal public-private loss respectively. Step 11: Minimize the sum of the two losses and optimize multiple encoder parameters Step 12: Regenerate data using the optimized encoder Step 13: Generate data z through normal distribution Step 14: Assign ACRS and VCRS the label 0, z the label 1, and input them into the discriminator to get the result Step 15: Solve the loss and optimize the discriminator parameters by formula 12 Step 16: Lock the discriminator parameters, assign ACRS and VCRS to label 1, and input them into the discriminator to get the result Step 17: Solve the loss and optimize the shared encoder parameters using Equation 12 Step 18: Regenerate data using the optimized encoder Step 19: Input ACRS and APRS into the voice decoder to get the restored AFS Tr , namely AFS Tr ' Step 20: Input VCRS and VPRS into the video decoder to get the restored VFS Tr , namely VFS Tr ' Step 21: Solve the reconstruction loss according to formula 9-11 and optimize the encoder and decoder Step 22: Regenerate data using the optimized encoder Step 23: Use the emotional state recognition classifier to obtain the emotional state recognition result y Step 24: Solve the loss according to formula 12 and optimize the encoder and emotional state recognition classifier Step 25: Get the IP3S model and use AFS on the validation set V V 、VFS V Verification of IP3S model Step 26: If the verification result is as expected, use the AFS of the test set Te Te 、VFS Te Evaluate the IP3S model; If the verification result does not meet expectations, go to step 4 to retrain the model. Step 27: Export the IP3S model.

5. The method for emotional state recognition based on voice and video fusion of IP3S model according to claim 4, characterized in that: The emotional state recognition method based on voice and video fusion of IP3S model comprises: The first step is to create a data set DAIC-WOZ dataset; The depression recognition task released by AVEC2017 (The Audio / Visual Emotion Challenge and Workshop) uses the DAIC-WOZ dataset (Distress Analysis Interview Corpus–Wizard of Oz); The DAIC-WOZ dataset contains 189 clinical interview data, of which the interviews are divided into a training set (containing 107 data), a validation set (containing 35 data) and a test set (containing 47 data); the length of each speech is between 7 and 33 minutes, and provides the PHQ-8 depression scale score (obtained by answering eight questions). When the score is greater than or equal to 10, the person is considered to have depression, and if it is less than 10 points, it is normal; The DAIC-WOZ dataset consists of three different data types, namely speech (A), text (T), and video (V); considering the spurious correlation between the scale score labels and the text content stated by the person (the scale scores are derived from the text data), only the speech and video data are used for depression recognition research.

6. The method for emotional state recognition based on voice and video fusion of IP3S model according to claim 4, characterized in that: Verify the effectiveness and ablation of the improved parts of the IP3S model, including: Step 1: Verify the effectiveness of the improved parts of the IP3S model and verify the ablation of each part of the IP3S model (1) Verify the effectiveness of the number of shared encoders for multimodal fusion In the P3S model, two shared encoders are used for the two modalities to extract the common subspace representations corresponding to the two modalities, and the similarity loss between the two common representations is solved to achieve the purpose of extracting similar information; since the corresponding feature dimensions extracted by speech and video are the same, the two shared encoder parameters can be shared (equivalent to using the same shared encoder) to achieve consistency in the mapping function; to verify the effectiveness of this approach, the following three groups are set for analysis: 1) Use the features of the two modalities to concatenate and feed them into the MLP model for experiments (build a simple model for comparison); 2) Using the common representation of the two modalities as the classifier input, not specifying the shared subspace as a normal distribution space, and using the IP3S model with two shared encoders (i.e., the P3S model without any modification); 3) The IP3S model uses the common representation of the two modalities as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder; The present invention uses the training set, validation set and test set provided by the data official for training and result comparison, and finally uses four evaluation criteria of accuracy, precision, recall and F1-score to compare the results; (2) Verify the effectiveness of the classifier receiving more inputs for multimodal fusion; In the P3S model, the input for the target task is only the public representation of the two modalities, and the private representation of the two modalities and the features of the two modalities themselves are also sent to the depression recognition classifier; this provides more information for the target task, the public representation provides similar information between different modalities, the private representation provides unique information between different modalities, and the features of the modalities themselves provide information lost in the subspace mapping process; to verify the effectiveness of this approach, the following four groups are set for analysis: 1) Using the common representation of the two modalities (two pieces of information in total) as the classifier input, not specifying the shared subspace as a normal distribution space, and using an IP3S model with a shared encoder; 2) The IP3S model uses the public and private representations of the two modalities (a total of four pieces of information) as classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder; 3) The IP3S model uses the common representation of the two modalities and the features of the modalities themselves (a total of four pieces of information) as the classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder; 4) The IP3S model uses the public representation, private representation and features of the two modalities (a total of six pieces of information) as classifier input, does not specify the shared subspace as a normal distribution space, and uses a shared encoder; (3) Verify the effectiveness of introducing normal distribution for multimodal fusion; In the P3S model, there is no guidance on the common subspace, and the model itself is left to learn and explore. The proposed IP3S model maps the common representation to the normal distribution space by introducing adversarial forces. This subspace distribution method speeds up the model's search for common representations. At the same time, mapping the common representation to the normal distribution space ensures that the output of the shared encoder is evenly distributed in the latent space, provides a stable training process, and reduces the learning difficulty of the depression recognition classifier. To verify the effectiveness of this approach, the following two groups are set for analysis: 1) The public representation, private representation and features of the two modalities (a total of six pieces of information) are used as classifier inputs, the shared subspace is not specified as a normal distribution space, and the IP3S model with a shared encoder is used; 2) The public representation, private representation and features of the two modalities (a total of six pieces of information) are used as classifier inputs, the shared subspace is specified as a normal distribution space and an IP3S model with a shared encoder is used.

7. The emotional state recognition method based on voice and video fusion of IP3S model according to claim 5 or 6, characterized in that: Verify the ablation of various parts of the IP3S model, including: (4) Verify the ablation of each part of the IP3S model In the proposed IP3S model, the modal features are mapped into public subspace and private subspace through shared coding and private encoder, and the information extraction ability of shared coding and private encoder is enhanced through the reconstruction process of the decoder; in order to verify the influence of each module on the overall structure of the IP3S model, the following four groups are set for analysis: (1) Experiments were conducted using the IP3S model without the reconstruction part; (2) Experiments were conducted using the IP3S model without the private encoder; (3) Experiments were conducted using the IP3S model without the shared encoder; (4) Conduct experiments using the proposed IP3S model; The training set, validation set, and test set provided by the official data source were used for training and result comparison, and the results were compared using four evaluation criteria.

Images