Augmented reality-oriented multi-modal emotion recognition method, device, product and system

Abstract

The invention provides an augmented reality-oriented multi-modal emotion recognition method, device, product and system S1. The method comprises the following steps: collecting multi-modal data; s2, respectively obtaining feature vectors, used for emotion recognition, of each mode; and S3, performing modal availability detection, proxy modal feature confrontation generation and emotional state prediction on each modal by using a deep neural network EmoSynNet. By means of the technical scheme, the accuracy and adaptability of emotion recognition in the augmented reality environment can be improved.

Description

Technical Field

[0001] The present invention relates to emotion recognition technology in an augmented reality environment, and in particular to a multimodal emotion recognition method, device, product and system for augmented reality. Background Art

[0002] In today's digital age, augmented reality (AR) as an innovative interactive technology has brought a new interactive experience to human society. By combining advanced graphics technology, virtual reality and the real world, AR has created an interactive platform that transcends the limitations of physical space. On this platform, emotion recognition technology, as a key component of interaction, is crucial to improving user experience and interaction effects. However, existing technologies still have limitations in emotion recognition and expression.

[0003] In the real world, recognizing and understanding human emotions is a complex task involving multiple modalities such as speech, facial expressions, and body language. This challenge is magnified in an augmented reality environment, where interactions require not only a high degree of realism, but also immediate and accurate emotional feedback. Existing emotion recognition systems often rely on a single modality, such as speech or images alone, which is insufficient and inflexible in the ever-changing augmented reality environment.

[0004] In addition, existing technologies perform poorly when dealing with incomplete or missing modal information. For example, when emotions cannot be accurately expressed through facial expressions or intonation, existing emotion recognition systems often fail to provide accurate emotion analysis, thus affecting user experience and interaction effects. Summary of the invention

[0005] Embodiments of the present invention provide a multimodal emotion recognition method, device, product and system for augmented reality, so as to effectively analyze the multimodal emotion state of a user in an augmented reality environment and improve the accuracy and adaptability of emotion recognition in the augmented reality environment.

[0006] In order to achieve the above object, on the one hand, a multimodal emotion recognition method for augmented reality is provided, comprising:

[0007] S1, collecting multimodal data input by the user in the augmented reality environment, the multimodal data including: voice, image, text and vital sign data, the vital sign data including: electrocardiogram data and body temperature data, obtaining a discrete time audio signal of the voice, a two-dimensional pixel matrix of the image, a character sequence of the text, a discrete time electrocardiogram signal of the electrocardiogram data and a discrete time body temperature signal of the body temperature data;

[0008] S2, obtain the feature vectors of each modality for emotion recognition, where:

[0009] Extracting a feature set of the discrete-time audio signal, and processing the feature set of the discrete-time audio signal using a deep belief network to obtain a speech feature vector for emotion recognition;

[0010] Extracting a feature set of the two-dimensional pixel matrix, and processing the feature set of the two-dimensional pixel matrix using a residual network to obtain an image feature vector for emotion recognition; and,

[0011] According to the selected preprocessing, the character sequence is converted into a term vector, a topic distribution is constructed for the term vector using latent Dirichlet allocation, and a context feature vector for emotion recognition is obtained according to the topic distribution;

[0012] Extracting a feature set from the discrete-time ECG signal using Fourier transform and statistics for obtaining frequency domain features and calculating heart rate variability, and processing the feature set extracted from the discrete-time ECG signal using a convolutional neural network to obtain an ECG feature vector; and,

[0013] A feature set is extracted from the discrete-time body temperature signal by calculating a moving average, and a long short-term memory network is used to process the feature set extracted from the discrete-time body temperature signal to obtain a body temperature feature vector; S3, a deep neural network EmoSynNet is used to perform modality availability detection, agent modality feature adversarial generation, and emotional state prediction on each modality.

[0014] Preferably, the multimodal emotion recognition method converts the character sequence into a term vector according to the selected preprocessing, constructs a topic distribution for the term vector using latent Dirichlet allocation, and obtains a context feature vector for emotion recognition according to the topic distribution, including:

[0015] Use the following formula to construct the topic distribution:

[0016]

[0017] Among them, Θ Text is the topic distribution of the character sequence Text, W Text is the set of term vectors in the character sequence Text, θ Text,topic is the weight of the topic in the character sequence Text, φ topic,w Represents the weight of word w in the topic topic;

[0018] The obtained topic distribution is converted into contextual features related to sentiment analysis using the following formula: C ,in:

[0019] fC =∑ topic α topic· Θ Text [topic]

[0020] Among them, α topic is the weight associated with the topic, Θ Text [topic] is the distribution of the character sequence Text on the topic topic.

[0021] Preferably, in the multimodal emotion recognition method for augmented reality, using the deep neural network EmoSynNet to perform modality availability detection on a modality comprises:

[0022] Score the input of modality i:

[0023] v i =σ(W i f i +b i )

[0024] Among them, f i is the input feature of the modality i, W i and b i are the weights and biases between the scoring network layers contained in the deep neural network EmoSynNet, and σ is the Sigmoid activation function;

[0025] Use the activation function softmax to make a comprehensive judgment on all inputs of this mode and obtain the corresponding value V I :

[0026]

[0027] Among them, α i The importance weight of the state i, the importance weight is obtained through training; v i is the usability score of the modality i, which ranges between 0 and 1;

[0028] Judge V I Is it greater than a predetermined availability threshold? If so, it is determined that the modality i is available; otherwise, it is determined that the modality i is unavailable.

[0029] Preferably, in the multimodal emotion recognition method for augmented reality, the deep neural network EmoSynNet includes a generative adversarial network, the generative adversarial network includes a generator network and a discriminator network, and the use of the deep neural network EmoSynNet to perform adversarial generation of proxy modality features includes:

[0030] Using the activation function tanh, a conditional vector c is created based on the available modal features:

[0031] The generator network generates proxy image modality features f using the conditional vector c and random noise I ',in:

[0032] f I ′=G(z,c)=tanh(W g [z,c]+b g )

[0033] Where z is a random noise vector sampled from a normal distribution, G is the generator function, and W g and b g They are the generator weight and bias respectively;

[0034] The discriminator network is used to evaluate the generated features f I Is ′ close enough to the actual image features?

[0035] D(f I ′,c)=σ(W d [f I ′,c]+b d )

[0036] Where D is the discriminator function, W d and b b are the weights and biases of the discriminator network; σ is the Sigmoid activation function used to output the authenticity score of the agent modality features;

[0037] Among them, the generator is used to minimize its own loss function L G :

[0038] L G = -logD(G(z,c))

[0039] The discriminator is used to minimize its own loss function L D :

[0040] L D = -[logD(f I )+log(1-D(G(z,c)))];

[0041] Adversarial training is performed by alternately optimizing the loss functions of the generator and the discriminator.

[0042] Preferably, in the multimodal emotion recognition method for augmented reality, the deep neural network EmoSynNet includes a multi-head self-attention mechanism, and the use of the deep neural network EmoSynNet to predict the emotion state includes:

[0043] Fuse all available modal features with the generated proxy modal features to obtain fused features;

[0044] Processing the fused features through the multi-head self-attention mechanism;

[0045] A fully connected layer is used to output the sentiment classification.

[0046] On the other hand, a multimodal emotion recognition device for augmented reality is provided, comprising a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of any of the multimodal emotion recognition methods for augmented reality as described above.

[0047] On the other hand, a computer program product is provided, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of the multimodal emotion recognition method for augmented reality as described in any one of the above are implemented.

[0048] On the other hand, a multimodal emotion recognition system for augmented reality is provided, comprising:

[0049] A multimodal data acquisition module is used to acquire multimodal data input by the user during the interaction phase of the augmented reality environment, wherein the multimodal data includes: voice, image, text and vital sign data, wherein the vital sign data includes: electrocardiogram data and body temperature data, and obtain a discrete-time audio signal of the voice, a two-dimensional pixel matrix of the image and a character sequence of the text, a discrete-time electrocardiogram signal of the electrocardiogram data and a discrete-time body temperature signal of the body temperature data;

[0050] The emotional information processing module is used to obtain the feature vectors of each modality, including: a speech modality processing submodule, an image modality processing submodule, a context modality processing submodule and a physical sign modality processing submodule; wherein:

[0051] The speech modality processing submodule is used to extract a feature set of the discrete-time audio signal, and process the feature set of the discrete-time audio signal using a deep belief network to obtain a speech feature vector for emotion recognition;

[0052] The image modality processing submodule is used to extract a feature set of the two-dimensional pixel matrix, and use a residual network to process the feature set of the two-dimensional pixel matrix to obtain an image feature vector for emotion recognition;

[0053] The context modality processing submodule is used to convert the character sequence into a term vector according to the selected preprocessing, construct a topic distribution for the term vector using latent Dirichlet allocation, and obtain a context feature vector for emotion recognition according to the topic distribution;

[0054] The physical sign modality processing submodule is used to extract a feature set from the discrete-time ECG signal using Fourier transform and statistics for obtaining frequency domain features and calculating heart rate variability, and use a convolutional neural network to process the feature set extracted from the discrete-time ECG signal to obtain an ECG feature vector; and, extract a feature set from the discrete-time body temperature signal by calculating a moving average, and use a long short-term memory network to process the feature set extracted from the discrete-time body temperature signal to obtain a body temperature feature vector;

[0055] The multimodal emotion fusion and prediction module is used to use the deep neural network EmoSynNet to perform modality availability detection, proxy modality feature adversarial generation, and emotion state prediction for each modality.

[0056] Preferably, the multimodal emotion recognition system is characterized in that the context modality processing submodule is further used for:

[0057] Use the following formula to construct the topic distribution:

[0058]

[0059] Among them, Θ Text is the topic distribution of the character sequence Text, W Text is the set of term vectors in the character sequence Text, θ Text,topic is the weight of the topic in the character sequence Text, φ topic,w Represents the weight of word w in the topic topic;

[0060] The obtained topic distribution is converted into contextual features related to sentiment analysis using the following formula: C ,in:

[0061] f C =∑ topic α topic ·Θ Text [topic]

[0062] Among them, α topic is the weight associated with the topic, Θ Text [topic] is the distribution of the character sequence Text on the topic topic.

[0063] Preferably, in the multimodal emotion recognition system, the selected correlation analysis algorithm is a typical correlation analysis algorithm; and the multimodal emotion fusion and prediction module is further used for:

[0064] The two modal eigenvectors f for correlation analysis are analyzed according to the following formula: i and fj Perform projection to obtain the projected new eigenvector f i ′ and f j ′:

[0065]

[0066] in, and The characteristic vector f is calculated by the canonical correlation analysis algorithm i and f j The linear transformation matrix with the greatest correlation in the new space, where i and j are index numbers;

[0067] The eigenvector f is calculated according to the following formula i ′ and f j The correlation score ρ(f i ′,f j ′):

[0068]

[0069] Here, cov(·) represents the covariance calculation, and σ represents the standard deviation calculation.

[0070] Preferably, in the multimodal emotion recognition system, in the multimodal emotion fusion and prediction module, using the deep neural network EmoSynNet to perform modality availability detection on one modality includes:

[0071] Score the input of modality i:

[0072] v i =σ(W i f i +b i )

[0073] Among them, f i is the input feature of the modality i, W i and b i are the weights and biases between the scoring network layers contained in the deep neural network EmoSynNet, and σ is the Sigmoid activation function;

[0074] Use the activation function softmax to make a comprehensive judgment on all inputs of this mode and obtain the corresponding value V I :

[0075]

[0076] Among them, α i The importance weight of the state i, the importance weight is obtained through training; v iis the usability score of the modality i, which ranges between 0 and 1;

[0077] Judge V I Is it greater than a predetermined availability threshold? If so, it is determined that the modality i is available; otherwise, it is determined that the modality i is unavailable.

[0078] The above technical solution has the following technical effects:

[0079] The technical solution of the embodiment of the present invention can not only process multimodal data, but also, when certain modal information is missing, generate proxy feature vectors and use other modal emotion data as a supplement to achieve more accurate and natural emotion recognition and interaction.

[0080] Specifically, the technical solution of the embodiment of the present invention has the following advantages:

[0081] (1) Using advanced data collection technology to collect multiple types of data from the user, including voice, image, text, and vital sign data; this provides rich and diverse information for in-depth analysis and understanding of the user's emotional state;

[0082] (2) Deep belief network, residual network and latent Dirichlet allocation algorithm are used to process and analyze data of different modalities, thus ensuring that the emotional features of each modality can be accurately extracted and analyzed;

[0083] (3) When faced with scenarios where modal data is missing, the system adapts through a mechanism of modal availability checking and generating proxy feature vectors. This allows the system to effectively create proxy features that are consistent with the existing modal data by using the generator and discriminator to compete with each other when certain modal information is missing. This strategy enables the system to effectively fuse and utilize information from each modality in the absence of complete information, thus ensuring the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a flowchart of a multimodal emotion recognition method for augmented reality according to an embodiment of the present invention;

[0085] Figure 2 Schematic diagram of the overall framework of a multimodal emotion recognition system for augmented reality according to an embodiment of the present invention;

[0086] Figure 3 It is a schematic diagram of the process of performing multimodal emotion recognition by a multimodal emotion recognition system for augmented reality according to an embodiment of the present invention. DETAILED DESCRIPTION

[0087] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0088] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0089] Embodiment 1:

[0090] Figure 1 FIG. 1 is a flow chart of a multimodal emotion recognition method for augmented reality according to an embodiment of the present invention. Figure 1 The multimodal emotion recognition method for augmented reality of this embodiment includes the following steps:

[0091] S1, collecting multimodal data input by the user in the augmented reality environment, the multimodal data includes: voice, image, text and vital sign data, the vital sign data includes: electrocardiogram data and body temperature data, obtaining a discrete time audio signal of the voice, a two-dimensional pixel matrix of the image, a character sequence of the text, a discrete time electrocardiogram signal of the electrocardiogram data and a discrete body temperature signal of the body temperature data;

[0092] S2, obtain the feature vectors of each modality for emotion recognition, where:

[0093] Extract the feature set of discrete-time audio signals, and use a deep belief network to process the feature set of discrete-time audio signals to obtain a speech feature vector for emotion recognition;

[0094] Extracting a feature set of a two-dimensional pixel matrix, and processing the feature set of the two-dimensional pixel matrix using a residual network to obtain an image feature vector for emotion recognition;

[0095] According to the selected preprocessing, the character sequence is converted into a term vector, the topic distribution of the term vector is constructed using latent Dirichlet allocation, and the context feature vector for sentiment recognition is obtained according to the topic distribution;

[0096] Extracting a feature set from the discrete-time ECG signal using Fourier transform and statistics for obtaining frequency domain features and calculating heart rate variability, and processing the feature set extracted from the discrete-time ECG signal using a convolutional neural network to obtain an ECG feature vector; and,

[0097] A feature set is extracted from the discrete-time body temperature signal by calculating a moving average, and a long short-term memory network is used to process the feature set extracted from the discrete-time body temperature signal to obtain a body temperature feature vector;

[0098] S3 uses the deep neural network EmoSynNet to perform modality availability detection, agent modality feature adversarial generation, and emotional state prediction for each modality.

[0099] Embodiment 2:

[0100] Figure 2 FIG. 1 is a schematic diagram of the overall framework of a multimodal emotion recognition system for augmented reality according to an embodiment of the present invention. Figure 2 , the multimodal emotion recognition system of this embodiment includes: a multimodal data acquisition module, an emotion information processing module and a multimodal emotion fusion and prediction module. Among them, the multimodal data acquisition module is used to collect multimodal data input by the user in the interactive stage of the augmented reality environment; in one implementation, data is collected through various sensors on the user side such as microphones, cameras, physiological detection equipment, etc. The collected multimodal data includes: various types of data such as voice, image, text, vital sign data, etc., to provide original input for subsequent emotion analysis. Among them, the emotion information processing module uses different algorithms to perform emotion data analysis on inputs of different modalities respectively, and finally outputs emotion features; the multimodal emotion feature fusion and prediction module is used to process the emotion feature data extracted by the emotion information processing module, and the processing here includes: modality availability check, proxy feature vector generation and final emotion feature fusion and prediction.

[0101] In a specific implementation, the multimodal data acquisition module reads the voice, image and text data output by the user during the interaction phase, and collects data in different modalities, including: audio signal collection, image data collection, text data collection and vital sign data collection; wherein:

[0102] Audio signal acquisition includes the process of converting a continuous-time audio signal s(t), where t represents continuous time, into a discrete-time audio signal, which is defined as follows:

[0103] s[n]=s(nT)

[0104] Where s[n] represents the signal obtained by sampling the continuous time signal s(t) at discrete time points, n is the discrete time index, and T = 1 / f s is the sampling interval, f s is the sampling frequency, that is, the number of samples per second;

[0105] Image data acquisition includes: processing the image data into a two-dimensional pixel matrix I (x, y), defined as follows:

[0106] I(x,y)=[R(x,y),G(x,y),B(x,y)]

[0107] Where (x, y) represents the pixel coordinates, and [R(x, y), G(x, y), B(x, y)] represents the RGB color channels contained in the pixel at (x, y);

[0108] Text data collection includes: representing text data as a character sequence Text, which is defined as follows:

[0109] Text = {c 1 ,c 2 ,…,c i ,…,c N}

[0110] Among them, c i Represents the i-th character in the text, with a total of N characters.

[0111] The vital sign data collection includes: the process of converting the continuous-time ECG signal h(t) and the continuous-time body temperature signal θ(t), where t represents continuous time, into discrete-time ECG signal and discrete-time body temperature signal, which are defined as follows:

[0112] h[n]=h(nT)

[0113] θ[n]=θ(nT)

[0114] Where h[n] and θ[n] represent the signals obtained by sampling the continuous-time ECG signal h(t) and the continuous-time body temperature signal θ(t) at discrete time points, n is the discrete time index, T is the sampling interval, and h[nT] is the sampling interval. θ[nT] f h With f θ They are the sampling frequencies of ECG and body temperature, that is, the number of samples per second.

[0115] Furthermore, the emotion information processing module includes four parts, which correspond to the emotion calculation and analysis of three modalities, namely: speech modality processing submodule, image modality processing submodule, context modality processing submodule and vital sign modality processing submodule.

[0116] Among them, the speech modality processing submodule first extracts features from the discrete time audio signal s[n], which is defined as follows:

[0117] F(n)=Φ s (s[n])

[0118] Among them, F(n) is the feature set extracted from the original audio signal s[n], Φ sis a feature extraction function, including spectrum analysis, energy distribution, etc.; then, the speech modality processing submodule will use deep belief networks (DBNs) to process the extracted feature set, which is defined as follows:

[0119] H sound,l =RBM l (H sound,l-1 )

[0120] f S =H L =RBM L (H sound,L-1 )

[0121] Among them, H sound,l represents the output of the lth layer of the deep belief network, H sound,0 =F n is the initial input feature, RBM l is the restricted Boltzmann machine (RBM) located at the lth layer of the network, f S It is the output of the last layer of restricted Boltzmann machine, that is, the speech feature representation used for subsequent emotion recognition, that is, the speech feature vector.

[0122] The image modality processing submodule first extracts features from the two-dimensional pixel matrix I(x,y), which is defined as follows:

[0123] F img =Φ img (I(x,y))

[0124] Among them, F img is the feature set extracted from the original two-dimensional pixel matrix, Φ img It is a feature extraction function, including edge detection, texture analysis, etc. Subsequently, the image modality processing submodule will use the residual network (Residual Network, ResNet) to process the extracted feature set, which is defined as follows:

[0125] H img,l =Conv l (H img,l-1 )+H img.l-1

[0126] f I =H img,L =Conv L (H img,L-1 )+H img.L-1

[0127] Among them, H img,lis the output of the lth residual block (ResBlock). For the initial residual block, H img,0 =F img , that is, the input image feature set, Conv l (·) represents the convolution operation in the lth residual block, f I It is the output of the last residual block, that is, the image feature representation used for subsequent emotion recognition, that is, the image feature vector.

[0128] The contextual modal processing submodule first performs selected preprocessing on the acquired text data, including word segmentation, stop word removal, stem extraction, etc., and converts the character sequence Text into a series of term vectors, represented as W Text Subsequently, this submodule will use Latent Dirichlet Allocation (LDA) to construct the topic distribution of the preprocessed term vectors. The specific definition is:

[0129]

[0130] Among them, Θ Text is the topic distribution of the previous input character sequence Text, W Text is the set of word vectors in the character sequence, θ Text,topic is the weight of the topic in the character sequence, φ topic,w represents the weight of word w in the topic. Next, the obtained topic distribution is converted into contextual features f related to sentiment analysis C , defined as follows:

[0131] f C =∑ topic α topic ·Θ Text [topic]

[0132] Among them, α topic is the weight associated with the topic, Θ Text [topic] is the distribution of text on topic.

[0133] For the extracted feature vector information, the multimodal emotional feature fusion and prediction module first performs a modality availability check, aiming to filter invalid data through this step. Among them, the modality availability check uses canonical correlation analysis (CCA) to calculate the correlation score ρ between each pair of input modalities. To calculate the correlation score, first obtain the new feature vector after projection through the following formula:

[0134]

[0135] Among them, f i and f j Given two input feature vectors, and The linear transformation matrix with the maximum correlation between the two modal features in the new space is obtained through CCA calculation;

[0136] The physical sign modality processing submodule consists of two parts, namely ECG data processing and body temperature data processing. ECG data processing first extracts features from the discrete-time ECG signal h[n]:

[0137] F H (n) = Φ h (h[n])

[0138] Among them, F H (n) is the feature set extracted from the original ECG signal, Φ h It is a feature extraction function, including Fourier transform, obtaining frequency domain features and calculating the statistics of heart rate variability (HRV). The ECG data processing will then use convolutional neural networks (CNNs) to process the extracted feature set, which is defined as follows:

[0139] H hr,l =ReLU(W l *H hr,l-1 +b l )

[0140] Among them, H hr,l represents the output of the lth layer of CNN, H hr,0 =F H (n) is the initial input, W l and b l Represent the weight and bias of the lth layer respectively, * represents the convolution operation, ReLU is the activation function, and f H =H hr,L It is the output of the last layer of CNN and is also the obtained ECG feature, namely the ECG feature vector.

[0141] The physical sign modality processing submodule is also used to process the body temperature data. It first extracts the features of the discrete time body temperature signal θ[n]:

[0142] F T (n) = Φ θ (θ[n])

[0143] Among them, F T (n) is the feature set extracted from the original body temperature signal, Φ θis a feature extraction function, which includes calculating moving average to smooth the data and extracting temperature gradient. This step then uses a long short-term memory network (LSTM) to process the extracted features:

[0144] H temp,t =LSTM(H temp,t-1 +F T (t); θ)

[0145] Among them, H temp,t is the output of the LSTM unit at time t, H temp,t-1 is the output of the previous time step, F T (t) is the input feature of the current time step, Θ is the parameter of the LSTM model, and f T =H temp,T is the final body temperature feature, that is, the body temperature feature vector, and T represents the last time step.

[0146] Furthermore, the multimodal emotion fusion and prediction module is mainly composed of a composite deep neural network called EmoSyncNet (EmotionSynchronized Network). The composite network integrates multiple deep neural networks to form different functional components, including modality availability detection, agent modality feature adversarial generation, and emotion state prediction.

[0147] Furthermore, the purpose of the modality availability detection is to determine the availability of input modality features. In the embodiment of the present invention, a small neural network is integrated in EmoSyncNet to determine whether the input modality is available. This part can be expanded in detail to I As an example, first score the modal input:

[0148] v i =σ(W i f i +b i )

[0149] Among them, f i is the input feature of the i-th image modality, W i and b i is the weight and bias between the scoring network layers, σ is the Sigmoid activation function, and v i is the availability score of the modality i, and its value is between 0 and 1, where 0 means unavailable and 1 means available.

[0150] Then, another activation function is used to make a comprehensive judgment on all inputs of this modality:

[0151]

[0152] Among them, α i are the importance weights of the corresponding modalities, which are obtained through training and are used to evaluate the influence of different inputs of the same modality.

[0153] Finally, this part sets a threshold to determine whether the input modality is sufficiently usable, which can be expressed as:

[0154]

[0155] Among them, τ I is the preset image modality availability check threshold, which is used to determine whether the image modality in the example is available.

[0156] Furthermore, the adversarial generation of the proxy modality features is mainly implemented by the Generative Adversarial Network (GAN) in EmoSyncNet. Taking the unavailable image modality as an example, we first need to create a conditional vector based on the available modality features:

[0157] c=tanh(W c [f S ,f C ,f H ,f T ]+b c )

[0158] Among them, [f S ,f C ,f H ,f T ] is the link of features of other modalities except image modality, W c and b c are the weights and biases of the fusion layer respectively, and tanh is the activation function used to generate the conditional vector c, which guides the generation process.

[0159] Furthermore, the generator network will use the conditional vector and random noise to generate the proxy image modality features, which can be specifically expressed as:

[0160] f I ′=G(z,c)=tanh(W g [z,c]+b g )

[0161] Where z is a random noise vector sampled from a normal distribution, G is the generator function, and W g and b g are the generator weights and biases respectively.

[0162] The discriminator network is used to evaluate the generated features fI Is ′ close enough to the actual image features?

[0163] D(f I ′,c)=σ(W d [f I ′,c]+b d )

[0164] Where D is the discriminator function, W d and b d are the weights and biases of the discriminator network, and σ is the Sigmoid activation function used to output the authenticity score of the agent modality features.

[0165] In the above two steps, the generator will try to minimize its loss function:

[0166] L G = -logD(G(z,c))

[0167] Where L G is the loss function of the generator. The goal of the generator is to maximize the probability that the discriminator judges its output as "real", that is, to try to make the discriminator unable to distinguish between proxy modal features and real modal features. The discriminator will also try to minimize its loss function in this process:

[0168] L D = -[logD(f I )+log(1-D(G(z,c)))]

[0169] Among them, L D is the loss function of the discriminator, whose goal is to correctly classify the real modal features and the proxy modal features. Adversarial training is performed by alternately optimizing the loss functions of the generator and the discriminator. The generator tries to deceive the discriminator, while the discriminator resists this deception. In this way, the generation ability of the generator will gradually improve, and the judgment ability of the discriminator will also improve accordingly. This process can be described by the following iterative algorithm:

[0170] Fix G and update D to minimize L D ;

[0171] Fix D and update G to minimize L G .

[0172] Furthermore, the emotional state prediction is implemented by the Multi-Head Self-Attention mechanism integrated into EmoSyncNet, which is an advanced technology borrowed from the Transformer architecture and can better handle complex relationships in multimodal data. The emotional state prediction aims to use the integrated modal features, whether original or generated proxy modal features, to predict the current emotional state.

[0173] First, the module fuses all available modal features with the generated proxy modal features, and continues to use the missing image modal features as an example:

[0174] f combined =Concat(f S ,f C ,f H ,f T ,f I ′)

[0175] The fused features are processed through the self-attention mechanism, allowing the model to dynamically allocate attention between different modalities:

[0176]

[0177] MultiHead(Q,K,V)=Concat(head 1 ,…,head h )W O

[0178] head i =Attention(QW i Q ,KW i K ,VW i V )

[0179] Among them, Q, K, V are query, key and value matrices respectively, and W i Q ,W i K ,W i V is the weight matrix of different heads, W O is the output linear transformation matrix, d k is the dimension of the key vector, used to scale the dot product and reduce the variance of the calculation result.

[0180] Furthermore, a fully connected layer is used to output the sentiment classification:

[0181] Emotion=softmax(We MultiHead(f combined ,f combined ,f combined )+b e )

[0182] Among them, W e and b e are the weights and biases of the fully connected layer. This module effectively processes multimodal inputs through a multi-head self-attention mechanism, which improves the depth and breadth of the model's understanding of the relationship between different information sources. The introduction of the self-attention mechanism can help the model better switch and predict between different emotional states, especially in situations where the emotions are more subtle or complex. In a specific implementation, the emotion classification includes one or more of the following: happy, angry, disgusted, fearful, sad, and surprised.

[0183] Figure 3 The flowchart of the method for multimodal emotion recognition in the multimodal emotion recognition system for augmented reality according to an embodiment of the present invention is as follows. Figure 3 The method for performing emotion recognition in a multimodal emotion recognition system for augmented reality according to an embodiment of the present invention comprises the following steps:

[0184] Collect multimodal data and perform multimodal data input;

[0185] According to the input multimodal data, multimodal emotional information processing is performed and multimodal emotional features are extracted;

[0186] According to the modal sentiment characteristics, the modality is checked for availability; if there is no modality that is unavailable, the modal feature vector is input into the deep learning neural network for multimodal feature fusion to perform multimodal sentiment fusion; in the specific implementation, for the available, i.e., effective modalities, their feature vectors are directly input into the above-mentioned deep learning neural network; if there is a modality that is unavailable, i.e., an invalid modality, GAN is used to generate a proxy feature vector for the invalid modality, and the proxy feature vector is input into the above-mentioned deep learning neural network for multimodal sentiment fusion; a multi-head self-attention mechanism is used for sentiment prediction, and a final prediction of sentiment classification is generated, such as generating sentiment feedback.

[0187] In a specific implementation, when performing emotion recognition, the multimodal emotion recognition system for augmented reality of an embodiment of the present invention executes various steps corresponding to the functions or steps implemented by the various functions or program modules contained in the above-mentioned system, and will not be repeated here.

[0188] Embodiment three:

[0189] The present invention also provides a multimodal emotion recognition device for augmented reality, which includes a processor and a memory and a computer program stored in the memory and run on the processor. The processor includes one or more processing cores, and the memory is connected to the processor via a bus. The memory is used to store program instructions. When the processor executes the computer program, the steps in the method embodiment of the present invention are implemented.

[0190] Further, as an executable solution, the device may be a computer unit, which may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art may understand that the composition structure of the above-mentioned computer unit is only an example of a computer unit and does not constitute a limitation on the computer unit. It may include more or less components than the above-mentioned, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.

[0191] Further, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, and various interfaces and lines are used to connect the various parts of the entire computer unit.

[0192] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer unit by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0193] Embodiment 4:

[0194] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0195] If the module / unit integrated in the computer unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0196] Embodiment five:

[0197] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described above are implemented.

[0198] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method for augmented reality, characterized in that: include: S1, collecting multimodal data input by the user in the augmented reality environment, the multimodal data including: voice, image, text and vital sign data, the vital sign data including: electrocardiogram data and body temperature data, obtaining a discrete time audio signal of the voice, a two-dimensional pixel matrix of the image, a character sequence of the text, a discrete time electrocardiogram signal of the electrocardiogram data and a discrete time body temperature signal of the body temperature data; S2, obtain the feature vectors of each modality for emotion recognition, where: Extracting a feature set of the discrete-time audio signal, and processing the feature set of the discrete-time audio signal using a deep belief network to obtain a speech feature vector for emotion recognition; Extracting a feature set of the two-dimensional pixel matrix, and processing the feature set of the two-dimensional pixel matrix using a residual network to obtain an image feature vector for emotion recognition; According to the selected preprocessing, the character sequence is converted into a term vector, a topic distribution is constructed for the term vector using latent Dirichlet allocation, and a context feature vector for emotion recognition is obtained according to the topic distribution; Extracting a feature set from the discrete-time ECG signal using Fourier transform and statistics for obtaining frequency domain features and calculating heart rate variability, and processing the feature set extracted from the discrete-time ECG signal using a convolutional neural network to obtain an ECG feature vector; and, Extracting a feature set from the discrete-time body temperature signal by calculating a moving average, and processing the feature set extracted from the discrete-time body temperature signal using a long short-term memory network to obtain a body temperature feature vector; S3 uses the deep neural network EmoSynNet to perform modality availability detection, agent modality feature adversarial generation, and emotional state prediction for each modality.

2. The multimodal emotion recognition method for augmented reality according to claim 1, characterized in that: According to the selected preprocessing, the character sequence is converted into a term vector, a topic distribution is constructed for the term vector using latent Dirichlet allocation, and a context feature vector for emotion recognition is obtained according to the topic distribution, including: Use the following formula to construct the topic distribution: Among them, Θ θText is the topic distribution of the character sequence Text, W Text is the set of term vectors in the character sequence Text, θ Text,topic is the weight of the topic in the character sequence Text, φ topic,w Represents the weight of word w in the topic topic; The obtained topic distribution is converted into contextual features related to sentiment analysis using the following formula: C ,in: f C =∑ topic a topic ·I Text [topic] Among them, α topic is the weight associated with the topic, Θ Text [topic] is the distribution of the character sequence Text on the topic topic.

3. The multimodal emotion recognition method for augmented reality according to claim 1, characterized in that: Using the deep neural network EmoSynNet to detect modality availability of a modality includes: Score the input of modality i: v i =σ(W i f i +b i ) Among them, f i is the input feature of the modality i, W i and b i are the weights and biases between the scoring network layers contained in the deep neural network EmoSynNet, and σ is the Sigmoid activation function; Use the activation function softmax to make a comprehensive judgment on all inputs of this mode and obtain the corresponding value V I : Among them, α i The importance weight of the state i, the importance weight is obtained through training; v i is the usability score of the modality i, which ranges between 0 and 1; Judge V I Is it greater than a predetermined availability threshold? If so, it is determined that the modality i is available; otherwise, it is determined that the modality i is unavailable.

4. The multimodal emotion recognition method for augmented reality according to claim 1, characterized in that: The deep neural network EmoSynNet includes a generative adversarial network, the generative adversarial network includes a generator network and a discriminator network, and the use of the deep neural network EmoSynNet to perform agent modality feature adversarial generation includes: Using the activation function tanh, a conditional vector c is created based on the available modal features: The generator network generates proxy image modality features f using the conditional vector c and random noise I ',in: f I ′=G(z,c)=tanh(W g [z,c]+b g ) Where z is a random noise vector sampled from a normal distribution, G is the generator function, and W g and b g They are the generator weight and bias respectively; The discriminator network is used to evaluate the generated features f I Is ′ close enough to the actual image features? D(f I ′,c)=σ(W d [f I ′,c]+b d ) Where D is the discriminator function, W d and b d are the weights and biases of the discriminator network; σ is the Sigmoid activation function used to output the authenticity score of the agent modality features; Among them, the generator is used to minimize its own loss function L G : L G =-logD(G(z,c)) The discriminator is used to minimize its own loss function L D : L D =-[logD(f I )+log(1-D(G(z,c)))]; Adversarial training is performed by alternately optimizing the loss functions of the generator and the discriminator.

5. The multimodal emotion recognition method for augmented reality according to claim 1, characterized in that: The deep neural network EmoSynNet includes a multi-head self-attention mechanism, and the use of the deep neural network EmoSynNet to predict the emotional state includes: Fuse all available modal features with the generated proxy modal features to obtain fused features; Processing the fused features through the multi-head self-attention mechanism; A fully connected layer is used to output the sentiment classification.

6. A multimodal emotion recognition device for augmented reality, characterized in that: It includes a memory and a processor, the memory stores at least one program, and the at least one program is executed by the processor to implement the steps of the multimodal emotion recognition method for augmented reality as described in any one of claims 1 to 5.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multimodal emotion recognition method for augmented reality as described in any one of claims 1 to 5 are implemented.

8. A multimodal emotion recognition system for augmented reality, characterized in that: include: A multimodal data acquisition module is used to acquire multimodal data input by the user during the interaction phase of the augmented reality environment, wherein the multimodal data includes: voice, image, text and vital sign data, wherein the vital sign data includes: electrocardiogram data and body temperature data, and obtain a discrete-time audio signal of the voice, a two-dimensional pixel matrix of the image and a character sequence of the text, a discrete-time electrocardiogram signal of the electrocardiogram data and a discrete-time body temperature signal of the body temperature data; The emotional information processing module is used to obtain the feature vectors of each modality, including: a speech modality processing submodule, an image modality processing submodule, a context modality processing submodule and a physical sign modality processing submodule; wherein: The speech modality processing submodule is used to extract a feature set of the discrete-time audio signal, and process the feature set of the discrete-time audio signal using a deep belief network to obtain a speech feature vector for emotion recognition; The image modality processing submodule is used to extract a feature set of the two-dimensional pixel matrix, and use a residual network to process the feature set of the two-dimensional pixel matrix to obtain an image feature vector for emotion recognition; The context modality processing submodule is used to convert the character sequence into a term vector according to the selected preprocessing, construct a topic distribution for the term vector using latent Dirichlet allocation, and obtain a context feature vector for emotion recognition according to the topic distribution; and The physical sign modality processing submodule is used to extract a feature set from the discrete-time ECG signal using Fourier transform and statistics for obtaining frequency domain features and calculating heart rate variability, and use a convolutional neural network to process the feature set extracted from the discrete-time ECG signal to obtain an ECG feature vector; and, extract a feature set from the discrete-time body temperature signal by calculating a moving average, and use a long short-term memory network to process the feature set extracted from the discrete-time body temperature signal to obtain a body temperature feature vector; The multimodal emotion fusion and prediction module is used to use the deep neural network EmoSynNet to perform modality availability detection, proxy modality feature adversarial generation, and emotion state prediction for each modality.

9. The multimodal emotion recognition system for augmented reality according to claim 8, characterized in that: In the multimodal emotion fusion and prediction module, using the deep neural network EmoSynNet to perform modality availability detection on one modality includes: Score the input of modality i: v i =σ(W i f i +b i ) Among them, f i is the input feature of the modality i, W i and b i are the weights and biases between the scoring network layers contained in the deep neural network EmoSynNet, and σ is the Sigmoid activation function; Use the activation function softmax to make a comprehensive judgment on all inputs of this mode and obtain the corresponding value V I : Among them, α i The importance weight of the state i, the importance weight is obtained through training; v i is the usability score of the modality i, which ranges between 0 and 1; Judge V I Is it greater than a predetermined availability threshold? If so, it is determined that the modality i is available; otherwise, it is determined that the modality i is unavailable.

Images