Network platform user emotion recognition method and device based on multi-modal representation fusion

Abstract

The invention discloses a network platform user emotion recognition method and device based on multi-modal representation fusion. The method comprises the steps that single-modal advanced semantic representation obtains modal consistency representation and modal difference representation through a consistency and difference representation module; the difference filter filters negative and redundant noise information in modal difference representation in a gating mode; the complementarity characterization module takes the filtered modal difference as guidance information, and generates multi-modal complementarity characterization through multimodal transport and an attention mechanism; platform user information is used as extra prior information to be embedded into the fusion layer; adaptively fusing the modal consistency representation, the modal difference representation, the modal complementarity representation and the user information by adopting a general fusion layer structure to generate global semantic representation; and finally, outputting the emotional state of the platform user through a classifier according to the global semantic representation, and promoting the optimization of the platform user experience by the network platform according to the emotional state of the user. The device comprises a processor and a memory.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal emotion recognition, and in particular to a method and device for network platform user emotion recognition based on multimodal representation fusion. Background Art

[0002] With the rapid development of the Internet, online platforms have become an important part of people's daily lives. Users express their opinions and emotions in multiple modal forms such as text, images, videos, and voice on shopping platforms, social media, and other online platforms, forming multimodal review data. With the development of artificial intelligence technology, especially the advancement of multimodal representation fusion methods, it has become possible to use multiple data sources (such as voice, text, images, etc.) to comprehensively analyze and identify emotions.

[0003] In the field of multimodal representation fusion, existing algorithms focus on the exploration of modal consistency and modal complementarity, while ignoring the positive role of modal difference. Modal difference constructs the link between modal consistency and modal complementarity, but existing algorithms lack research on this relationship, resulting in the need to improve the accuracy of multimodal emotion recognition tasks. Summary of the invention

[0004] The present invention provides a method and device for identifying user emotions on a network platform based on multimodal representation fusion. The present invention aims to enhance multimodal representation fusion by building a connection link of modal consistency and modal complementarity with modal differences, so as to accurately identify the user's emotional tendency, thereby improving the network platform's ability to understand user feedback, promoting the optimization of the platform's user experience, and also providing the network platform with a more insightful emotion analysis tool, as described below:

[0005] In a first aspect, a method for identifying user emotions on a network platform based on multimodal representation fusion is provided, the method comprising:

[0006] The high-level semantic representation of a single modality obtains modality consistency representation and modality difference representation through consistency and difference representation modules; the difference filter filters out negative and redundant noise information in the modality difference representation through gating;

[0007] The complementary representation module uses filtered modal differences as guidance information to generate multimodal complementary representations through multimodal transport and attention mechanisms; embeds platform user information into the fusion layer as additional prior information; and uses a general fusion layer structure to adaptively fuse modal consistency representation, modal difference representation, modal complementary representation, and user information to generate a global semantic representation;

[0008] Finally, the global semantic representation is outputted through a classifier to represent the emotional state of platform users, and the network platform promotes the optimization of platform user experience based on the emotional state of users.

[0009] The method also includes: constructing a semantic similarity measurement function through the spatial distance between feature vectors and sentiment representation; constructing the semantic similarity measurement function as a loss function, penalizing the parameters of the modal joint encoder-decoder and the complementary representation module, and enhancing the consistency of the model and the complementary representation learning ability; the semantic similarity loss, KL divergence and classifier loss will jointly update the model parameters.

[0010] The high-level semantic representation of the single modality is obtained by using the consistency and difference representation module to obtain the modality consistency representation and the modality difference representation, which are specifically:

[0011] The single modal features are encoded x through the modality joint encoder and the universal fusion layer m Mapping to the distribution of latent space:

[0012]

[0013] in, is x m Through the output of the modality joint encoder and the universal fusion layer, μ φ (x′) and is the mean and log variance of the latent variable:

[0014] μ φ (x′)=x′

[0015]

[0016] from Sampling latent variables from a distribution

[0017]

[0018] Where exp(·) represents the natural exponential function, ∈ is the standard normal distribution. Gaussian noise, I is the identity matrix, and the latent variable z is mapped back to the original data space through the modal joint decoder to generate the modal reconstruction feature variable:

[0019]

[0020] in, is the reconstructed feature variable of each single mode; the modal reconstructed feature x r_m With the original modal feature x m Subtracting them, the remaining part is the feature representation of each modality that is specific to the other modalities:

[0021] x d_m =x m -x r_m ,m∈{v,l,a}

[0022] in, Represents a unimodal specific modality representation, and all unimodal specific representations x d_m Obtain preliminary modality difference representation through universal fusion layer The consistency and difference representation module generates the modal consistency representation x in a hierarchical and progressive manner. con and preliminary modal difference characterization

[0023] The universal fusion layer is applicable to the parts of the whole framework that need to be fused. The universal fusion layer fuses all input features to obtain fused feature representation by constructing feature channel attention. Assuming that the input feature vector Construct a new feature dimension for each feature vector, namely: Concatenate all feature dimensions into a single feature vector:

[0024]

[0025] Among them, Concat represents dimension concatenation, followed by the fusion layer parameters Multiply to get the fused attention factor:

[0026]

[0027] in, is the fusion attention factor, softmac is the activation function, and the input feature with att f Multiply them, and then pass through the batch normalization layer BN, the hyperbolic tangent activation function Tanh and the discard layer Dropout to get the fused output:

[0028]

[0029] Where T represents transpose, Represents the fusion output result.

[0030] In the second aspect, a network platform user emotion recognition device based on multimodal representation fusion, the device comprises: a processor and a memory, the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0031] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0032] The beneficial effects of the technical solution provided by the present invention are:

[0033] (1) This paper defines the relationship between modal consistency, modal difference and modal complementarity, and proposes a multimodal representation fusion framework that connects modal consistency and modal complementarity with modal difference as the link;

[0034] (2) The present invention designs a difference filter in a parameter-gated manner, which can effectively filter out redundant noise data in modal differences and reduce modal feature offsets; then, using differences as guiding information, the present invention designs an integrated module for extracting multimodal complementary representations, which mines complementary associations in multimodal data through multimodal transport and attention mechanisms; the present invention also proposes a semantic similarity measurement method, which measures the semantic similarity of two semantic feature vectors by comparing their spatial distance and sentiment representation;

[0035] (3) Finally, the global semantic representation output by the representation fusion framework is used to obtain the emotional state of the users of the network platform through a classifier. This promotes the optimization of the user experience of the network platform based on the user's emotional state, and also provides the network platform with a more insightful sentiment analysis tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a method for identifying user emotions on a network platform based on multimodal representation fusion;

[0037] Figure 2 This is the network structure diagram of CDCnet;

[0038] Figure 3 It is a structural diagram of the general fusion layer;

[0039] Figure 4 is the structural diagram of the modal joint codec;

[0040] Figure 5 A structural diagram of the complementarity characterization module;

[0041] Figure 6 A structural diagram for differential interaction aggregation;

[0042] Figure 7 Schematic diagram of semantic similarity measurement;

[0043] Figure 8Spatial embedding visualization of classification results on IEMOCAP and MELD datasets;

[0044] Fig. 9 Visualization of the distances between semantic representations generated for different distance metrics. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0046] Current multimodal representation fusion methods mainly focus on two aspects of multimodal data: modal consistency and modal complementarity. They use complex fusion strategies to improve multimodal representation fusion and extract the consistency and complementarity between different modes by developing high-quality fusion frameworks. Although these methods are promising, they ignore the important role of differences in multimodal data for multimodal representation fusion, and fail to recognize the link between consistency and complementarity, resulting in the need to improve the accuracy of multimodal emotion recognition tasks.

[0047] Modal difference is the link between modal consistency and modal complementarity, and the three are in a progressive relationship in modal representation. There are two main reasons for this: 1) First, modal consistency and modal difference are in an antagonistic relationship, and they are two non-intersecting features. Second, modal difference is the basis of modal complementarity. Constructing complementarity through differences can generate more accurate and expressive complementary features. 2) Modal difference is the modality-specific information carried by individual modalities, which itself has rich semantic representation information. Therefore, taking modal difference as the starting point for the design of multimodal representation fusion algorithm is the key to improving the accuracy of multimodal emotion recognition.

[0048] Example 1

[0049] The embodiment of the present invention proposes a network platform user emotion recognition method based on multimodal representation fusion, and names it as the consistency, difference, and complementarity representation fusion framework (CDCnet). Figure 2As shown in the figure. CDCnet is a representation fusion framework that connects modal consistency and modal complementarity through a layer-by-layer progressive generation method, with modal difference as the link. Its components include: modal encoding module, consistency and difference representation module, complementary representation module and fusion output module, among which the consistency and difference representation module and complementary representation module are the core components. Specifically, CDCnet first generates a high-level semantic representation of a single modality through the modal encoding module. Then, the high-level semantic representations of all modalities are generated through the modal joint encoder to generate modal consistency, and the modal consistency is then used to generate modal reconstruction features through the modal decoder. The modal reconstruction features are subtracted from the original modal features to generate modal differences. The modal difference is filtered out by the difference filter to remove heterogeneous noise. The complementary representation module then uses the filtered modal difference as the guiding information to generate modal complementarity through multimodal transport and attention mechanism. The attention fusion module adaptively fuses modal consistency, difference, complementary representations and user embedding information to generate multimodal global semantic representations. Finally, the global semantic representation outputs the emotional state of platform users through the classifier. The network platform adjusts the service process and methods according to the user's emotional state, promotes the optimization of the platform user experience, and also provides the network platform with more insightful sentiment analysis tools.

[0050] like Figure 1 As shown, the embodiment of the present invention provides a method for identifying user emotions on a network platform based on multimodal representation fusion, and the technical steps for solving the problem are as follows:

[0051] 101: First, the original multimodal comment data published by users of the network platform is obtained, including data of modes such as video, image, voice, and text, and then the original multimodal comment data is feature encoded through the modality encoding module to obtain a high-level semantic representation of a single modality;

[0052] 102: The general fusion layer is applicable to the parts in CDCnet that need feature fusion. It fuses all input features to obtain fused feature output by constructing feature channel attention. The structure of the general fusion layer is as follows: Figure 3 As shown. There are many applications in CDCnet, such as Figure 2 , Figure 4 , Figure 6 The "universal fusion layer" in the structure. "Fused feature output" refers to the result of multiple inputs at these positions being output through the universal fusion layer.

[0053] 103: The high-level semantic representation of a single modality obtains modality consistency representation and modality difference representation through consistency and difference representation modules;

[0054] 104: In order to reduce the noise interference of modal heterogeneity, the difference filter filters out negative and redundant noise information in the modal difference representation by gating;

[0055] 105: The complementary representation module then uses the filtered modal differences as guidance information to generate multimodal complementary representations through multimodal transport and attention mechanisms;

[0056] Among them, “multimodal complementary representation” Figure 2 "X com ", which will be consistent with the multimodal representation "X con "and multimodal difference representation"X diff "Let's merge together.

[0057] 106: The platform user information is embedded into the fusion layer as an additional prior information; and the universal fusion layer structure is used to adaptively fuse the modal consistency representation, modal difference representation, modal complementarity representation and user information to generate a global semantic representation; finally, the global semantic representation is output through a classifier to output the emotional state of the platform user, and the network platform promotes the optimization of the platform user experience according to the user's emotional state;

[0058] 107:In order to measure the semantic similarity between various feature vectors in CDCnet, a semantic similarity measurement function is constructed through the spatial distance and sentiment representation between feature vectors.

[0059] 108: The semantic similarity metric function is constructed as a loss function to penalize the parameters of the modality joint encoder-decoder and complementary representation module, and enhance the consistency and complementary representation learning ability of the model. The semantic similarity loss, KL divergence and classifier loss will jointly update the model parameters.

[0060] Among them, step 107 and step 108 are the loss functions needed to train CDCnet. Step 107 refers to designing a semantic similarity measurement function, such as Figure 7 As shown in FIG. 1 , the objects it processes include any feature vectors, for example, all feature vectors generated in CDCnet are applicable. Step 108 constructs this semantic similarity measurement function as a loss function to train the CDCnet model. The network structure of CDCnet, the universal fusion layer, the modality joint encoder-decoder, the complementary representation module and the semantic similarity measurement are shown in FIG. Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 7 shown.

[0061] In summary, the embodiment of the present invention designs a network framework based on multimodal representation fusion to identify the emotional state of network platform users. It obtains modal consistency, modal difference and modal complementarity through a layer-by-layer progressive generation method, and modal difference serves as a link to connect modal consistency and modal complementarity. The network platform then promotes the optimization of the platform user experience based on the user's emotional state.

[0062] Example 2

[0063] The scheme in Example 1 is further introduced below in combination with specific calculation formulas and examples, as described below for details:

[0064] Furthermore, in step 101, the original multimodal comment data posted by the user is first obtained, including data in modalities such as video, image, voice, and text. The video data is parsed into data in three modalities: image, voice, and text. Before the original modal data is sent to the downstream task, it needs to be feature encoded to obtain a high-level semantic representation of a single modality. The embodiment of the present invention follows the feature extraction method of previous studies, and uses DenseNet, RoBERTa-Large, and OpenSmile toolkits to extract visual, text, and acoustic features, respectively. In order to ensure that each modality generates the same feature dimension, a linear layer is used for dimensionality conversion operations. The output of the modality encoding module can be expressed as Where m∈{v,l,a}, T represents the time length, d represents the feature dimension, and {v,l,a} represents vision, text, and acoustics, respectively.

[0065] Further, in step 102, the universal fusion layer proposed in the embodiment of the present invention is first introduced. The universal fusion layer is applicable to the part of the entire framework that needs to perform feature fusion. The universal fusion layer fuses all input features to obtain a fused feature representation by constructing feature channel attention. Assuming that the input feature vector Construct a new feature dimension for each feature vector, namely: Concatenate all feature dimensions into a single feature vector:

[0066]

[0067] Among them, Concat represents dimension concatenation, followed by the fusion layer parameters Multiply to get the fused attention factor:

[0068]

[0069] in, is the fusion attention factor, softmax is the activation function, and the input feature with att fMultiply them, and then pass through the batch normalization layer BN, the hyperbolic tangent activation function Tanh and the discard layer Dropout to get the fused output:

[0070]

[0071] Where T represents transpose, Represents the fusion output result.

[0072] Furthermore, in step 103, the consistency and difference characterization module follows a hierarchical and progressive generation method. The embodiment of the present invention uses a modality joint codec to generate a modality consistency representation, adopts the structure of a variational autoencoder, introduces the concept of multimodality, and enables the latent variables sampled in its latent space to be identified as a multimodal consistency representation. First, the single modality feature encoding x is converted into a multimodal consistency representation by using a modality joint encoder and a universal fusion layer. m The distribution mapped to the latent space is as follows:

[0073]

[0074] in, is x m Through the output of the modality joint encoder and the universal fusion layer, μ φ (x′) and is the mean and log variance of the latent variable, expressed as follows:

[0075] μ φ (x′)=x′

[0076]

[0077] Then from Sampling latent variables from a distribution

[0078]

[0079] Where exp(·) represents the natural exponential function, ∈ is the standard normal distribution. Gaussian noise, I is the unit matrix. Then the latent variable z is mapped back to the original data space through the modal joint decoder to generate the modal reconstruction feature variable. As shown below:

[0080]

[0081] Among them, is the reconstructed feature of each single modality. Looking back at the whole process, the modal joint encoder encodes the data of each modality into latent representations. These latent representations contain the representation information of each modality. The universal fusion layer fuses these latent representations to generate the joint mean and log variance to ensure that the information of each modality is integrated together. Then the latent representation is sampled from the joint distribution through the reparameterization technique. The reconstruction loss ensures that the decoder can reconstruct multimodal information from the latent representation, and the KL divergence makes the distribution of the latent representation close to the prior normal distribution. Through the above analysis, the latent representation z generated by the entire modal joint encoder and decoder conforms to the definition and properties of multimodal consistent representation. It integrates the information of different modalities and can express the common characteristics between different modalities. Therefore, it can be determined that the latent representation z is the required multimodal consistent representation.

[0082] At this point, the reconstruction features x of each single modality generated by modal consistency have been obtained r_m For each mode, x r_m represents the maximum amount of information that modal consistency can express for each mode, and is also the optimal representation that modal consistency can achieve for each mode. Since modal consistency is opposite to modal difference, the modal reconstruction feature x r_m With the original modal feature x m Subtracting, the remaining part is the feature representation of each mode specific to other modes. It can be expressed as:

[0083] x d_m =x m -x r_m ,m∈{v,l,a} (8)

[0084] in, represents the specific modality representation of a single modality. Then all the single modality specific representations x d_m Obtain preliminary modality difference representation through universal fusion layer In this way, the entire consistency and difference representation module generates the modal consistency representation x in a hierarchical and progressive manner. con and preliminary modal difference representation

[0085] Further, in step 104, a preliminary representation of the modality difference has now been obtained. However, the difference representation at this time contains both semantic representations that are beneficial to downstream tasks and noise interference caused by heterogeneity between modalities, so a difference filter is designed to adaptively distinguish positive and negative difference representations. First, an encoding function is used to encode the difference representation:

[0086]

[0087] in, is the encoding representation, E f is the encoding function, and its parameterized implementation uses two layers of linear layer functions, θ f is represented as the parameter of the encoding function. Then the gated representation is constructed through the encoding representation:

[0088]

[0089] Among them, σ represents the Sigmoid activation function, Represents a uniform distribution in the range of (0,1). The embodiment of the present invention adds a hard concrete distribution to the gated representation, so that it has a discrete-continuous mixed distribution in the interval (0,1) and allows the result to be continuous in the unit interval. In this way, the encoding parameters can be optimized based on the gradient through the reparameterization technique. Then generate the gated switch:

[0090]

[0091] Among them, ρ is a threshold used to determine the switch size of retaining or filtering the difference representation. In order to transfer the gradient and implement parameter update, it is also necessary to set:

[0092]

[0093] Among them, * indicates gradient truncation. Finally, the preliminary representation of the difference is filtered through the gate switch to obtain the filtered difference representation:

[0094]

[0095] in, The output of is controlled by the difference filter parameters, and through model training, the redundant noise in the difference preliminary representation can be adaptively filtered out, while retaining the difference semantic representation that is beneficial to downstream tasks.

[0096] Furthermore, in step 105, the embodiment of the present invention proposes a new multimodal complementary representation module, which constructs intra-modal and cross-modal complementary representations with self-attention and cross-attention respectively. And with modal differences as guiding information, multimodal data is modeled complementary from the perspective of multimodal transport. And the semantic similarity metric distance between the complementary representation output by the module and the label embedding is used as the learning goal of the entire module to promote the module to learn complementary representation. Its detailed introduction is as follows:

[0097] 1. Intra-modal aggregation

[0098] Assuming that an object has K modes, we first explore the complementarity between the components of each single modality. Multi-head self-attention is used to generate the complementary information representation of each object in the single modality. First, given a single modality input representation as where k∈k, then x k Input the multi-head self-attention layer for intra-modal interaction, and introduce jump connections to generate intra-modal complementary representations. Then, the complementary representations are accelerated to converge through layer normalization LN, and the linear layer Linear is used for dimensional transformation to obtain the output where d h is the hidden feature dimension of the network. The whole process can be expressed as follows:

[0099]

[0100] Among them, MH_SelfAttention represents the multi-head self-attention layer.

[0101] 2. Differentiated Interaction Aggregation

[0102] The main function of differential interaction aggregation is to integrate modal differences into modal information, such as Figure 6 The modal information here refers to the single modal information x k Interactive aggregation of modal differences and cross-modal information is to provide cross-modal information raw materials for the next step of cross-modal aggregation. First, construct cross-modal information:

[0103]

[0104] Then Obtain cross-modal information fusion representation through universal fusion layer Then The interactive aggregation embedding feature representation is obtained through bilinear mapping and difference interaction as follows:

[0105]

[0106] in, It is the parameter of the first layer linear mapping, and its main function is to generate cross-modal embedding. It is a representation of modal difference, where the difference is interactively fused with cross-modal embedding through tensor fusion. is the parameter of the second-layer linear mapping, which mainly converts the dimension of the interactive aggregation feature into the hidden feature dimension d h , so as to be suitable for the subsequent cross-modal aggregation process. Finally, the interactive aggregation features are passed through the batch normalization layer, the hyperbolic tangent activation function and the discard layer to obtain the final differential interactive aggregation features. It will serve as the cross-modal information raw material in the next step of cross-modal aggregation.

[0107] 3. Cross-modal aggregation

[0108] So far, we have obtained the unimodal complementarity representation and the differential cross-modal interaction representation. At this point, there is still a limitation for the entire modal complementarity, that is, the unimodal complementarity representation only considers the complementarity between objects within the unimodality, while ignoring the rich complementary representation between different objects in different modalities. Therefore, the unimodal complementarity representation is aggregated with the differentially embedded cross-modal interaction representation, which not only considers the guiding information brought by the difference to the complementarity, but also realizes the aggregation of complementarity between different objects in different modalities.

[0109] A multi-head cross-attention approach is used to obtain cross-modal aggregate representation. Acts as a decoder input mapping to a query vector, a unimodal complementarity representation Acts as a mapping of the encoding output to the key vector and the value vector. As shown below:

[0110]

[0111] in, is a learnable weight matrix, j∈H is the index of the attention head, and H is the number of attention heads. j , K j 、V j Calculate the output of the cross attention. The formula in this part is consistent with the calculation formula in the multi-head self-attention layer. Finally, a skip connection and layer normalization are used to obtain the cross-modal aggregate feature representation. What is captured is the cross-modal interaction representation Complementarity with unimodal representation The interaction relationship between any objects is guided by modal differences.

[0112] 4. Inter-modal aggregation

[0113] After the above intra-modal aggregation and cross-modal aggregation, the cross-modal complementarity obtained at this time is expressed as It contains complementary information within the modality and complementary information of different objects between different modalities guided by differences. In order to model the overall complementary information between multiple modalities, the next step is to fuse the complementary information of all modalities into the overall complementary representation of multiple modalities. Therefore, inter-modal aggregation is designed to fuse multi-modal complementary representations. First, concatenate the cross-modal aggregation representations of all modalities:

[0114]

[0115] x ′ cGenerate embedded encoding representation through linear mapping:

[0116]

[0117] in, is a learnable parameter matrix. In order to make full use of the modal differences The representation information of is again embedded into the inter-modal aggregation as the guiding information, and it is first mapped to the difference embedding vector through a linear layer:

[0118]

[0119] in, is the parameter matrix of the linear layer. Then, the cross-modal encoding representation x is aggregated by tensor fusion m and the difference embedding vector x′ diff , generating multimodal complementary attention factors as follows:

[0120]

[0121] Among them, x att Represents multimodal complementarity x m The importance of complementarity of each mode in x m After multiplication, the final multimodal overall complementary representation is obtained through linear mapping, batch normalization layer, hyperbolic tangent activation function and discard layer, as shown below:

[0122] x com = Dropout(Tanh(BN((x att ) T ·x m ·w o ))) (twenty two)

[0123] Where T is the matrix transpose, is the parameter matrix of the output linear mapping, whose main function is to transform the multimodal complementarity representation from the hidden feature dimension d h Map back to the original feature dimension d to facilitate the next step of representation fusion operation. It is the output of the entire complementarity representation module, which represents the overall complementarity representation of multimodal data. It is an overall multimodal complementarity representation obtained by exploring intra-modal complementarity, cross-modal complementarity and inter-modal complementarity through multimodal transport with difference as the guiding information.

[0124] During the training of the model, the labels are mapped to And x com With x pThe semantic similarity distance between them is used as the learning target of the complementary representation module to enhance the complementary representation ability of the module.

[0125] Furthermore, in step 106, the consistency, difference and complementarity representations of the multimodal data have been generated in a hierarchical and progressive manner, and the difference is like a link that connects the consistency and complementarity in the generation process. Next, it is necessary to adaptively fuse these three representations to generate a global semantic representation. In addition, the embodiment of the present invention also embeds the platform user information into the fusion layer as an additional prior information. First, the platform user information is encoded as a single hot vector, and then it is mapped to the vector space of the modal representation through a linear layer. The user information embedding vector can be expressed as Next, the universal fusion layer is directly used to fuse the multimodal representation information and the user information embedding vector, which can be expressed as follows:

[0126]

[0127] Among them, θ f represents the fusion layer parameters, FL represents the general fusion layer, represents the fusion layer output, which is the global semantic expression transformed by the entire CDCnet from the original multimodal data. out Output the prediction result y through the classifier i .

[0128] The classifier used in the embodiment of the present invention is a fully connected network constructed with two linear layers. i Determine the user's emotional state and promote the optimization of the platform's user experience based on the user's emotional state.

[0129] Furthermore, in step 107, for a semantic feature vector, its composition should include two aspects of semantics: spatial distance and emotional expression. Spatial distance refers to the absolute size of the semantic vector, which is a scalar value similar to the length of a line segment. Emotional expression is the vectorized feature of the semantic vector, which is an abstract representation with direction, and different directions represent different meanings.

[0130] The embodiment of the present invention starts with the vector size and direction angle in the coordinate system to construct a similarity measure between semantic feature vectors. Assume that there are two semantic feature vectors with the same feature dimension size in the vector coordinate system. i and p i , i represents the index number of different vectors, and l i The vector is the reference vector. First, calculate the angle between the two vectors and replace it with the cosine value, as shown below:

[0131]

[0132] where · represents the vector dot product and ||·|| represents the vector norm. Next, calculate p i With l i The relative spatial distance between them is as follows:

[0133]

[0134] Among them, the relative spatial distance r di ∈(-∞,+∞), the entire vector space is divided into three regions according to the relative spatial distance, and the semantic similarity of the two vectors is calculated in these three regions respectively, which can be expressed as follows:

[0135]

[0136] Among them, d si ∈[0,+∞), d si The smaller the value of , the higher the semantic similarity between the two vectors. si The larger the value of , the lower the semantic similarity of the two vectors. The three intervals in the formula are divided according to two principles. First, according to 0≤r di and r di <0 divides the interval into the same-direction zone (β1 and β2 zones) and the opposite-direction zone (β3 zone). The value of β3 should be greater than the values ​​of β1 and β2 to increase the penalty between opposite semantic vectors. Then divide the same-direction zone into the spatial distance saturation zone (β1 zone) and the spatial distance shortage zone (β2 zone). The value of β2 should be greater than the value of β1 to reduce the spatial distance between semantic vectors in the spatial distance shortage zone. Set β2 = 1, β1∈[0.5,1.0], β3∈[1.0,1.5].

[0137] According to formula (26), the embodiment of the present invention represents the diagonal distance ε between two semantic feature vectors as the tone representation difference, and the vector size difference γ as the spatial distance difference. The sum of the two represents the semantic similarity d between the two semantic feature vectors. si .

[0138] Furthermore, in step 108, the loss function of the modality joint codec includes: the reconstruction loss of the decoder and the KL divergence. The embodiment of the present invention adopts the semantic similarity metric as the reconstruction loss of the decoder. First, d si A nonlinear transformation is performed to obtain the semantic similarity measurement loss function as follows:

[0139]

[0140] The hyperbolic tangent function is used to smooth the output features, and the logarithmic function is combined to reduce the instability in numerical calculations. The reconstruction loss of the decoder can be expressed as follows:

[0141]

[0142] Among them, KL divergence is used to measure the difference between the encoder's posterior distribution and the prior distribution. The assumed prior distribution is a standard normal distribution. Then the calculation of KL divergence can be expressed as:

[0143]

[0144] Where D represents the feature dimension, μ and σ are the mean and variance of the encoder output.

[0145] For the complementary representation module, the semantic similarity distance is used as its learning target, and the complementary representation loss can be expressed as:

[0146]

[0147] in, is a representation of modal complementarity, is a tag embedding.

[0148] For the loss function of the model training output result, the embodiment of the present invention adopts the cross entropy loss function to optimize the entire framework. Finally, the loss function of the entire model training can be expressed as follows:

[0149]

[0150] Among them, λ is the weight of reconstruction loss and complementary representation loss, η is the weight of L2 regularization, and W represents the parameters of the entire model. Except for the modality encoding module, the rest of the entire CDCnet adopts an end-to-end training method, and the model parameters are updated through the above multi-task loss.

[0151] Example 3

[0152] The theoretical discussion of the complete technical solution of the embodiment of the present invention has been described in detail in the previous text, and will not be repeated here. Next, the implementation process of the emotion recognition framework proposed by the present invention will be described in detail.

[0153] First, we introduce the datasets used in the experiment. The present invention uses two public multimodal emotion recognition datasets to verify the performance of CDCnet. These two datasets are:

[0154] IEMOCAP: This dataset contains 151 binary dialogues with 10 speakers and 7433 utterances, which are labeled as one of 6 emotion categories: happy, sad, neutral, angry, excited, or frustrated. It contains three modalities: video, text, and speech. In the modality encoding module, the feature embedding dimensions of the three modalities are 342, 1024, and 1582, respectively. In the experiment, 120 dialogues and 5810 utterances are used for training and validation, and the rest are used for testing.

[0155] MELD: This dataset is a multi-party emotional dialogue dataset collected from a video. MELD contains 1433 rounds of dialogue and 13708 utterances. Each utterance is annotated with one of seven emotion labels: anger, disgust, sadness, joy, surprise, fear, or neutral. It contains three modalities: speech, video, and text. In the modality encoding module, the feature embedding dimensions of the three modalities are 342, 1024, and 300, respectively. In the experiment, 1039 dialogues and 9989 utterances were used for training, 114 dialogues and 1109 utterances were used for verification, and the rest were used for testing.

[0156] For evaluation indicators, the embodiment of the present invention follows previous research work and uses classification accuracy (Acc) and weighted F1 score to evaluate the performance of the model. In addition, the parameter size of each method is also recorded to evaluate the computational complexity and resource consumption of each method.

[0157] CDCnet was developed in the environment of CUDA12.1 and Torch 2.2.1, and the GPU used for training was a single NVIDIA Tesla A40. CDCnet was trained using the AdamW optimizer. All hyperparameter values ​​of the model were the best hyperparameters obtained by searching 50 times with the Bayesian optimization algorithm. Please refer to Table 1 for detailed hyperparameter settings. The experiment also set early stopping with a patience of 10 to obtain the best generalization performance.

[0158] Table 1. Details of hyperparameter settings of CDCnet

[0159]

[0160]

[0161] Table 2 shows the experimental results of CDCnet on the IEMOCAP and MELD datasets and the comparison results with other excellent methods. According to the results in the table, CDCnet achieved the best performance on the two datasets. The classification accuracy on the two datasets reached 71.41% and 68.93% respectively, which is 1% higher than the results of the best baseline method. This proves the effectiveness of CDCnet in emotion recognition and provides technical guarantee for the network platform to accurately identify the emotional changes of platform users. In addition, the number of parameters of CDCnet is at the downstream level, which means that the computational complexity of CDCnet is low and CDCnet can be run smoothly with lower hardware resources. This provides the necessary conditions for the simple deployment of the network platform to identify the user's emotional function and the real-time identification of the user's emotional state.

[0162] Table 2 Experimental results of CDCnet on IEMOCAP and MELD datasets and baseline comparison results. The best result is bolded and the second best result is underlined.

[0163]

[0164] Figure 8 The spatial embedding visualization of the modal representation vectors and output features obtained by CDCnet on the IEMOCAP (first row) and MELD (second row) datasets is shown. The embodiment of the present invention uses t-SNE to embed the multimodal representation and output features learned by CDCnet into the visualization space. The results shown in the figure are consistent with expectations. CDCnet obtains multimodal consistency, difference and complementarity representations through a progressive generation method, and difference serves as a link between consistency and complementarity. As can be seen from the figure, the effective representation of multimodality starts from modal consistency, gradually flows to modal complementarity through modal difference before filtering and modal difference after filtering, and finally the three are fused to obtain the output features. The feature space of the whole process gradually shows that the intra-class distribution is more compact and the inter-class distribution is more dispersed. This also proves the consistency of the experimental results with the proposed theory.

[0165] In addition, the embodiment of the present invention also explores the influence of different distance measurement methods on semantic similarity to prove the effectiveness of the semantic similarity measurement method proposed in the embodiment of the present invention. Fig. 9As shown. The abbreviations of different distance measurement methods in the figure are: NO: no distance measurement is used; CS: cosine similarity; MSE: mean square error; ED: Euclidean distance; SS: semantic similarity measurement. The first row is the experimental results on the IEMOCAP dataset, and the second row is the experimental results on the MELD dataset. According to the results in the figure, the semantic similarity measurement designed in the embodiment of the present invention has achieved the most advanced performance. Whether it is the semantic distance between the modal reconstruction features and the original modal features, or the semantic distance between the modal complementarity and the label embedding, the semantic similarity measurement generates the lowest semantic distance value. This proves the effectiveness of the semantic similarity measurement in narrowing the semantic distance between two vectors, and also provides technical support for accurately identifying the emotional state of platform users.

[0166] In summary, the embodiment of the present invention proposes a network framework (CDCnet) based on multimodal representation fusion to identify the emotional state of network platform users. It obtains modal consistency, modal difference and modal complementarity through a hierarchical and progressive generation method, and modal difference serves as a link between modal consistency and modal complementarity. The experimental results show that CDCnet is superior to other excellent baseline methods in emotion recognition, which provides technical support for accurately identifying the emotional state of platform users. Finally, the network platform needs to promote the optimization of the platform user experience in real time according to the user's emotional state, and make more targeted marketing decisions.

[0167] Example 4

[0168] A network platform user emotion recognition device based on multimodal representation fusion, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute the following method steps in Example 1:

[0169] The high-level semantic representation of a single modality obtains modality consistency representation and modality difference representation through consistency and difference representation modules; the difference filter filters out negative and redundant noise information in the modality difference representation through gating;

[0170] The complementary representation module uses filtered modal differences as guidance information to generate multimodal complementary representations through multimodal transport and attention mechanisms; embeds platform user information into the fusion layer as additional prior information; and uses a general fusion layer structure to adaptively fuse modal consistency representation, modal difference representation, modal complementary representation, and user information to generate a global semantic representation;

[0171] Finally, the global semantic representation is outputted through a classifier to represent the emotional state of platform users, and the network platform promotes the optimization of platform user experience based on the emotional state of users.

[0172] Among them, the device also includes: constructing a semantic similarity measurement function through the spatial distance and sentiment representation between feature vectors; constructing the semantic similarity measurement function as a loss function, penalizing the parameters of the modal joint encoder-decoder and the complementary representation module, and enhancing the consistency of the model and the complementary representation learning ability; the semantic similarity loss, KL divergence and classifier loss will jointly update the model parameters.

[0173] Among them, the high-level semantic representation of a single modality obtains modal consistency representation and modal difference representation through consistency and difference representation modules, specifically:

[0174] The single modal features are encoded x through the modality joint encoder and the universal fusion layer m Mapping to the distribution of latent space:

[0175]

[0176] in, is x m Through the output of the modality joint encoder and the universal fusion layer, μ φ (x′) and is the mean and log variance of the latent variable:

[0177] μ φ (x′)=x′

[0178]

[0179] from Sampling latent variables from a distribution

[0180]

[0181] Where exp(·) represents the natural exponential function, ∈ is the standard normal distribution. Gaussian noise, I is the identity matrix, and the latent variable z is mapped back to the original data space through the modal joint decoder to generate the modal reconstruction feature variable:

[0182]

[0183] in, is the reconstructed feature variable of each single mode; the modal reconstructed feature x r_m With the original modal feature x m Subtracting them, the remaining part is the feature representation of each modality that is specific to the other modalities:

[0184] x d_m =x m -x r_m ,m∈{v,l,a}

[0185] in, Represents a unimodal specific modality representation, and all unimodal specific representations x d_m Obtain preliminary modality difference representation through universal fusion layer The consistency and difference representation module generates the modal consistency representation x in a hierarchical and progressive manner. con and preliminary modal difference characterization

[0186] Among them, the general fusion layer is applicable to the parts of the whole framework that need feature fusion. The general fusion layer fuses all input features to obtain fused feature representation by constructing feature channel attention. Assuming that the input feature vector Construct a new feature dimension for each feature vector, namely: Concatenate all feature dimensions into a single feature vector:

[0187]

[0188] Among them, Concat represents dimension concatenation, followed by the fusion layer parameters Multiply to get the fused attention factor:

[0189]

[0190] in, is the fusion attention factor, softmax is the activation function, and the input feature with att f Multiply them, and then pass through the batch normalization layer BN, the hyperbolic tangent activation function Tanh and the discard layer Dropout to get the fused output:

[0191]

[0192] Where T represents transpose, Represents the fusion output result.

[0193] Example 5

[0194] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0195] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0196] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0197] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0198] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0199] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0200] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying user emotions on a network platform based on multimodal representation fusion, characterized in that: The method comprises: The high-level semantic representation of a single modality obtains modality consistency representation and modality difference representation through consistency and difference representation modules; the difference filter filters out negative and redundant noise information in the modality difference representation through gating; The complementary representation module uses filtered modal differences as guidance information to generate multimodal complementary representations through multimodal transport and attention mechanisms; embeds platform user information into the fusion layer as additional prior information; and uses a general fusion layer structure to adaptively fuse modal consistency representation, modal difference representation, modal complementary representation, and user information to generate a global semantic representation; Finally, the global semantic representation is outputted through a classifier to represent the emotional state of platform users, and the network platform promotes the optimization of platform user experience based on the emotional state of users.

2. According to claim 1, a method for identifying user emotions on a network platform based on multimodal representation fusion is characterized in that: The method also includes: constructing a semantic similarity measurement function through the spatial distance between feature vectors and sentiment representation; constructing the semantic similarity measurement function as a loss function, penalizing the parameters of the modal joint encoder-decoder and the complementary representation module, and enhancing the consistency and complementary representation learning ability of the model; the semantic similarity loss, KL divergence and classifier loss will jointly update the model parameters.

3. The method for identifying user emotions on a network platform based on multimodal representation fusion according to claim 1 is characterized in that: The high-level semantic representation of the single modality obtains modality consistency representation and modality difference representation through consistency and difference representation modules, specifically: The single modal features are encoded x through the modality joint encoder and the universal fusion layer m Mapping to the distribution of latent space: in, is x m Through the output of the modality joint encoder and the universal fusion layer, μ φ (x′) and is the mean and log variance of the latent variable: m φ (x′)=x′ from Sampling latent variables from a distribution Where exp(·) represents the natural exponential function, ∈ is the standard normal distribution. Gaussian noise, I is the identity matrix, and the latent variable z is mapped back to the original data space through the modal joint decoder to generate the modal reconstruction feature variable: in, is the reconstructed feature variable of each single mode; the modal reconstructed feature x r_m With the original modal feature x m Subtracting them, the remaining part is the feature representation of each modality that is specific to the other modalities: x d_m =x m -x r_m ,m∈{v,l,a} in, Represents a unimodal specific modality representation, and all unimodal specific representations x d_m Obtain preliminary modality difference representation through universal fusion layer The consistency and difference representation module generates the modal consistency representation x in a hierarchical and progressive manner. con and preliminary modal difference characterization 4. The method for identifying user emotions on a network platform based on multimodal representation fusion according to claim 1, characterized in that: The general fusion layer is applicable to the parts of the whole framework that need feature fusion. The general fusion layer fuses all input features to obtain fused feature representation by constructing feature channel attention. Assuming that the input feature vector Construct a new feature dimension for each feature vector, namely: Concatenate all feature dimensions into a single feature vector: Among them, Concat represents dimension concatenation, followed by the fusion layer parameters Multiply to get the fused attention factor: in, is the fusion attention factor, softmax is the activation function, and the input feature with att f Multiply them, and then pass through the batch normalization layer BN, the hyperbolic tangent activation function Tanh and the discard layer Dropout to get the fused output: Where T represents transpose, Represents the fusion output result.

5. A network platform user emotion recognition device based on multimodal representation fusion, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 4.

Images