A user attribute detection method based on three-modal representation learning
By employing a trimodal representation learning method, cross-modal information fusion is achieved using one-dimensional convolution and trimodal multi-head attention (TMHA). Combined with GRU and fully connected operations, the problem of incomplete modal information interaction is solved, and efficient user attribute detection is realized.
Patent Information
- Application Number
- CN202211357140.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-01
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2042-11-01
AI Technical Summary
Existing multimodal analysis methods suffer from incomplete modal information interaction, resulting in high computational complexity and inaccurate analysis results, and are unable to effectively integrate information from multiple modalities.
A trimodal representation learning method is adopted, which maps text, visual and audio features to the same feature space through one-dimensional convolution. After adding positional encoding, trimodal multi-head attention (TMHA) is used for cross-modal interaction. Combined with GRU and fully connected operations, a joint loss function is designed for training.
It improves the accuracy of user attribute detection, overcomes the problem of incomplete modal information interaction, enhances analysis precision, and reduces computational complexity.
Smart Images
Figure CN115687620B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of user attribute detection, and particularly relates to a user attribute detection method based on three-modal representation learning. BACKGROUND
[0002] With the rise and rapid development of network social media, people publish their own opinions or comments on commodities on various platforms every day, and these opinions or comments often carry some attributes of the users themselves, such as emotions. Therefore, how to effectively judge and mine the value of the topic from the emotions of the users to analyze the related comments and make decisions is crucial.
[0003] In real life, people usually express their emotions in multiple ways, which can be text, expressions, and tone of voice, that is, multi-modal data. In the current research, there are a large number of modal fusion analysis methods, but most of the methods are computationally complex and inefficient. Therefore, how to effectively fuse the information of multiple modalities while reducing the computational complexity and improving the analysis accuracy still has strong theoretical and application value.
[0004] At present, with the continuous maturity and development of various neural network models and computing resources, deep learning methods have achieved very good results in various fields. Especially the transformer, which has achieved good accuracy in various tasks of natural language processing and computer vision. In the multi-modal analysis framework based on transformer, the traditional transformer decoder structure is usually adopted, the features of one modality are taken as query, and the features of another modality are taken as key and value, so as to complete the cross-modal interaction through attention calculation. However, one obvious deficiency of this is that with the increase of modalities, the number of transformer blocks required will increase, so that the entire network framework is not only very deep in depth due to the characteristics of the transformer itself, but also wider in width due to the characteristics of the multi-modal. In addition to the increase of computational complexity, the interaction between modalities also always has the problem of incomplete information, which cannot consider all the information between multiple modalities in one calculation, which will also affect the final analysis result. SUMMARY
[0005] The purpose of the application is to provide a user attribute detection method based on three-modal representation learning, which overcomes the problem of incomplete modal information interaction when using a transformer decoder alone and further improves the final detection performance.
[0006] In order to achieve the above object, the technical scheme of the present application is as follows:
[0007] A user attribute detection method based on three-modal representation learning, comprising:
[0008] Obtain user text, visual and audio three-modal features, and map the three-modal features to the same feature space through one-dimensional convolution respectively;
[0009] Add the obtained three-modal features to the position coding composed of the sine and cosine functions respectively, to obtain the three-modal features after position coding;
[0010] Perform self-attention processing on the text features through a single-modal transformer encoder to eliminate redundant semantic information;
[0011] Perform cross-modal interaction on the text, visual and audio features processed in the previous steps through two three-modal transformer encoders respectively, the three-modal transformer encoder uses three-modal multi-head attention TMHA to extract all three-modal information, and the text features are used as query, one of the three-modal transformer encoders uses the visual features as key and the audio features as value, and the other three-modal transformer encoder uses the audio features as key and the visual features as value;
[0012] Splice the outputs of the two three-modal transformer encoders to obtain multi-modal features, and then perform GRU and full connection operations on the multi-modal features to output user attribute prediction values.
[0013] Further, the single-modal transformer encoder has N layers, and each layer includes multi-head attention MHA and feed-forward network FFN.
[0014] Further, the user attribute detection method based on three-modal representation learning further comprises:
[0015] For the network used in the user attribute detection method based on three-modal representation learning, the following joint loss function is used for training:
[0016] L overall =L1+L AR +L TSC
[0017] Wherein, L overall represents the joint loss function, L1 represents the mean absolute error loss, L AR represents the attraction and repulsion loss, and L TSC represents the supervised contrast loss.
[0018]
[0019]
[0020] wherein y represents a user attribute label of a training sample, represents a user attribute prediction value, represents a user attribute category value rounded off by the user attribute label y, represents rounding up the user attribute prediction value , represents rounding down the user attribute prediction value .
[0021] L TSC =(Loss L +Loss V +Loss A ) / 3
[0022]
[0023]
[0024]
[0025] wherein P {L,V,A} represents a set of positive samples of each modality, |P {L,V,A} | represents the number of positive samples in each modality, S {L,V,A} represents a total set of positive and negative samples of each modality, x {l,a,v} represents a single sample of each modality, τ represents a temperature coefficient, represents a positive sample of a video, audio modality relative to text x l , represents all positive and negative samples of a video, audio modality relative to text x l , represents a positive sample of a text, audio modality relative to video x v , represents all positive and negative samples of a text, audio modality relative to video x v , represents a positive sample of a text, video modality relative to audio x a , represents all positive and negative samples of a text, video modality relative to audio x a .
[0026] The user attribute detection method based on three-modal representation learning provided in the application uses a TMHA module to obtain information of three modes at one time, and simultaneously uses seven-classification emotion fine-grained AR loss and two-classification emotion fine-grained TSC loss as auxiliary, which not only can overcome the problem of incomplete interaction of modal information when using a pure traditional transformer encoder, but also can improve the accuracy of user attribute detection. BRIEF DESCRIPTION OF DRAWINGS
[0027] Figure 1 The flow chart of the user attribute detection method based on three-modal representation learning of the application;
[0028] Figure 2 The network schematic diagram of the user attribute detection method based on three-modal representation learning of the application;
[0029] Figure 3 The structure schematic diagram of each layer of the three-modal transformer encoder of the application. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.
[0031] In one embodiment, as shown in Figure 1 A user attribute detection method based on three-modal representation learning is provided, which includes:
[0032] Step S1, obtaining user text, visual and audio three-modal features, and mapping the three-modal features to the same feature space through one-dimensional convolution respectively.
[0033] Various texts, audios and videos data published by users on the platform can obtain feature data after feature extraction. Among them, the text feature is extracted by the Glove pre-training model, the visual feature is extracted by Facet, and the audio feature is extracted by COVAREP, that is, the three-modal feature data of the user is obtained, which is represented as Where T and d {L,V,A} respectively represent the sequence length and feature dimension, L represents the text, V represents the video, and A represents the audio. The three-modal feature data has been aligned at the word level on the time step. The time step is the sequence length T. For the text L, the sequence length is the number of words; for the video V, the sequence length is the number of frames; for the audio A, the sequence length is the number of sampling points.
[0034] Then the three-modal features are mapped to the same feature space through one-dimensional convolution network respectively, as shown in the following formula:
[0035]
[0036] where k {L,V,A} is the kernel size used when performing one-dimensional convolution operation on the three modal features, and d represents the common feature dimension held by the three modal features after one-dimensional convolution.
[0037] Step S2, respectively add the obtained three modal features to the position encoding composed of the sine and cosine functions to obtain the three modal features after position encoding.
[0038] The present application adds position encoding, so that the Transformer can perceive sequence information, as shown in the following formula:
[0039]
[0040] where PE(T, d) represents the position encoding function, which calculates the position index of each time step in the multi-modal feature. Here, the sine and cosine position encoding in the traditional transformer is followed, as shown in the following formula:
[0041]
[0042] PE (pos,2i+1) = cos(pos / 10000 2i / d )
[0043] where pos = 1,..., T,
[0044] Step S3, the text feature is processed by the single-modal transformer encoder to do self-attention, and the redundant semantic information is removed.
[0045] The single-modal transformer encoder in this step has N layers, each layer includes multi-head attention MHA and feed-forward network FFN, which belongs to the traditional transformer encoder structure.
[0046] Definition Thus, for the i-th (i = 1,..., N) layer, the following calculation is performed:
[0047]
[0048]
[0049] where MHA denotes Multi-Head Attention, FFN denotes Feed-Forward Network, and LN denotes Layer Normalization.
[0050] The calculation of MHA is shown in the following formula:
[0051] MHA(Q, K, V) = Concat(head1,..., head h )W O
[0052] head i = Attention(QW i Q , KW i K , VW i V )
[0053]
[0054] where W O , W i Q , W i K , W i V all denote weight matrices, and h denotes the number of attention heads. The calculation of FFN is shown in the following formula:
[0055] FFN(x) = max(0, xW1 + b1)W2 + b2
[0056] where W1 and W2 both denote weight matrices, and b1 and b2 denote added offsets.
[0057] The calculation of LN is shown in the following formula:
[0058]
[0059] where E[x] denotes the expectation of x, and Var[x] denotes the variance of x.
[0060] Step S4, the text, visual and audio features processed in the previous steps are respectively passed through two three-modal transformer encoders for cross-modal interaction, the three-modal transformer encoders use three-modal multi-head attention TMHA to extract information of all three modalities, and the text feature is used as the query, one three-modal transformer encoder uses the visual feature as the key and the audio feature as the value, and the other three-modal transformer encoder uses the audio feature as the key and the visual feature as the value.
[0061] As shown in Figure 2 , the left three-modal transformer encoder uses the visual feature as the key and the audio feature as the value, and the right three-modal transformer encoder uses the audio feature as the key and the visual feature as the value. In the three-modal transformer encoder, the text feature is used as the query, and TMHA is used to extract information of all three modalities.
[0062] Unlike the conventional transformer that can only use the self-attention mechanism of MHA to extract single-modal information, the three-modal transformer can effectively aggregate the information of the three modalities through the attention mechanism of TMHA.
[0063] Thus, for the jth(j = 1,...,D) layer of the three-modal transformer, as shown in Figure 3 , the following calculation formula is used:
[0064]
[0065]
[0066] where TMHA represents three-modal multi-head attention, and the subscripts L, V, and A represent that the visual feature is used as the key and the audio feature is used as the value in the calculation of TMHA, and the subscripts L, A, and V represent that the audio feature is used as the key and the visual feature is used as the value in the calculation of TMHA. For each attention head m, the similarity between the three modalities is calculated using the Einstein summation convention:
[0067] C i,j,k =(Q m ) in (K m ) jn (V m ) kn
[0068]
[0069] where, are the input query, key, value in TMHA, is the weight matrix, n = 1, …, d head is a dummy index. The calculated similarity cube C i,j,k denotes the element value at (i, j, k) in C. d head denotes the feature dimension of each attention head, which is calculated by d head = d / h.
[0070] Thus, the calculation formula of TMHA can be obtained:
[0071] TMHA(Q, K, V) = Concat(head1, …, head h W O
[0072]
[0073] where, denotes the weight matrix. Here, in order to complete the entire attention calculation process, the last dimension of C is selected to be reduced by average and max pooling.
[0074] Step S5, the outputs of the two three-modal transformer encoders are spliced to obtain multi-modal features, and then the multi-modal features are subjected to GRU and full connection operations to obtain the predicted value of the output.
[0075] After the multi-modal interaction is completed in step S4, and are spliced to obtain multi-modal features, as shown in the following formula:
[0076]
[0077] Then, h is subjected to a layer of GRU, which is a variant of RNN, and the output of each time step is determined by the input of the current time step and the output of the last time step, which is used to process time series data. Due to the characteristics of RNN, the last time step covers the information of the entire time series data, so that the time series data with a length of T can be compressed to 1, and the feature of the last time step is taken as the output of the entire network, as shown in the following formula:
[0078]
[0079] Finally, is obtained through a full connection layer to obtain the user attribute prediction value, as shown in the following formula:
[0080]
[0081] In one specific embodiment, the present application designs a joint loss function to train the network, as shown in the following formula:
[0082] L overall = L1+ L AR + L TSC
[0083] Wherein, L1 represents the mean absolute error loss, L AR represents the attraction and repulsion loss, and L TSC represents the supervised contrast loss. Among them, L1 is a traditional loss function, and L AR and L TSC are unique loss functions adopted by the present application.
[0084] L1 and L AR loss, as shown in the following formula:
[0085]
[0086]
[0087] Wherein, α represents the offset, mainly used to control when is greater than the emotional category , it can be ensured that is in , the loss function is still not 0, so as to continue to push x to one side. Wherein, y represents the user attribute label, represents the user attribute prediction value, represents the user attribute category value obtained by rounding the user attribute label y, represents rounding up the user attribute prediction value , represents rounding down the user attribute prediction value .
[0088] In order to enable the same emotional category samples in the data set to attract each other, and the different emotional category samples to repel each other, the present embodiment further calculates the attention of the text, visual and audio features, converges the features of each sample at each time step, and calculates the supervised contrast loss L TSC .
[0089] For attention calculation, first calculate the attention weight of the feature of each modality along its time step, as shown in the following formula:
[0090]
[0091] w {v,a} = softmax(tanh(Z {V,A} W {v,a} + b {v,a} ))
[0092] where and denote the weight matrix and bias, respectively.
[0093] Then, the sample features of each modality are summed up at time step t to obtain the sample features of each modality:
[0094]
[0095]
[0096] where B denotes the number of samples, and d denotes the feature dimension.
[0097] For the sample features of the three modalities, their similarities are also calculated using the Einstein summation convention, as shown in the following formula:
[0098] c i,j,k = (c l ) in (c v ) jn (c a ) kn
[0099] where n = 1, …, d is a dummy index, and c i,j,k denotes the element value of the coordinate (i, j, k) in the similarity cube c. Thus, for a single sample and , there are:
[0100]
[0101] After obtaining the similarities between the three modalities, the TSC loss is further calculated to complete the three-modality contrast learning, as shown in the following formula:
[0102]
[0103]
[0104]
[0105] L TSC = (Loss L + Loss V + Loss A ) / 3
[0106] Among them, P {L,V,A} Let |P| represent the set of positive samples for each modality. {L,V,A} | represents the number of positive samples in each modality set, S {L,V,A} This represents the total set of positive and negative samples for each modality. This represents a single sample for each mode, where τ represents the temperature coefficient. Loss... L For example, Represents relative to text x l Positive samples of video and audio modalities (i.e., those with x) l x with the same user attribute category v x a sample), Represents relative to text x l In terms of all positive and negative samples of the video and audio modalities (i.e., regardless of x) v x a Regardless of the user attribute category, it is related to x l Calculate similarity.
[0107] Similarly, Represents relative to video x v Positive samples of text and audio modalities. Represents relative to video x v In terms of text and audio modalities, all positive and negative samples Represents relative to audio x a Positive samples in text and video modalities. Represents relative to audio x a This refers to all positive and negative samples of the text and video modalities. `sim` represents the similarity function, and `exp` represents the exponential function with base e in advanced mathematics. `{L,V,A}` represents one of the three modalities.
[0108] In this application, the technical solution was also experimentally verified. The experiments were conducted using the CMU-MOSI and CMU-MOSEI datasets, and compared with current mainstream multimodal sentiment analysis algorithms. Performance was measured using seven-class classification accuracy (Acc7), binary classification accuracy (Acc2), binary classification F1 score (F1), mean absolute error (MAE), and Pearson product-moment correlation coefficient (Corr). Except for MAE, which is considered better the lower it is, all other metrics were considered better the higher they were.
[0109] The experimental results are shown in Tables 1 and 2:
[0110]
[0111] Table 1
[0112]
[0113]
[0114] Table 2
[0115] Table 1 and Table 2 are respectively the test results on the CMU-MOSI dataset and the CMU-MOSEI dataset, and it can be known from the comparison of the two tables that the method of the present application is superior to other multi-modal sentiment analysis algorithms in 2017, 2018, 2019 and 2021 in terms of multi-modal sentiment analysis performance, and the Acc7, Acc2, F1 score, MAE and Corr indexes are all optimal, so the method of the present application has certain superiority compared with other algorithms.
[0116] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.
Claims
1.A user attribute detection method based on three-modal representation learning, characterized in that, The user attribute detection method based on three-modal representation learning comprises the following steps: Obtaining user text, visual and audio three-modal features, and mapping the three-modal features to the same feature space through one-dimensional convolution respectively; Adding the obtained three-modal features to the position coding composed of the sine and cosine functions respectively to obtain the three-modal features after position coding; Performing self-attention processing on the text features through a single-modal transformer encoder to eliminate redundant semantic information; Performing cross-modal interaction on the text, visual and audio features processed in the previous step through two three-modal transformer encoders respectively, wherein the three-modal transformer encoder uses three-modal multi-head attention TMHA to extract all three-modal information, and the text features are used as query, one of the three-modal transformer encoders uses the visual features as key and the audio features as value, and the other three-modal transformer encoder uses the audio features as key and the visual features as value; Splicing the outputs of the two three-modal transformer encoders to obtain multi-modal features, and then performing GRU and full connection operations on the multi-modal features to output user attribute prediction values. The user attribute detection method based on three-modal representation learning further comprises the following steps: The network used in the user attribute detection method based on three-modal representation learning is trained by using the following joint loss function: ； wherein, identifying a joint loss function, denotes a mean absolute error loss, denotes an attraction-repulsion loss, denotes a supervised contrastive loss; ； ； wherein, represents a user attribute label of a training sample, represents a user attribute prediction value, represents a user attribute class value rounded up from the user attribute label by the user attribute prediction value, represents the user attribute prediction value rounded up, represents the user attribute prediction value rounded down, represents an offset amount; ； ； ； ； in, This represents the set of positive samples for each modality. This represents the number of positive samples in each modality set. This represents the total set of positive and negative samples for each modality. This represents a single sample for each modality. Indicates the temperature coefficient. Represents relative to text Positive samples of video and audio modalities. Represents relative to text In terms of all positive and negative samples of the video and audio modalities, Indicates relative to video Positive samples of text and audio modalities. Indicates relative to video In terms of text and audio modalities, all positive and negative samples Relative to audio Positive samples in text and video modalities. Relative to audio In other words, all positive and negative samples of text and video modalities. 2.The user attribute detection method based on three-modal representation learning according to claim 1, characterized in that, The single-modal transformer encoder has N layers, and each layer comprises multi-head attention MHA and feed-forward network FFN.