A multimodal advertising popularity prediction method based on text-supervised attention

Abstract

This invention discloses a method for predicting the popularity of multimodal advertisements based on text-supervised attention. The method comprises the following steps: 1. constructing a combined representation of a multimodal advertisement dataset; 2. extracting features of the multimodal advertisements; 3. introducing a text-based attention mechanism; 4. fusing the textual and visual features of the multimodal advertisements; and 5. using the fused global features to predict the advertisement's popularity score. This method comprehensively utilizes the textual and visual information in multimodal advertisements when predicting popularity scores, effectively reducing the impact of noise information to obtain an effective data feature representation, thereby improving the prediction accuracy of multimodal advertisements.

Description

Technical Field

[0001] The present invention relates to the field of multimodal data prediction, and in particular to a multimodal advertisement popularity prediction method based on text attention. Background Art

[0002] In the Web 2.0 era, online social media (such as Twitter and Facebook) have greatly facilitated our daily lives. Every day, a large amount of text and visual content is generated and disseminated on social media platforms. An interesting finding is that some content can receive millions of clicks, likes, shares, and comments, while some content has only a small amount of attention and dissemination. This discovery has prompted researchers to analyze the popularity of these contents on social media. The task of popularity prediction is to estimate the number of interactions between users and a given content (such as the number of views or likes), or to identify potential popular content in advance from a large amount of information. Currently, accurate popularity prediction is increasingly valued by academia and industry due to its wide range of applications, such as advertiser decision-making, content recommendation, and public opinion monitoring.

[0003] Recently, predicting the popularity of social media content has become a focus of a range of research. The core idea is to leverage various features or time series patterns to build regression or classification models for popularity prediction, primarily using textual and visual features. The paper [Image popularity prediction in social media using sentiment and context features. in Proceedings of the 23rd ACM international conference on Multimedia. 2015] combines visual sentiment features with contextual features (such as tags and titles) to predict image popularity. The paper [A multiple feature fusion approach with deep neural networks. in Proceedings of the 27th ACM International Conference on Multimedia. 2019] uses deep neural networks to learn high-level representations from images and text, then feeds these features into a regression model to predict the popularity of posts collected from Flickr. While these methods combine visual and textual features, they only consider the influence of a single image corresponding to textual content (such as a post). Furthermore, by neglecting irrelevant and unimportant parts of the image or text, these methods introduce noise, leading to prediction bias. Therefore, how to effectively use the text and image information of content generated by social media to predict the popularity of the content is a problem worth studying. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention proposes a multimodal advertising popularity prediction method based on text supervision, so as to comprehensively utilize text and visual information and other auxiliary information and effectively reduce noise when processing the popularity prediction task of multimodal data, so as to quickly and accurately learn data feature representation, thereby improving the accuracy and precision of popularity prediction.

[0005] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0006] The present invention is characterized by a multimodal advertising popularity prediction method based on text-supervised attention, which is performed in the following steps:

[0007] Step 1: Construct a multimodal advertising data set D, where |D| represents the number of multimodal advertising data sets D. Any i-th advertisement in the multimodal advertising data set D contains multimodal information such as text description, image, label, title, author, and time, i∈[1,|D|];

[0008] Step 1.1: Construct a text information set of the multimodal advertisement data set D. The text information of the i-th advertisement in the text information set is recorded as Among them, w i,n The text information d of the i-th advertisement i The nth word in N i Indicates text information d i The total vocabulary size, n∈[1,N i ];

[0009] Step 1.2: Construct a visual information set of the multimodal advertisement data set D, wherein the visual information I of the i-th advertisement in the visual information set is i , recorded as Among them, p i,m Represents the visual information I of the i-th advertisement i The mth picture in M i Represents visual information I i The total number of pictures, m∈[1,M i ];

[0010] Step 1.3: Construct other auxiliary information sets of the multimodal advertising data set D, where the label information set of the i-th advertisement is recorded as l i , the title information set of the i-th advertisement is recorded as t i , the author information set of the i-th advertisement is recorded as a i , the time information set of the i-th advertisement is recorded as T i ;

[0011] Step 2: Extract the features of multimodal advertisements;

[0012] Step 2.1, obtain feature representation of text information based on denoising topic model;

[0013] Step 2.1.1, define the topic distribution of all text information sets as θ = {θ1, θ2, ..., θ i ,…,θ D}, where θ i The text information d of the i-th advertisement i The topic distribution of , and obeys the Dirichlet distribution with parameter α, defining the text information d of the i-th advertisement i The topic distribution θ i K topics related to popularity prediction {θ i1,θ i2 ,…,θ ik ,…,θ iK} and a background theme, where θ ik The text information d of the i-th advertisement i The probability distribution value on the kth topic related to popularity prediction, k∈[1,K];

[0014] Step 2.1.2, let φ k represents the kth topic word distribution, and φ k It follows a Dirichlet distribution with parameter β0 and has φ k ={φ k1 ,φ k2 ,…φ kv ,…φ kV}, where φ kv represents the probability that the vth word is assigned to the kth topic, v∈[1,V], where V represents the total number of non-repeated words in the text information of all advertisements;

[0015] make represents the word distribution under the background topic, and It obeys the Dirichlet distribution with parameter β1 and has in, represents the probability that the vth word is assigned to the background topic;

[0016] Step 2.1.3: Define binary indicator variable b = (b1, b2, ..., b i ,...,b D ), where b i Indicates the text information d of the i-th advertisement i The subject of all words in Among them, b in The text information d of the i-th advertisement i The nth word w in i,n The subject belongs to the situation and obeys the Bernoulli distribution;

[0017] If b in =1, indicating the text information d of the i-th advertisement i The nth word w in i,n Belonging to the topic related to popularity prediction, the text information d of the i-th advertisement is defined as i The topic number associated with popularity prediction for the nth word in is z in , and the obedience parameter is θ i Multinomial distribution; define the text information d of the i-th advertisement i The nth word w i,n The obedience parameter is A multinomial distribution where The text information d of the i-th advertisement i The nth word w i,n Belongs to the zth related to popularity prediction in Probability distribution of words under a topic;

[0018] If b in =0, indicating the text information d of the i-th advertisement i The nth word w i,n Belongs to the background theme; defines the text information d of the i-th advertisement i The nth word w i,n The obedience parameter is Multinomial distribution of ;

[0019] Step 2.1.4: Use the collapsed Gibbs sampling method shown in formulas (1) to (3) to calculate the text information d of the i-th advertisement. i The topic distribution θ i , word distribution φ under the kth topic related to popularity prediction k And word distribution under background topics

[0020]

[0021]

[0022]

[0023] In formulas (1) to (3), The text information d of the i-th advertisement i The number of words assigned to the kth topic related to popularity prediction; It represents the number of v-th words in all text information assigned to the k-th topic related to popularity prediction when the binary variable b is 1; It represents the number of v-th words in all text information assigned to the background topic when the binary variable b is 0;

[0024] According to the word distribution φ under the kth topic related to popularity prediction k , take the top E words with the largest probability to vectorize the kth topic related to popularity prediction, and combine it with the text information d of the i-th advertisement i The topic distribution θ i , get the feature matrix of text information in, The text information d of the i-th advertisement i Feature representation of the kth topic related to popularity prediction in ;

[0025] Step 2.2: Extract image region features using the pre-trained VGG-16 network.

[0026] The visual information of the i-th advertisement I i The mth picture p i,m Divide into Z regions and get a set of image regions in, Represents the visual information I of the i-th advertisement i The mth picture p i,m The zth region in Input VGG-16 network and output the mth picture p i,m The zth region The feature representation v i,m,z , thus obtaining the feature representation set V of the image area i,m =[v i,m,1 ,…,v i,m,z ,…,v i,m,Z ];

[0027] Step 2.3, obtain feature representation of other auxiliary information;

[0028] Step 2.3.1, obtain the feature representation of label and title information;

[0029] The label information l of the i-th advertisement i The jth label l i,j Represented as one-hot encoding y i,j , the label information l of the i-th advertisement i All labels are represented as corresponding one-hot encodings and constitute encoding vectors, which are then input into the LSTM model to obtain label information l i The eigenvector of

[0030] By tag information i The feature vector constructs label information l i The label feature matrix of the i-th advertisement is obtained by performing the mean pooling operation. i The feature representation f i label Similarly, get the title information of the i-th advertisement t i The feature representation f i title ;

[0031] Step 2.3.2, obtain the feature representation of author information;

[0032] Use author information collection a i Number of fans f i followers , author information set ai Number of followers i followings , author information set a i Number of works f i works As the author information set a of the i-th advertisement i The feature representation is denoted as f i author =[f i followers ,f i followings ,f i works ];

[0033] Step 2.3.3, obtain feature representation of time information;

[0034] The time information of the i-th advertisement is set T i Hour f i hour 、Sunday i week 、Japanese i day 、Moon i month 、yearf i year As the time information T of the i-th advertisement i The characteristic representation is denoted as f i time =[f i hour ,f i week ,f i day ,f i month ,f i year ];

[0035] Step 3: Introduce text-based attention mechanism;

[0036] Step 3.1: Introduce the attention mechanism shown in Equations (4) and (5), and use Equation (6) to obtain the text information d of the i-th advertisement i Feature representation Q is the text feature dimension:

[0037]

[0038]

[0039]

[0040] In formulas (4) to (6), It is a word-level text feature representation The hidden representation, S is the hidden layer dimension, W topic Implicit Representation The weight matrix, b topic Implicit representation The bias vector, α i,k ∈R is the text feature representation The attention score, W a Score for text attention The weight matrix, tanh represents the tangent activation function, Representation of text features The implicit representation of , k′∈[1,K];

[0041] Step 3.2: Obtain feature representation of each image through the text-based attention mechanism;

[0042] Using formula (7) and formula (8) to obtain the image region feature representation combined with text information and the zth region The attention score α i,m,z ∈R, and then use formula (9) to get the mth picture p i,m Feature representation

[0043]

[0044]

[0045]

[0046] In formulas (7) to (9), W region Image region feature representation The weight matrix, W text Text feature representation The weight matrix, W v Attention score for image region The weight matrix of Represents the visual information I of the i-th advertisement i The mth picture p i,m The feature representation of the z′th region in , z′∈[1,Z], ⊙ represents the element product;

[0047] Step 3.3: Use the attention mechanism to fuse the features of multiple images and finally obtain the visual information I of the i-th advertisement. i Feature representation

[0048] Using formula (10) to get the visual information I of the i-th advertisement i The mth picture p i,m The latent feature representation h i,m ∈R Q , use formula (11) to calculate the mth picture p i,m The attention score a i,m ∈R, using formula (12) to obtain the visual information I of the i-th advertisement i Feature representation

[0049]

[0050]

[0051]

[0052] In formulas (10) to (12), W image Hidden feature representation for images The weight matrix, b image Hidden feature representation for images The bias vector, W h Score image attention The weight matrix of h i,m′ Represents the visual information I of the i-th advertisement i The m′th picture p i,m′ The latent feature representation, m′∈[1,M i ];

[0053] Step 4: Fusion of text and visual features of multimodal advertisements;

[0054] The linear function l shown in formula (13) is used to transform the text features and visual features Perform fusion to obtain the fusion representation of the i-th advertisement

[0055]

[0056] In formula (13), is the outer product; [] represents the linearization of the vector matrix;

[0057] Step 5: Use the fused global features to predict the popularity score;

[0058] Step 5.1, Connect them together to get the global feature f of the i-th advertisement i global , thus obtaining the global features f of all advertisements globalAnd input the predicted popularity score into the classifier as shown in formula (14)

[0059]

[0060] In formula (14), W g and b g is the weight matrix and bias vector of the classifier;

[0061] Step 5.2: Use the binary cross entropy as the objective function J as shown in formula (15):

[0062]

[0063] In formula (15), C represents the number of data items in the multimodal data set D used as the training set, y c represents the true popularity score of the cth advertisement in the training set, represents the predicted popularity score of the c-th advertisement in the training set; 1<C<|D|;

[0064] Step 5.3. Set the maximum number of iterations epoch_number and use backpropagation and gradient descent to train the objective function J so that J reaches the minimum. When the number of iterations reaches epoch_number, stop training to obtain the optimal prediction model, which is used to predict the popularity score of the input multimodal advertising data.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] 1. This paper proposes a new text-guided attention neural network (TGANN), which integrates the text, visual, label, title, author and time features of advertisements to obtain a global feature. The global feature is used to predict the popularity of advertisements. Compared with the existing technology, it makes more effective use of the features of multimodal advertisement data and greatly reduces the noise information in the popularity prediction process, effectively improving the accuracy and efficiency of popularity prediction.

[0067] 2. The present invention proposes a new denoising topic model, which effectively reduces the noise words irrelevant to popularity prediction in text information by introducing background topics, retains the words that are valuable for popularity prediction, and thus improves the effectiveness of text information in popularity prediction.

[0068] 3. This paper introduces a text-based attention mechanism, which uses text information to supervise images to select information in the image that is helpful for popularity prediction and filter out areas in the image that are of no value for popularity prediction, greatly improving the accuracy of popularity prediction.

[0069] 4. This invention is the first to attempt to solve the problem of popularity prediction for multimodal advertisements. It fully utilizes the characteristics of heterogeneous and multimodal data in advertisements, such as text features and image features, to effectively predict the popularity of advertisements. This can help marketers evaluate the success rate of online advertisements and help marketers make better decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a topic model graph based on noise reduction in the present invention;

[0071] Figure 2 The present invention is based on the text of the attention mechanism model diagram;

[0072] Figure 3 This is a framework diagram of the multimodal advertising popularity prediction method based on text-supervised attention in the present invention. DETAILED DESCRIPTION

[0073] In this embodiment, Figure 3 As shown in the figure, a method for predicting the popularity of multimodal advertisements based on text-supervised attention is used to predict the popularity of multimodal advertisements based on a text-supervised attention neural network model. The text information and visual information in the advertisement are extracted and fused. At the same time, the label information, title information, author information and time information in the advertisement are extracted. These features are connected in series to obtain a global feature, and the popularity of the advertisement is predicted by the global feature. In this process, Figure 1 As shown in the figure, a denoising-based topic model is proposed. The background topic is introduced on the basis of the existing LDA (latent Dirichlet allocation) topic model, so that the word distribution under the background topic Obey the Dirichlet distribution with parameter β1 and introduce the binary variable b in Determine the topic of each word, thereby filtering out words in the text that are irrelevant to popularity prediction and reducing noise in the text; Figure 2 As shown, a text-based attention mechanism is introduced to obtain the feature representation of text information through the attention mechanism. Use the features of text information to supervise visual information and obtain the feature representation of visual information Reduce noise in visual information; use Multimodal Compact Bilinear (MCB) pooling to fuse text and visual features, learn accurate data feature representation, and thus provide more accurate ad popularity prediction. Specifically, the process is as follows:

[0074] Step 1: Construct a multimodal advertising data set D, where |D| represents the number of multimodal advertising data sets D. Any i-th advertisement in the multimodal advertising data set D contains multimodal information such as text description, image, label, title, author, and time, i∈[1,|D|];

[0075] Step 1.1: Construct a text information set of the multimodal advertising data set D. The text information of the i-th advertisement in the text information set is recorded as Among them, w i,n The text information d of the i-th advertisement i The nth word in N i Indicates text information d i The total vocabulary size, n∈[1,N i ];

[0076] Step 1.2: Construct a visual information set of the multimodal advertising data set D. The visual information of the i-th advertisement in the visual information set is I i , recorded as Among them, p i,m Represents the visual information I of the i-th advertisement i The mth picture in M i Represents visual information I i The total number of pictures, m∈[1,M i ];

[0077] Step 1.3: Construct other auxiliary information sets of the multimodal advertising data set D, where the label information set of the i-th advertisement is recorded as l i , the title information set of the i-th advertisement is recorded as t i , the author information set of the i-th advertisement is recorded as a i , the time information set of the i-th advertisement is recorded as T i ;

[0078] Step 2: Extract the features of multimodal advertisements;

[0079] Step 2.1, such as Figure 1 As shown, the feature representation of text information is obtained based on the denoising topic model;

[0080] Step 2.1.1. The text information of an advertisement usually contains words that reflect the theme of the advertisement and some common words in advertisements. These common words are of no value for popularity prediction and are regarded as noise information in popularity prediction. Therefore, a background topic is introduced into the denoising topic model to remove the noise in the text information. The topic distribution of all text information sets is defined as θ = {θ1, θ2, …, θ i ,…,θ D}, where θ i The text information d of the i-th advertisement i The topic distribution of , and obeys the Dirichlet distribution with parameter α, defining the text information d of the i-th advertisement i The topic distribution θ i K topics related to popularity prediction {θ i1 ,θ i2 ,…,θ ik ,…,θ iK} and a background theme, where θ ik The text information d of the i-th advertisement i The probability distribution value on the kth topic related to popularity prediction, k∈[1,K];

[0081] Step 2.1.2, let φ k represents the kth topic word distribution, and φ k It follows a Dirichlet distribution with parameter β0 and has φ k ={φ k1 ,φ k2 ,…φ kv ,…φ kV}, where φ kv represents the probability that the vth word is assigned to the kth topic, v∈[1,V], where V represents the total number of non-repeated words in the text information of all advertisements;

[0082] make represents the word distribution under the background topic, and It obeys the Dirichlet distribution with parameter β1 and has in, represents the probability that the vth word is assigned to the background topic;

[0083] Step 2.1.3: Define binary indicator variable b = (b1, b2, ..., b i ,...,b D ), where b i Indicates the text information d of the i-th advertisement i The subject of all words in Among them, b in The text information d of the i-th advertisementi The nth word w in i,n The subject belongs to the situation and obeys the Bernoulli distribution;

[0084] If b in =1, indicating the text information d of the i-th advertisement i The nth word w in i,n Belonging to the topic related to popularity prediction, the text information d of the i-th advertisement is defined as i The topic number associated with popularity prediction for the nth word in is z in , and the obedience parameter is θ i Multinomial distribution; define the text information d of the i-th advertisement i The nth word w i,n The obedience parameter is A multinomial distribution where The text information d of the i-th advertisement i The nth word w i,n Belongs to the zth related to popularity prediction in Probability distribution of words under a topic;

[0085] If b in =0, indicating the text information d of the i-th advertisement i The nth word w i,n Belongs to the background theme; defines the text information d of the i-th advertisement i The nth word w i,n The obedience parameter is Multinomial distribution of ;

[0086] Step 2.1.4: Use the collapsed Gibbs sampling method shown in formulas (1) to (3) to calculate the text information d of the i-th advertisement. i The topic distribution θ i , word distribution φ under the kth topic related to popularity prediction k And word distribution under background topics

[0087]

[0088]

[0089]

[0090] In formulas (1) to (3), The text information d of the i-th advertisement i The number of words assigned to the kth topic related to popularity prediction; It represents the number of v-th words in all text information assigned to the k-th topic related to popularity prediction when the binary variable b is 1; It represents the number of v-th words in all text information assigned to the background topic when the binary variable b is 0;

[0091] According to the word distribution φ under the kth topic related to popularity prediction k , take the top E words with the largest probability to vectorize the kth topic related to popularity prediction, and combine it with the text information d of the i-th advertisement i The topic distribution θ i , get the feature matrix of text information in, The text information d of the i-th advertisement i Feature representation of the kth topic related to popularity prediction in ;

[0092] Step 2.2: Extract image region features using the pre-trained VGG-16 network.

[0093] The visual information of the i-th advertisement I i The mth picture p i,m Divide into Z regions and get a set of image regions in, Represents the visual information I of the i-th advertisement i The mth picture p i,m The zth region in Input VGG-16 network and output the mth picture p i,m The zth region The feature representation v i,m,z , thus obtaining the feature representation set V of the image area i,m =[v i,m,1 ,…,v i,m,z ,…,v i,m,Z ];

[0094] Step 2.3, obtain feature representation of other auxiliary information;

[0095] Step 2.3.1, obtain the feature representation of label and title information;

[0096] The label information l of the i-th advertisement i The jth label l i,j Represented as one-hot encoding y i,j , the label information l of the i-th advertisement i All labels are represented as corresponding one-hot encodings and constitute encoding vectors, which are then input into the LSTM model to obtain label information l i The eigenvector of

[0097] By tag information i The feature vector constructs label information l i The label feature matrix of the i-th advertisement is obtained by performing the mean pooling operation. i The feature representation f i label Similarly, get the title information of the i-th advertisement t i The feature representation f i title ;

[0098] Step 2.3.2, obtain the feature representation of author information;

[0099] Use author information collection a i Number of fans f i followers , author information set a i Number of followers i followings , author information set a i Number of works f i works As the author information set a of the i-th advertisement i The feature representation is denoted as f i author =[f i followers ,f i followings ,f i works ];

[0100] Step 2.3.3, obtain feature representation of time information;

[0101] The time information of the i-th advertisement is set T i Hour f i hour 、Sunday i week 、Japanese i day 、Moon i month 、yearf i year As the time information T of the i-th advertisement i The characteristic representation is denoted as f i time =[f i hour ,f i week ,f i day ,f imonth ,f i year ];

[0102] Step 3: Figure 2 As shown in the figure, a text-based attention mechanism is introduced. The attention mechanism can focus on text information that is helpful for popularity prediction and better obtain the feature representation of text information;

[0103] Step 3.1: Introduce the attention mechanism shown in Equations (4) and (5), and use Equation (6) to obtain the text information d of the i-th advertisement i Feature representation Q is the text feature dimension:

[0104]

[0105]

[0106]

[0107] In formulas (4) to (6), It is a word-level text feature representation The hidden representation, S is the hidden layer dimension, W topic Implicit representation The weight matrix, b topic Implicit representation The bias vector, α i,k ∈R is the text feature representation The attention score, W a Score for text attention The weight matrix, tanh represents the tangent activation function, Representation of text features The implicit representation of , k′∈[1,K];

[0108] Step 3.2: Obtain feature representations for each image using a text-based attention mechanism. Not all regions of an image are helpful for popularity prediction; in different contexts, the same image may have different regions that contribute to prediction. This method utilizes the text feature representations obtained in step 3.1 to provide context for the image, thereby filtering out noise and ultimately obtaining an accurate image representation.

[0109] Using formula (7) and formula (8) to obtain the image region feature representation combined with text information and the zth region The attention score α i,m,z ∈R, and then use formula (9) to get the mth picture p i,m Feature representation

[0110]

[0111]

[0112]

[0113] In formulas (7) to (9), W region Image region feature representation The weight matrix, W text Text feature representation The weight matrix, W v Attention score for image region The weight matrix of Represents the visual information I of the i-th advertisement i The mth picture p i,m The feature representation of the z′th region in , z′∈[1,Z], ⊙ represents the element product;

[0114] Step 3.3: Use the attention mechanism to fuse the features of multiple images and finally obtain the visual information I of the i-th advertisement. i Feature representation

[0115] Using formula (10) to get the visual information I of the i-th advertisement i The mth picture p i,m The latent feature representation h i,m ∈R Q , use formula (11) to calculate the mth picture p i,m The attention score a i,m ∈R, using formula (12) to obtain the visual information I of the i-th advertisement i Feature representation

[0116]

[0117]

[0118]

[0119] In formulas (10) to (12), W image Hidden feature representation for images The weight matrix, b image Hidden feature representation for images The bias vector, W h Score image attention The weight matrix of h i,m′ Represents the visual information I of the i-th advertisement iThe m′th picture p i,m′ The latent feature representation, m′∈[1,M i ];

[0120] Step 4: Fusion of text and visual features of multimodal ads. Text and visual information may reflect different perspectives of the target information. The combined effect of the two determines the popularity of the ad, so it is necessary to fuse the text and visual features. For this process, the present invention uses the following formula:

[0121] The linear function l shown in formula (13) is used to transform the text features and visual features Perform fusion to obtain the fusion representation of the i-th advertisement

[0122]

[0123] In formula (13), is the outer product; [] represents the linearization of the vector matrix;

[0124] Step 5: Use the fused global features to predict the popularity score;

[0125] Step 5.1, Connect them together to get the global feature f of the i-th advertisement i global , thus obtaining the global features f of all advertisements global And input the predicted popularity score into the classifier as shown in formula (14)

[0126]

[0127] In formula (14), W g and b g is the weight matrix and bias vector of the classifier;

[0128] Step 5.2: Use the binary cross entropy as the objective function J as shown in formula (15):

[0129]

[0130] In formula (15), C represents the number of data items in the multimodal data set D as the training set, y c represents the true popularity score of the cth advertisement in the training set, represents the predicted popularity score of the c-th advertisement in the training set; 1<C<|D|;

[0131] Step 5.3. Set the maximum number of iterations epoch_number and use backpropagation and gradient descent to train the objective function J so that J reaches the minimum. When the number of iterations reaches epoch_number, stop training to obtain the optimal prediction model, which is used to predict the popularity score of the input multimodal advertising data.

Claims

1. A multimodal advertising popularity prediction method based on text-supervised attention, characterized by Proceed as follows: Step 1: Construct a multimodal advertising data set D, where |D| represents the number of multimodal advertising data sets D. Any i-th advertisement in the multimodal advertising data set D contains multimodal information such as text description, image, label, title, author, and time, i∈[1,|D|]; Step 1.1: Construct a text information set of the multimodal advertisement data set D. The text information of the i-th advertisement in the text information set is recorded as Among them, w i,n The text information d of the i-th advertisement i The nth word in N i Indicates text information d i The total vocabulary size, n∈[1,N i ]; Step 1.2: Construct a visual information set of the multimodal advertisement data set D, wherein the visual information I of the i-th advertisement in the visual information set is i , recorded as Among them, p i,m Represents the visual information I of the i-th advertisement i The mth picture in M i Represents visual information I i The total number of pictures, m∈[1,M i ]; Step 1.3: Construct other auxiliary information sets of the multimodal advertising data set D, where the label information set of the i-th advertisement is recorded as l i , the title information set of the i-th advertisement is recorded as t i , the author information set of the i-th advertisement is recorded as a i , the time information set of the i-th advertisement is recorded as T i ; Step 2: Extract the features of multimodal advertisements; Step 2.1, obtain feature representation of text information based on denoising topic model; Step 2.1.1, define the topic distribution of all text information sets as θ = {θ1, θ2, ..., θ i ,…,θ |D| }, where θ i The text information d of the i-th advertisement i The topic distribution of , and obeys the Dirichlet distribution with parameter α, defining the text information d of the i-th advertisement i The topic distribution θ i K topics related to popularity prediction {θ i1 ,θ i2 ,…,θ ik ,…,θ iK } and a background theme, where θ ik The text information d of the i-th advertisement i The probability distribution value on the kth topic related to popularity prediction, k∈[1,K]; Step 2.1.2, let φ k represents the kth topic word distribution, and φ k It follows a Dirichlet distribution with parameter β0 and has φ k ={φ k1 ,φ k2 ,…φ kv ,…φ kV }, where φ kv represents the probability that the vth word is assigned to the kth topic, v∈[1,V], where V represents the total number of non-repeated words in the text information of all advertisements; make represents the word distribution under the background topic, and It obeys the Dirichlet distribution with parameter β1 and has in, represents the probability that the vth word is assigned to the background topic; Step 2.1.3: Define binary indicator variable b = (b1, b2, ..., b i ,...,b |D| ), where b i Indicates the text information d of the i-th advertisement i The subject of all words in Among them, b in The text information of the i-th advertisement d i The nth word w in i,n The subject belongs to the situation and obeys the Bernoulli distribution; If b in =1, indicating the text information d of the i-th advertisement i The nth word w in i,n Belonging to the topic related to popularity prediction, the text information d of the i-th advertisement is defined as i The topic number associated with popularity prediction for the nth word in is z in , and the obedience parameter is θ i Multinomial distribution; define the text information d of the i-th advertisement i The nth word w i,n The obedience parameter is A multinomial distribution where The text information d of the i-th advertisement i The nth word w i,n Belongs to the zth related to popularity prediction in Probability distribution of words under a topic; If b in =0, indicating the text information d of the i-th advertisement i The nth word w i,n Belongs to the background theme; defines the text information d of the i-th advertisement i The nth word w i,n The obedience parameter is Multinomial distribution of ; Step 2.1.4: Use the collapsed Gibbs sampling method shown in formulas (1) to (3) to calculate the text information d of the i-th advertisement. i The topic distribution θ i , word distribution φ under the kth topic related to popularity prediction k And word distribution under background topics In formulas (1) to (3), The text information d of the i-th advertisement i The number of words assigned to the kth topic related to popularity prediction; It represents the number of v-th words in all text information assigned to the k-th topic related to popularity prediction when the binary variable b is 1; It represents the number of v-th words in all text information assigned to the background topic when the binary variable b is 0; According to the word distribution φ under the kth topic related to popularity prediction k , take the top E words with the largest probability to vectorize the kth topic related to popularity prediction, and combine it with the text information d of the i-th advertisement i The topic distribution θ i , get the feature matrix of text information in, The text information d of the i-th advertisement i Feature representation of the kth topic related to popularity prediction in ; Step 2.2: Extract image region features using the pre-trained VGG-16 network. The visual information of the i-th advertisement I i The mth picture p i,m Divide into Z regions and get a set of image regions in, Represents the visual information I of the i-th advertisement i The mth picture p i,m The zth region in Input VGG-16 network and output the mth picture p i,m The zth region The feature representation v i,m,z , thus obtaining the feature representation set V of the image area i,m =[v i,m,1 ,…,v i,m,z ,…,v i,m,Z ]; Step 2.3, obtain feature representation of other auxiliary information; Step 2.3.1, obtain the feature representation of label and title information; The label information l of the i-th advertisement i The jth label l i,j Represented as one-hot encoding y i,j , the label information l of the i-th advertisement i All labels are represented as corresponding one-hot encodings and constitute encoding vectors, which are then input into the LSTM model to obtain label information l i The eigenvector of By tag information i The feature vector constructs label information l i The label feature matrix of the i-th advertisement is obtained by performing the mean pooling operation. i The feature representation f i label Similarly, get the title information of the i-th advertisement t i The feature representation f i title ; Step 2.3.2, obtain the feature representation of author information; Use author information collection a i Number of fans f i followers , author information set a i Number of followers i followings , author information set a i Number of works f i works As the author information set a of the i-th advertisement i The feature representation is denoted as f i author =[f i followers ,f i followings ,f i works ]; Step 2.3.3, obtain feature representation of time information; The time information of the i-th advertisement is set T i Hour f i hour 、Sunday i week 、Japanese i day 、Moon i month 、yearf i year As the time information T of the i-th advertisement i The characteristic representation is denoted as f i time =[f i hour ,f i week ,f i day ,f i month ,f i year ]; Step 3: Introduce text-based attention mechanism; Step 3.1: Introduce the attention mechanism shown in Equations (4) and (5), and use Equation (6) to obtain the text information d of the i-th advertisement i Feature representation Q is the text feature dimension: In formulas (4) to (6), It is a word-level text feature representation The hidden representation, S is the hidden layer dimension, W topic Implicit representation The weight matrix, b topic Implicit representation The bias vector, α i,k ∈R is the text feature representation The attention score, W a Score for text attention The weight matrix, tanh represents the tangent activation function, Representation of text features The implicit representation of , k′∈[1,K]; Step 3.2: Obtain feature representation of each image through the text-based attention mechanism; Using formula (7) and formula (8) to obtain the image region feature representation combined with text information and the zth region The attention score α i,m,z ∈R, and then use formula (9) to get the mth picture p i,m Feature representation In formulas (7) to (9), W region Image region feature representation The weight matrix, W text Text feature representation The weight matrix, W v Attention score for image region The weight matrix of Represents the visual information I of the i-th advertisement i The mth picture p i,m The feature representation of the z′th region in , z′∈[1,Z], ⊙ represents the element product; Step 3.3: Use the attention mechanism to fuse the features of multiple images and finally obtain the visual information I of the i-th advertisement. i Feature representation Using formula (10) to get the visual information I of the i-th advertisement i The mth picture p i,m The latent feature representation h i,m ∈R Q , use formula (11) to calculate the mth picture p i,m The attention score a i,m ∈R, using formula (12) to obtain the visual information I of the i-th advertisement i Feature representation In formulas (10) to (12), W image Hidden feature representation for images The weight matrix, b image Hidden feature representation for images The bias vector, W h Score image attention The weight matrix of h i,m′ Represents the visual information I of the i-th advertisement i The m′th picture p i,m′ The latent feature representation, m′∈[1,M i ]; Step 4: Fusion of text and visual features of multimodal advertisements; Using the linear function shown in formula (13) Text features and visual features Perform fusion to obtain the fusion representation of the i-th advertisement In formula (13), is the outer product; [] represents the linearization of the vector matrix; Step 5: Use the fused global features to predict the popularity score; Step 5.1, Connect them together to get the global feature f of the i-th advertisement i global , thus obtaining the global features f of all advertisements global And input the predicted popularity score into the classifier as shown in formula (14) In formula (14), W g and b g is the weight matrix and bias vector of the classifier; Step 5.2: Use the binary cross entropy as shown in formula (15) as the objective function J: In formula (15), C represents the number of data items in the multimodal data set D used as the training set, y c represents the true popularity score of the cth advertisement in the training set, represents the predicted popularity score of the c-th advertisement in the training set; 1<C<|D|; Step 5.

3. Set the maximum number of iterations epoch_number and use backpropagation and gradient descent to train the objective function J so that J reaches the minimum. When the number of iterations reaches epoch_number, stop training to obtain the optimal prediction model, which is used to predict the popularity score of the input multimodal advertising data.

Images