A click-through rate prediction model based on feature representation under attention mechanism

Abstract

In order to complete the click-through rate estimation according to the object features of the object to be tested, it can be used as a data sorting link in the fields of enterprise-level recommendation systems, search systems and online advertising systems. The present invention provides a click-through rate estimation model based on feature representation under an attention mechanism, the model includes: a feature embedding layer, which is used to vectorize continuous features and discrete features to form stacked features, an explicit feature cross network, an explicit feature combination of the stacked features through an attention cross network, an implicit feature cross network, an implicit feature combination of the stacked features through a multi-layer perceptron, and an estimated probability output layer, and the click-through rate is estimated according to the received combined features. Among them, the attention cross network eliminates the dependence of the estimation model on artificial feature engineering, and the introduction of the attention mechanism distinguishes the importance of each combined feature to the model estimation, and eliminates the influence of useless and redundant features on the model.

Description

Technical Field

[0001] The present invention belongs to the field of data mining technology, and specifically is an end-to-end click prediction technology, which uses a deep learning model to automatically complete feature characterization and then predict a click rate model. Background Art

[0002] As a key technology that directly affects user platform experience and advertising revenue, click-through rate estimation has always been one of the core research topics in the industry. At present, research work at home and abroad is mainly at the feature representation level, and existing methods are mainly divided into two categories: machine learning click-through rate model and deep learning click-through rate model.

[0003] In the early stage, the industry was limited by computing power, online learning and model deployment, and mainly built lightweight machine learning models. The most classic model was the Logistic Regression model. LR quickly became the mainstream model for CTR estimation in the industry with its advantages such as good mathematical meaning, strong interpretability and easy engineering deployment. In 2010, Brendan McMahan et al. proposed an online learning algorithm FTRL (Follow The Regularized Leader) for LR, which further promoted the application of LR in the industry. However, the LR model is essentially a linear model with limited model learning ability. Its prediction effect usually depends on the feature engineering ability of data scientists. Therefore, the industry began to explore the use of the binomial model (Polynomial Regression) to construct second-order combination features and use the feature crossover to perform explicit feature combination. Its brute force crossover method solves the feature combination problem to a certain extent, but it can only learn the co-occurrence features that appear in the training data set, and it is difficult to generalize the non-co-occurrence combination features in large-scale sparse data scenarios such as recommendation and advertising. In order to solve the defects of the binomial model, Steffen Rendle et al. from the University of Konstanz in Germany proposed FM (Factorization Machine) in 2010. They began to learn a latent weight vector for each feature and used the inner product of the latent vector as the feature cross weight, which effectively solved the problem of feature combination in sparse feature scenarios. In addition, by changing the form of the objective function, the complexity of FM training was further reduced. Therefore, FM gradually became an important choice for CTR models in the industry from 2012 to 2014. In 2015, the FFM (Field-aware Factorization Machine) proposed based on FM won the championship in many CTR prediction competitions and was subsequently applied by companies such as Criteo and Meituan in recommendation and advertising scenarios. Compared with the FM model, the FFM model mainly introduces the concept of "domain". When performing feature cross, each feature selects the latent vector corresponding to the combined feature domain and performs the inner product operation to obtain the cross feature weight, which makes the model more expressive. However, due to the high spatial complexity of FFM and the limitation that it can only perform second-order feature cross, it has not been widely used in the industry.In addition, in 2014, Xinran He et al. proposed a solution based on the GBDT (Gradient Boost Decision Tree) + LR combination model to deal with high-dimensional feature combination and screening problems. It uses GBDT to automatically screen and combine features, encodes leaf nodes with unique codes, and then uses the encoded features as the input of the LR model to complete CTR estimation. It has opened up a precedent for using models to construct and screen high-order features, and has more efficiently solved the very difficult feature combination and screening problems in the past, greatly promoting the important trend of feature engineering modeling.

[0004] During this period, researchers found that high-order combination features are more likely to tap into personalized needs and achieve a "one-size-fits-all" push effect. However, due to the rapid increase in sparse features and the difficulty in understanding the business logic of high-order combination features, traditional manual feature engineering is unsustainable in mining high-order combination features. People began to try to continue the advantages of model extraction features and complete personalized push for users in big data scenarios. With the great success of deep learning in computer vision and natural language processing, people began to try to use neural networks to automatically represent features, thereby replacing manual feature engineering to complete click-through rate estimation.

[0005] In 2016, deep learning began to be widely used in click-through rate prediction. Microsoft Ying Shan et al. proposed the DeepCrossing serial network structure, which covers the classic elements of the CTR prediction neural network model, namely, converting sparse features into low-dimensional dense features by adding an embedding layer, splicing segmented feature vectors with stacking layers, and then completing the combination and conversion of features through a multi-layer neural network. Finally, the CTR prediction is completed through the Sigmoid activation function, and the residual network structure is formally introduced into the click-through rate prediction model to enhance the model's high-order feature extraction capabilities. In the same year, Zhang Weinan et al. from Shanghai Jiaotong University proposed FNN. Based on the previous Deep Crossing network structure, the hidden layer vectors of FM were used as the embedding of users and materials, thereby avoiding the training of the embedding matrix completely from a random state, greatly reducing the model's training time and the instability of the embedding layer. Using the pre-training method to complete the training of the network embedding layer is undoubtedly an effective engineering experience to reduce the complexity of deep learning models and training instability. However, traditional DNNs directly use multiple layers of fully connected layers to complete feature cross-combinations, which lacks the "targetedness" of feature combination for click-through rate prediction scenarios. Therefore, Yanru Qu et al. proposed PNN (Product-based Neural Network), adding a Product layer between the embedding layer and the fully connected layer, aiming to combine features between different feature domains and enhance the model's ability to represent different data patterns.

[0006] In 2016, Heng-Tze Cheng et al. from Google proposed the Wide&Deep parallel network structure, which concatenates the Wide part of a single input layer and the Deep part after passing through a multi-layer perceptron and passes them to the output layer. The Wide part is used to implement the "memorization" of the model, and the Deep part is used to implement the "generalization" of the model. DNN mines implicit high-order feature combinations, and LR connects the Wide part and the Deep part to form a unified CTR model. Wide&Deep has established a parallel framework for click-through rate estimation based on deep learning, but it still has not gotten rid of the limitation of relying on artificial feature engineering. In response to the shortcomings of the Wide part, in 2017, Huifeng Guo et al. from Huawei proposed DeepFM. On the basis of continuing the Wide&Deep parallel network structure, FM was used to replace the original Wide part, which strengthened the feature combination capability of the shallow network and got rid of the reliance of the deep model for click-through rate estimation on artificial feature engineering. In the same year, Ruoxi Wang et al. proposed Deep & Cross Network (DCN), and began to use Cross network to replace the original Wide part, realizing bit-level explicit feature interaction, and further refined the feature cross granularity of the Wide part. Xiangnan He et al. proposed NFM (Neural Factorization Machines) to improve the Deep part, introduced Bi-interaction Pooling layer to replace FM for feature cross, and further strengthened the feature combination ability of the Deep layer. In 2018, Guorui Zhou et al. of Alibaba proposed DIN (Deep Interest Network), a deep learning network based on attention mechanism, to extract real-time interest features from user behavior sequences, and further improved the user-side feature representation. In the same year, Jianxun Lian et al. proposed xDeepFM parallel network structure to model vector-level explicit feature interaction. Its Wide part uses CIN (Compressed Interaction Network) to strengthen the explicit feature combination ability of the model, and achieved certain results. Subsequently, in 2019, Guorui Zhou et al. proposed DIEN (Deep Interest Evolution Network), and began to introduce the sequence model AUGRU based on DIN, connecting user interests at different times to form a chain of interest evolution. Finally, the "interest vector" at the current moment was input into the upper-layer multi-layer perceptron, and combined with other features to complete the click-through rate estimation, achieving good results.

[0007] In summary, the current deep learning model still cannot replace manual feature engineering with neural networks for feature extraction, feature combination, and feature screening. Furthermore, it is impossible to accurately estimate the trend of feature changes under the premise of replacing manual feature engineering with neural networks, and it is also impossible to automatically obtain accurate click-through rate estimates. Summary of the invention

[0008] In order to solve the above problems, a click rate prediction model based on feature representation under the attention mechanism is provided. The present invention adopts the following technical solutions:

[0009] The present invention discloses a click rate prediction model based on feature representation under an attention mechanism, which includes: a feature embedding layer, an explicit feature cross network, an implicit feature cross network and an estimated probability output layer. The feature embedding layer vectorizes continuous features and discrete features and then performs stacking and embedding processing to form stacked features; the explicit feature cross network inputs the stacked features into the attention cross network to perform explicit feature combination to form an explicit output vector; the implicit feature cross network inputs the stacked features into a multi-layer perceptron to perform implicit feature combination to form an implicit output vector; the estimated probability output layer combines the explicit output vector and the implicit output vector to form a high-order nonlinear combination feature, and at the same time passes the combination feature to a Sigmoid activation function to predict the click rate, thereby obtaining the click rate; wherein the attention cross network includes: a cross layer, which processes the stacked features through a cross algorithm and generates a multidimensional vector; and an attention layer, which processes the multidimensional vector through a fully connected neural network and generates an attention score, and normalizes the attention score to generate a feature coefficient, and further generates an explicit output vector based on the feature coefficient through an output calculation formula.

[0010] The present invention provides a labor contraction discrimination system based on machine learning, which also has such a technical feature, wherein the vectorization processing is: performing one-hot encoding conversion on discrete features, and using the encoded discrete features as embedding vectors; performing data standardization on continuous features based on data distribution characteristics to form dense features; stacking embedding processing on the embedded vectors and dense features as stacked features, and the matrix calculation formula of the one-hot encoding conversion is:

[0011] x embed，i =W embed，i x i #(1)

[0012] In the formula, x embed，i is the embedding vector, x i is the binary input of the ith class, is the embedding matrix that will be optimized together with other parameters in the network, and n e and n vare the input dimension and the embedding vector dimension respectively.

[0013] The present invention provides a labor contraction discrimination system based on machine learning, which also has the following technical features: the calculation formula of the crossover algorithm is:

[0014]

[0015] In the formula, are column vectors, representing the cross-layer outputs from the lth layer and the l+1th layer respectively; are the weights and biases of the lth layer, and the function f represents the cross formula of the feature vectors of each layer.

[0016] The machine learning-based labor contraction discrimination system provided by the present invention also has such a technical feature, wherein, the calculation logic of the normalization processing in the attention layer is:

[0017] a′ i =h T Relu(Wx i +b)#(3)

[0018]

[0019] In the formula, is a model parameter and the attention score is normalized by Softmax.

[0020] The present invention provides a labor contraction discrimination system based on machine learning, which also has the following technical features: the output calculation formula of the attention cross network is:

[0021]

[0022] In the formula, a i is the attention weight.

[0023] The present invention provides a labor contraction discrimination system based on machine learning, which also has the following technical features: the calculation logic of each layer of the multi-layer perceptron is as follows:

[0024] H l+1 =f(W l H l +B l )#(7)

[0025] In the formula, H l+1 represents the hidden layer, and f(·) is the ReLU function.

[0026] The machine learning-based labor contraction discrimination system provided by the present invention also has such a technical feature, wherein the formula of the Sigmoid activation function is:

[0027]

[0028] In the formula, They are the outputs of the explicit feature cross network and the multi-layer perceptron, and the final click rate prediction value is obtained through the Sigmoid function.

[0029] A machine learning-based labor contraction discrimination system provided by the present invention also has such a technical feature, wherein the click-through rate prediction model based on feature representation under the attention mechanism performs error backpropagation on the click-through rate through the Log1oss loss function until the click-through rate output by the output layer converges, thereby completing the parameter update of the click-through rate prediction model based on feature representation under the attention mechanism.

[0030] The present invention provides a labor contraction discrimination system based on machine learning, which also has the following technical features: the formula of the Log1oss loss function is as follows:

[0031]

[0032] In the formula, p i is the output of the click rate prediction model, y i is the label corresponding to the sample, N is the number of training samples, and λ is the L2 regularization coefficient. The error is back-propagated through the Logloss loss function and the parameters are updated based on this until convergence to complete the final click-through rate model training.

[0033] Function and Effect of the Invention

[0034] According to a click-through rate prediction model based on feature representation under an attention mechanism of the present invention, due to the feature embedding layer, the feature embedding layer overcomes the problem of excessive vector dimension after unique hot encoding processing by vectorizing continuous features and discrete features. At the same time, it also has an explicit feature cross network, which realizes dynamic weighting of combination items through an attention cross network, more efficiently utilizes the combination features, and eliminates the influence of redundant features on the click-through rate prediction model. Further, it also has an implicit feature cross network, which completes the capture of highly nonlinear interactive features by using a multi-layer perceptron, thereby overcoming the problem that the feature expression ability of the model is limited by the parameter scale. Finally, the present invention has an estimated probability output layer, which estimates the click-through rate based on the output of the explicit feature cross network and the implicit feature cross network through a Sigmoid activation function, so that the obtained estimated data is more accurate. Further, the estimated data can be used as a data precision ranking link in the fields of enterprise-level recommendation systems, search systems, and online advertising systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1It is a structural block diagram of a click rate prediction model based on feature representation under the attention mechanism in an embodiment of the present invention;

[0036] Figure 2 is a workflow diagram of a feature embedding layer in an embodiment of the present invention;

[0037] Figure 3 is a network structure diagram of an attention cross network in an embodiment of the present invention;

[0038] Figure 4 is a network structure diagram of a multi-layer perceptron in an embodiment of the present invention;

[0039] Figure 5 is a flow chart of the training process of a click rate prediction model based on feature representation under the attention mechanism in an embodiment of the present invention;

[0040] Figure 6 is a flow chart of the deployment of the click rate prediction model in an embodiment of the present invention;

[0041] Figure 7 is a schematic diagram of a comparison result of single model performance in an embodiment of the present invention;

[0042] Figure 8 is a schematic diagram of the comparison results of the integrated model performance in the embodiment of the present invention;

[0043] Fig. 9 Schematic diagram of the comparison result of the number of network parameters in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of a click-through rate prediction model based on feature representation under an attention mechanism of the present invention in combination with embodiments and drawings.

[0045] <Example>

[0046] Figure 1 It is a structural block diagram of a click rate prediction model based on feature representation under the attention mechanism in an embodiment of the present invention.

[0047] like Figure 1 As shown, the click rate prediction model 100 based on feature representation under the attention mechanism includes: a feature embedding layer 101, an explicit feature cross network 102, an implicit feature cross network 103 and an estimated probability output layer 104.

[0048] In this embodiment, user-side features, advertising-side features, and context features are collected, and the collected features are divided into two categories: continuous features and discrete features. These two types of features are used as input features, and are organized to form a training data set. The training data set is further used to complete the model training of the click-through rate prediction model 100 based on feature representation under the attention mechanism. This training data set consists of n samples (x, y), and the input features are continuous features and discrete features. The goal is to build a click-through rate prediction model y=model(x), where x∈R^n is used to predict the user u's click on the candidate material v in a specified context. t The probability y, y∈[0,1]

[0049] The feature embedding layer 101 vectorizes the received input features, and stacks the obtained embedding vectors and dense features to form stacked features and output them.

[0050] Figure 2 4 is a flowchart of the feature embedding layer in an embodiment of the present invention.

[0051] like Figure 2 As shown, the steps of forming stacked features in the feature embedding layer 101 are as follows:

[0052] Step S1, perform one-hot encoding conversion on discrete features, use the encoded discrete features as embedding vectors, and then proceed to step S2;

[0053] Step S2, for continuous features, data standardization is performed based on data distribution characteristics to form dense features, and then step S3 is entered;

[0054] Step S3, stacking and embedding the embedding vector formed in step S1 and the dense feature formed in step S2, that is, stacking the embedding vector and the dense feature into one vector, and calling the vector the stacked feature, and then entering the end state.

[0055] In this embodiment, the matrix calculation formula for stacking embedding processing is:

[0056] x embed，i =W embed，i x i #(1)

[0057] In the formula, x embed，i is the embedding vector, x i is the binary input of the ith class, and is the embedding matrix that will be optimized together with other parameters in the network, and n e and n v are the input dimension and the embedding vector dimension respectively.

[0058] The explicit feature cross network 102 receives the stacked features formed by the feature embedding layer 101, and performs a cross algorithm on the stacked features through the attention cross network to generate a multi-dimensional vector;

[0059] Figure 3 4 is a network structure diagram of the attention cross network in an embodiment of the present invention.

[0060] like Figure 3 As shown in Figure A, the attention cross network includes: a cross layer 21 and an attention layer 22.

[0061] like Figure 3 As shown in Figure B, the number of neurons in each layer of the cross layer 21 is the same and equal to the dimension of the input vector x0.

[0062] In this embodiment, the calculation formula of the crossover algorithm is:

[0063]

[0064] In the formula, are column vectors, representing the cross-layer outputs from the lth layer and the l+1th layer respectively; are the weights and biases of the lth layer, and the function f represents the cross formula of the feature vectors of each layer.

[0065] The attention layer 22 processes the multidimensional vector through a fully connected neural network and generates an attention score, normalizes the attention score to generate a feature coefficient, and generates an explicit output vector through an output calculation formula based on the feature coefficient.

[0066] In this embodiment, the fully connected neural network uses the Attention network, the activation function uses ReLU, and the network size is represented by the attention factor. In addition, the calculation logic of the normalization processing in the attention layer 22 is:

[0067] a′ i =h T Relu(Wx i +b)#(3)

[0068]

[0069] In the formula, is a model parameter, and the attention score is normalized by Softmax. The result of the above formula is used as the feature coefficient, where the output calculation formula of the attention cross network is:

[0070]

[0071] In the formula, a i is the attention weight.

[0072] The implicit feature cross network 103 performs implicit feature combination by inputting the stacked features into a multi-layer perceptron to form an implicit output vector.

[0073] Figure 4 1 is a network structure diagram of a multilayer perceptron in an embodiment of the present invention.

[0074] like Figure 4 As shown, the multilayer perceptron in this embodiment is a fully connected feedforward neural network, and the calculation logic of each layer is as follows:

[0075] H l+1 =f(W l H l +B l )#(7)

[0076] In the formula, H l+1 represents the hidden layer, and f(·) is the ReLU function.

[0077] The estimated probability output layer 104 combines the explicit output vector and the implicit output vector to form a high-order nonlinear combined feature, and passes the combined feature to the Sigmoid activation function for click-through rate prediction to obtain an estimated click-through rate.

[0078] In this embodiment, the formula of the Sigmoid activation function is:

[0079]

[0080] In the formula, They are the outputs of the explicit feature cross network and the multi-layer perceptron, and the final click rate prediction value is obtained through the Sigmoid function;

[0081] After obtaining the predicted click-through rate value, the click-through rate prediction model 100 based on feature representation under the attention mechanism performs error backpropagation on the estimated click-through rate through the Logloss loss function until the estimated click-through rate output by the output layer converges, thereby completing the parameter update of the click-through rate prediction model based on feature representation under the attention mechanism, and then completing the model training of the click-through rate prediction model 100 based on feature representation under the attention mechanism based on the training data set in this embodiment.

[0082] In this embodiment, the formula of the Logloss loss function is as follows:

[0083]

[0084] In the formula, p i is the output of the click rate prediction model, y i is the label corresponding to the sample, N is the number of training samples, and λ is the L2 regular term coefficient.

[0085] Figure 5 It is a flow chart of the training process of the click rate prediction model based on feature representation under the attention mechanism in an embodiment of the present invention.

[0086] like Figure 5 As shown in FIG. 1 , the training process of the click rate prediction model based on feature representation under the attention mechanism in this embodiment is:

[0087] Step U1, construct a data set and divide the data features in the data set into continuous data features and discrete data features, and then proceed to step U2;

[0088] Step U2, constructing a feature embedding layer, vectorizing continuous features and discrete features, stacking and embedding them to form stacked features, and then proceeding to step U3;

[0089] Step U3, construct an explicit feature cross network, input the stacked features into the attention cross network to perform explicit feature combination, form an explicit output vector, and then enter step U4;

[0090] Step U4, construct an implicit feature cross network, input the stacked features into a multi-layer perceptron to perform implicit feature combination, form an implicit output vector, and then enter step U5;

[0091] Step U5, construct an estimated probability output layer, combine the explicit output vector and the implicit output vector to form a high-order nonlinear combination feature, and pass the combined feature to the Sigmoid activation function for click-through rate prediction to obtain the estimated click-through rate, and then enter step U6;

[0092] In step U6, the error of the estimated click rate is back-propagated through the Logloss loss function until the estimated click rate output by the output layer converges, completing the parameter update of the click rate estimation model based on feature representation under the attention mechanism, and then completing the model training, and then entering the end state.

[0093] The programming environment used for the system implementation in this embodiment is Pycharm, and the Python version is 3.6.

[0094] Figure 6 It is a flow chart of the deployment work of the click rate prediction model in the embodiment of the present invention.

[0095] like Figure 6 As shown, the specific working steps after the training of the click rate prediction model based on feature representation under the attention mechanism of this embodiment are as follows:

[0096] Step T1, collect original data for real business scenarios, that is, obtain original data through splicing and aggregation through early data embedding, background log extraction and online information collection, and then enter step T2;

[0097] Step T2, pre-processing the raw data collected in step T1 to form sorted data, and then proceeding to step T3;

[0098] In this embodiment, the preprocessing includes: performing outlier processing, missing value processing and noise data processing on the collected original data.

[0099] Step T3, construct the sorted data into a training data set, a test data set, and a validation data set for the click rate prediction model. The proportion of the data set should be determined based on the data volume and the business, and then proceed to step T4;

[0100] Step T4, input each data set into the click rate prediction model to obtain the estimated click rate, and then proceed to step T5;

[0101] Step T5, combining the estimated click rate to mine feature engineering that meets the user's interest preferences, no manual feature engineering is performed here, and then proceed to step S6;

[0102] Step T6, extract the corresponding optimal hyper-parameter combination according to the performance of the offline data set, and then proceed to step T7;

[0103] Step T7, evaluate the prediction performance of the click prediction model by using the manually set model measurement indicators (Logloss and AUC are often used as evaluation indicators in the click rate prediction scenario), and then proceed to step T8;

[0104] In step T8, the algorithm engineer conducts an online small-traffic test on the click prediction model to verify the online effect of the model. The model can be deployed online only after the test passes, and then enters the end state.

[0105] In this embodiment, the hyperparameter selection methods in T6 are grid search, random search and Bayesian search.

[0106] After completing the above model deployment, a series of experiments were conducted using the click-through rate prediction model 100 (Deep & Attention Cross Network, DACN) based on feature representation under the attention mechanism of the present invention. The programming environment used for the implementation of the attention cross network (ACN) in the click-through rate prediction model 100 is Pycharm, and the Python version is 3.6. The operating environment of the experiment is Core i7 CPU, 32GB memory, and Linux operating system. The experimental data set comes from the real click data of Criteo Lab and the movie review data of MovieLens. Two indicators are used for model evaluation: AUC and Logloss, which evaluate the performance of the model from different levels.

[0107] The experiment will implement a new feature cross network DACN (Deep & Attention Cross Network) that combines explicit and implicit features, that is, the present invention, and compare it with LR (Logistic Regression), DNN, FM (Factorization Machines), Wide & Deep, DCN (Deep & Cross Network) and DeepFM. As mentioned above, these models are currently mainstream and industry-verified click-through rate prediction models. Because DACN aims to extract feature combinations through the model, in order to control variables, we will not perform any artificial feature engineering on the original features.

[0108] This paper implements DACN on TensorFlow. For dense features, logarithmic transformation is used to normalize the data; for categorical features, the features are embedded into a 6×dimension 1 / 4 The dense vector of ; Adam optimizer is used, Mini-batch stochastic gradient descent is adopted, where the batch size is set to 512, and the DNN network is set to batch normalization. For the comparison model, the parameter settings for FNN and PNN in PNN are followed. Among them, the DNN module sets Dropout to 0.5, the network structure is set to 400-400-400, the optimization algorithm uses Mini-batch gradient descent based on Adam, the activation function is uniformly used Relu, the embedding dimension of FM is set to 10, and the rest of the model settings are consistent with DACN.

[0109] Figure 7 Schematic diagram of the comparison results of single model performance in an embodiment of the present invention.

[0110] First, we verify the effect of explicit combination features under the attention mechanism on the overall model prediction performance. Among the comparison models, FM explicitly measures the second-order feature interaction, DNN models implicit high-order feature interaction, Cross Network models explicit high-order feature interaction, and ACN models explicit feature interaction and has built-in feature filtering. The performance of each single model on the two public datasets is as follows: Figure 7 shown.

[0111] Experiments show that the proposed ACN is always better than other comparison models. As a conclusion, on the one hand, for practical data sets, high-order interactions on sparse features are necessary, which is proved by the fact that DNN, Cross Network and ACN are significantly better than FM on the above two data sets; on the other hand, ACN is the best individual model, which verifies the effectiveness of ACN in modeling explicit high-order feature interactions.

[0112] Figure 8Schematic diagram of the performance comparison results of the integrated model in the embodiment of the present invention

[0113] DACN integrates ACN and DNN into an end-to-end network structure. ACN is used for explicit combination feature extraction and screening, and DNN is used for implicit combination feature extraction. The two are combined in parallel to maximize feature representation. The performance of DACN and the current mainstream click-through rate prediction model on two public data sets is compared. The experimental results are as follows: Figure 8 shown.

[0114] from Figure 8 It is easy to see that LR is worse than all other models, which shows that models based on factor decomposition are crucial for modeling sparse class interaction features; while Wide&Deep, DCN and DeepFM are significantly better than DNN. The results show that DNN's implicit feature extraction ability is relatively limited, and it usually needs to use artificial feature engineering to make up for the shortcomings of insufficient feature combination ability. Secondly, DACN has significantly improved indicators compared to DCN. The advantages of DACN have been demonstrated from a theoretical perspective above. By adding the Attention network structure, the screening of combined features of each specified order can be realized, the weight of important combined features can be increased, and the influence of redundant features can be eliminated. Experimental results show that this structure can effectively realize feature screening and greatly improve the overall model performance.

[0115] The DACN proposed in the present invention achieves the best performance on both public datasets, which shows that the combination of explicit and implicit high-order features can better represent the original features. At the same time, the experimental results also verify that the use of ACN for the specified-order explicit feature combination has a great improvement on the final model performance, which indirectly verifies the rationality of the DACN proposed in the present invention.

[0116] Fig. 9 Schematic diagram of the comparison result of the number of network parameters in the embodiment of the present invention

[0117] Considering the additional parameters introduced by ACN, we compared ACN, CrossNet and DNN on the Criteo dataset to compare the minimum number of parameters required for each model to achieve the optimal logarithmic loss threshold. Since the number of embedding matrix parameters of each model is equal, the number of parameters in the embedding layer is omitted in the parameter calculation. The experimental results are shown in Figure 2. Fig. 9 shown.

[0118] It is not difficult to see from the experimental results that the storage efficiency of the ACN and Cross Network proposed in the present invention is nearly an order of magnitude higher than that of DNN. The main reason is that the shared feature cross structure realizes the interaction of specified-order features with linear space complexity. In addition, the number of parameters of ACN and Cross Network are of the same order of magnitude. The Attention network introduced by ACN contains only one hidden layer, and the number of required parameters can be approximately ignored, but it has a significant improvement in the prediction accuracy of the model's click rate. This verifies from the side that the ACN structure proposed in the present invention has a great advantage in terms of space complexity.

[0119] Example Function and Effect

[0120] According to this embodiment, a click-through rate prediction model based on feature representation under an attention mechanism is provided. Due to the feature embedding layer, the feature embedding layer vectorizes continuous features and discrete features, thereby overcoming the problem of excessive vector dimension after unique hot encoding. At the same time, it also has an explicit feature cross network, which realizes dynamic weighting of combined items through an attention cross network, more efficiently utilizes combined features, and eliminates the influence of redundant features on the click-through rate prediction model. Further, it also has an implicit feature cross network, which completes the capture of highly nonlinear interactive features by using a multi-layer perceptron, thereby overcoming the problem that the feature expression ability of the model is limited by the parameter scale. Finally, due to the estimated probability output layer, the estimated probability output layer uses the Sigmoid activation function to perform click-through rate estimation based on the output of the explicit feature cross network and the implicit feature cross network, so that the obtained estimated data is more accurate. Further, the estimated data can be used as a data precision ranking link in enterprise-level recommendation systems, search systems, online advertising systems and other fields.

[0121] In the embodiment, since discrete features are converted into one-hot encoding in the feature embedding layer, the encoded discrete features are used as embedding vectors and the continuous features are normalized according to the data distribution characteristics to form dense features; the above two low-dimensional dense vectors can more effectively retain the original semantic information.

[0122] In an embodiment, the crossover algorithm proposed in the crossover layer makes the combination of explicit features more efficient.

[0123] In the embodiment, the attention layer compresses different parts together so that the contribution of different parts is different, and further allows the model to learn the combined feature weights, thereby achieving automatic feature extraction.

[0124] In the embodiment, the display feature cross network realizes dynamic weighting of combination items through the mechanism of attention cross network, utilizes the combination features more efficiently, and eliminates the influence of redundant features on the click rate prediction model.

[0125] In the embodiment, the explicit feature cross network and the implicit feature cross network are combined in parallel to further enhance the feature representation ability of the model and improve the accuracy of click rate prediction.

[0126] In the embodiments, experiments are conducted to compare the single model performance, integrated model performance and number of network parameters of the present invention with the current mainstream and industry-verified click-through rate prediction model, which further verifies the rationality of the present invention and the great advantage of the ACN structure proposed in the present invention in terms of space complexity.

[0127] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A click rate prediction model based on feature representation under an attention mechanism, which is used to complete the click rate prediction according to the object features of the object to be tested, wherein the object features are divided into continuous features and discrete features, and is characterized in that: include: A feature embedding layer, which performs vectorization processing on the continuous features and the discrete features, and then stacks and embeds them to form stacked features; An explicit feature cross network, which performs explicit feature combination by inputting the stacked features into an attention cross network to form an explicit output vector; An implicit feature cross network, which forms an implicit output vector by inputting the stacked features into a multi-layer perceptron for implicit feature combination; The estimated probability output layer combines the explicit output vector and the implicit output vector to form a high-order nonlinear combined feature, and passes the combined feature to a Sigmoid activation function to predict the click rate, thereby obtaining the click rate; Wherein, the attention cross network includes: A cross layer, processing the stacked features through a cross algorithm and generating a multi-dimensional vector; and The attention layer processes the multidimensional vector through a fully connected neural network and generates an attention score, normalizes the attention score to generate a feature coefficient, and further generates the explicit output vector through an output calculation formula based on the feature coefficient. The output calculation formula of the attention cross network is: In the formula, a i is the attention weight, are model parameters, x i is the binary input of the ith class.

2. According to claim 1, a click rate prediction model based on feature representation under an attention mechanism, Features: The vectorization process is as follows: Performing one-hot encoding conversion on the discrete features, and using the encoded discrete features as embedding vectors; Performing data standardization on the continuous features according to data distribution characteristics to form dense features; Performing the stacking embedding process on the embedding vector and the dense feature as a stacking feature, The matrix calculation formula for the one-hot encoding conversion is: x embed，i =W embed，i x i (1) In the formula, x embed，i is the embedding vector, x i is the binary input of the ith class, and is the embedding matrix that will be optimized together with other parameters in the network, and n e and n v are the input dimension and the embedding vector dimension respectively.

3. The click rate prediction model based on feature representation under the attention mechanism according to claim 1, characterized in that: in, The calculation formula of the crossover algorithm is: In the formula, are column vectors, representing the cross-layer outputs from the lth layer and the l+1th layer respectively; are the weights and biases of the lth layer, and the function f represents the cross formula of the feature vectors of each layer.

4. The click rate prediction model based on feature representation under the attention mechanism according to claim 1, characterized in that: in, The calculation logic of the normalization process in the attention layer is: a′ i =h T Air(Wx i +b) (3) In the formula, are model parameters, x i is the binary input of the i-th class, a i is the attention weight, and the attention score is normalized by Softmax.

5. The click rate prediction model based on feature representation under the attention mechanism according to claim 1, characterized in that: in, The calculation logic of each layer of the multi-layer perceptron is as follows: H l+1 =f(W l H l +B l ) (7) In the formula, H l+1 represents the hidden layer, f(·) is the Relu function, are the weights and biases of layer l.

6. The click rate prediction model based on feature representation under the attention mechanism according to claim 1, characterized in that: in, The formula of the Sigmoid activation function is: In the formula, are the outputs of the explicit feature cross network and the multi-layer perceptron, respectively, and W logits is the weight of the Sigmoid function, and the final click rate prediction value is obtained through the Sigmoid function.

7. The click rate prediction model based on feature representation under the attention mechanism according to claim 1, characterized in that: in, The click-through rate prediction model based on feature representation under the attention mechanism performs error backpropagation on the click-through rate through the Logloss loss function until the click-through rate output by the output layer converges, thereby completing the parameter update of the click-through rate prediction model based on feature representation under the attention mechanism.

8. The click rate prediction model based on feature representation under the attention mechanism according to claim 1, characterized in that: in, The formula for the Logloss loss function is as follows: In the formula, w l is the weight of the lth layer, p i is the output of the click rate prediction model, y i is the label corresponding to the sample, N is the number of training samples, and λ is the L2 regularization coefficient. The error is back-propagated through the Logloss loss function and the parameters are updated based on this until convergence to complete the final click-through rate model training.

Images