A click rate prediction method based on deep feature fusion

Abstract

A method for estimating click rate based on deep feature fusion includes the following steps: obtaining historical data, converting the data into high-dimensional sparse features composed of 0 and 1 after hot encoding, and obtaining basic features; constructing feature vectors of the basic features through an embedding layer, and using the constructed vectors as model training input; learning explicit and implicit feature interactions through a shallow model FmFM and a deep model DenseDNN; splicing the results of the FmFM model and the DenseDNN network, and obtaining the output result through a Sigmoid function, and using AUC and LogLoss as two indicators to judge the accuracy of model prediction. The present invention improves model performance by feature fusion, making the results of click rate estimation more accurate and improving the benefits of advertising media.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and is a click rate prediction method based on deep feature fusion. Background Art

[0002] Internet platforms such as e-commerce platforms and search engine platforms determine the price of online advertising based on the click-through rate and click value of the advertisement. If these Internet platforms can accurately estimate the user's preference for a certain advertisement, and then precisely deliver the advertisement to the users with higher preference, thereby greatly increasing the probability of users clicking on the advertisement, then the advertising revenue obtained by the Internet platform will also increase accordingly. The indicator that reflects the user's preference for a certain advertisement is the click-through rate (CTR), which represents the ratio of the number of ad clicks to the number of ad impressions.

[0003] Traditional CTR prediction models such as the Factorization Machine (FM) model simulate the interaction between two features through the dot product between the embedding vectors corresponding to the two features, which reduces the overhead but ignores the differences when features from different domains interact. The FFM (Field-aware Factorization Machines) model introduces the concept of feature domain awareness on the basis of the FM model, making the model more expressive, but its parameters are too large and are not advisable in actual production systems. The FwFM (Field-weighted Factorization Machines) model effectively models the interactions between different domains, using only about 4% of the parameters of the FFM, but only using a scalar to express the intensity of the domain interaction, which has insufficient degrees of freedom and limited expressive power. The FmFM (Field-matrixed Factorization Machines) model uses a domain matrix between two feature vectors to simulate their interactions, which has higher degrees of freedom and stronger expressive power.

[0004] In recent years, deep neural networks (DNNs) have achieved success in computer vision, speech recognition, and natural language processing with their powerful feature representation learning capabilities. There are currently many click-through rate prediction models based on deep learning. These models generally use a combination of shallow models and deep models to learn feature interactions, which can effectively capture high-order feature interactions and significantly improve model performance. For example, the Wide&Deep model proposed by Google in 2016, which mixed a linear model and a Deep model, but the input of the linear part relied on manual feature engineering; Huawei proposed the DeepFM model in 2017, which replaced the Wide part with an FM model based on the Wide&Deep model. It does not require manual feature engineering and has high training efficiency, but its accuracy is still insufficient; Google proposed the DCN model in 2017, which uses a cross network and does not require manual feature engineering. The network structure is simple and efficient, but the feature interaction is at the element level; Microsoft proposed the xDeepFM model in 2018, whose CIN part can automatically learn explicit high-order feature interactions, and the interaction is at the vector level, but it is highly complex and difficult to apply; Google proposed the DCN-v2 model in 2020, whose core is the cross layer, which inherits the simple cross-network structure in DCN, but performs very well in learning explicit and bounded cross features; Purdue University and Yahoo Research proposed the DeepLight model in 2021, which uses a high-quality, low-consumption, and low-latency model to increase the model's inference speed by dozens of times without losing prediction accuracy.

[0005] The existing click-through rate prediction models are usually a combination of shallow models and deep models in parallel, and the deep models usually use deep neural networks in deep learning. The click-through rate prediction models based on deep learning can effectively capture high-order feature interactions and significantly improve the performance of the model. The industry has proposed many click-through rate prediction models combined with deep neural networks. Most of the parallel models improve the model effect by improving the shallow model part that learns explicit feature interactions. For example, the DeepFM model replaces the Wide part of Wide&Deep with the FM model, the xDeepFM model designs the CIN network, and the DeepLight model replaces the FM part of the DeepFM model with the FwFM model. However, there is a lack of research on the deep neural network part. In theory, in order to improve the performance of deep neural networks, the number of layers of the DNN network can be increased or the number of neuron parameters can be increased. However, in practical applications, this will lead to problems such as gradient vanishing, gradient explosion, and excessive number of parameters. The performance of the DNN will drop rapidly after reaching saturation as the number of layers increases. Summary of the invention

[0006] The present invention discloses a click rate prediction method based on deep feature fusion. The technology uses feature fusion to improve model performance, making the results of click rate prediction more accurate and improving the benefits of advertising media.

[0007] A click rate prediction method based on deep feature fusion includes the following steps:

[0008] S1. Get the historical data, heat encode the data into high-dimensional sparse features composed of 0 and 1, and obtain the basic features;

[0009] S2, construct feature vectors of basic features through the embedding layer, and use the constructed vectors as model training input;

[0010] S3, through the shallow model FmFM and the deep model DenseDNN, learn explicit and implicit feature interactions at the same time;

[0011] S4. After concatenating the results of the FmFM model and the DenseDNN network, the output result is obtained through the Sigmoid function, and the AUC and LogLoss indicators are used to judge the accuracy of the model prediction.

[0012] The purpose of the present invention is to solve the problem of the lack of research on deep models in existing click-through rate prediction models, and propose a DenseDNN network, which converts the input of each layer of the DNN network into the sum of the outputs of all previous layers, strengthens feature propagation, and achieves feature fusion. In order to further improve the accuracy of the model, the shallow model FmFM with the best effect in the current research is selected and combined with the deep model DenseDNN, and the combined model is named DDNNFMFM. Compared with the existing click-through rate prediction model method, the advantages of the present invention are:

[0013] 1) A DenseDNN network is proposed, which takes the sum of the outputs of each previous layer of the DNN network as the input of the next layer to strengthen feature propagation and achieve feature fusion.

[0014] 2) A DDNNFMFM model is proposed, which combines the shallow model FmFM with the deep model DenseDNN in parallel, and simultaneously learns the interaction of high-order and low-order features to further improve the estimation accuracy.

[0015] 3) The DDNNFMFM model was fully compared in two classic datasets, Criteo and Avazu. In the actual application of advertising recommendation systems with huge data volumes, a small improvement in the estimation accuracy may lead to a significant increase in the online click-through rate. This verifies the effectiveness of the DDNNFMFM model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the required drawings:

[0017] Figure 1This is a technical solution flow chart of an implementation case of the present invention, showing the approximate technical solution of the click-through rate prediction method based on deep feature fusion.

[0018] Figure 2 This is the DDNNFMFM model diagram of the present invention, which shows the calculation steps in the implementation process in an intuitive way.

[0019] Figure 3 This is an interactive operation example of the shallow model FmFM used in the present invention.

[0020] Figure 4 This is the structure of the innovative deep model DenseDNN of the present invention. DETAILED DESCRIPTION

[0021] This paper proposes a new click rate prediction model DDNNFMFM, which can simultaneously learn explicit and implicit feature interactions and learn implicit feature interactions more efficiently by feature fusion. Figure 1 The following is a technical solution flow chart: Figure 2 The DDNNFMFM model diagram is shown. Combined with the accompanying drawings of the present invention, the following is a specific description of the technical solution of the present invention:

[0022] A click rate prediction method based on deep feature fusion includes the following steps:

[0023] S1. Get the historical data and pre-process the data to get the basic features. Extract user data from the user's historical click data. In actual production activities, the generated data may be missing or abnormal, and if used directly, it is easy to produce adverse results. Therefore, fill in the missing parts in the data set and normalize the numerical features.

[0024] S2. Feature vector construction as model training input. The basic features are converted into a sparse matrix after one-hot encoding, and then transformed into a dense matrix through some linear transformations. This dense matrix not only has a one-to-one correspondence with the original features, but also has a large number of intrinsic relationships between features. These intrinsic relationships are represented by the parameters learned by the embedding layer. Feature vector construction is to associate independent vectors and transform the data from high-dimensional sparse to low-dimensional dense features.

[0025] S3. Simultaneously learn the interaction between explicit and implicit features to obtain more implicit features. Based on the low-dimensional dense feature vectors in step S2, these feature vectors are used as the input of the feature interaction layer in the model to obtain multivariate cross features. These features are more abstract than the original features and can represent implicit features that are difficult to deduce from the original features. This can significantly improve the performance of click-through rate prediction. The feature interaction is divided into two parts:

[0026] S31, the embedding vector passes through the FmFM model, which uses the idea of ​​domain matrix to efficiently learn explicit feature interactions; the FmFM model is developed from the FwFM model, replacing the scalar describing the interaction intensity between different domain features in the latter with a two-dimensional matrix M, which improves the model's degree of freedom and expressiveness. Its formula is:

[0027]

[0028] Among them, x i , x j is the i-th feature and the j-th feature, v i , v j is the embedding vector of feature i and feature j, F(i), F(j) is the domain to which feature i and feature j belong, M F(i),F(j) That is, the weight matrix related to the domain where feature i and feature j are located, w i is the weight parameter and w0 is the bias term.

[0029] As shown in the attached figure of the instruction manual Figure 3 The process of the FmFM interaction term is illustrated, showing the interaction pair (v i , v j ) and (v i , v k ), and features i, j, and k come from three different domains. The calculation can be divided into three steps: 1) EmbeddingLookup: Look up v from the embedding table i , v j , v k Three embedding vectors; 2) Transformation: v i and the matrix M F(i),F(j) , M F(i),F(k) Calculate separately to obtain two sets of corresponding intermediate vectors v i , F(j) and v i , F(k); 3) Dotproduct: obtain the final interaction term through simple click calculation;

[0030] S32, the embedding vector passes through the DenseDNN network and uses the feature fusion method to learn implicit feature interactions. Figure 4 As shown in Figure 1, DenseDNN changes the input of each layer of the DNN network to the sum of the outputs of all previous layers. Its advantages are: a) solving the gradient vanishing problem of deep networks; b) strengthening feature propagation; c) reducing model parameters; d) reducing the problem of sample overfitting. The following formula is its output, where x i represents the output of the i-th layer, W (n) and b n is the training parameter. The output of each layer is:

[0031]

[0032] where x i represents the output of the i-th layer, W (n) and b n is the training parameter.

[0033] S4, get the output result, concatenate the results of the FmFM model and the DenseDNN network, and then pass the Sigmoid function to get the output result. The output result is used to judge the model accuracy through the binary cross entropy loss function Logloss and AUC (Area Under Curve).

[0034] Logloss defines a binary cross entropy loss function to evaluate the accuracy of the model. The performance of the model is evaluated by ranking all ads with expected CTR:

[0035]

[0036] Where N is the number of samples, for the i-th sample, y (i) is its true label, and y (i) ∈[0,1], the predicted probability is and

[0037] AUC is defined as the area under the ROC curve, with a value between 0 and 1. The size of AUC is positively correlated with the performance of the CTR prediction model. Before introducing AUC, the concept of confusion matrix is ​​introduced. In the confusion matrix, there are four concepts: true, false, positive, and negative. The correct prediction is true, the wrong is false, the prediction category is 1 is positive, and 0 is negative. The concepts that need to be explained are as follows:

[0038] (1) TP: The predicted sample is a positive sample, and the actual sample is also a positive sample.

[0039] (2) FP: The predicted sample is a positive sample, but the actual sample is a negative sample.

[0040] (3) FN: The predicted sample is a negative sample, but it is actually a positive sample.

[0041] (4) TN: predicted to be a negative sample, and actually also a negative sample.

[0042] (5) TPR: True positive rate, which is the ratio of TP to (TP+FN).

[0043] (6) FPR: false positive rate, which is the ratio of FP to (FP+TN).

[0044] The calculation steps of AUC are as follows: a) solve the values ​​of true positive rate and false positive rate through confusion matrix to obtain coordinate point pairs; b) the curve formed by different coordinate point pairs is the ROC curve; c) AUC is the area under the ROC curve;

[0045] When the AUC value is closer to 1.0, the authenticity of the method is higher and it is a close to perfect classifier; when the AUC value is equal to 0.5, it proves that the model has no predictive value; when the AUC value is less than 0.5, it proves that the effect is worse than random guessing. From another perspective, as long as we perform reverse prediction, the effect will be better than random guessing.

[0046] Finally, the present invention conducted sufficient comparative experiments based on two sets of classic advertising datasets: Criteo and Avazu.

[0047] Table 1 separately verifies the performance of the two models, DenseDNN and FwFM, in DDNNFMFM, and compares them with a single model with similar functions.

[0048] Table 1 Comparison of single element models on Criteo and Avazu datasets

[0049]

[0050] It can be seen that:

[0051] 1) Compared with the single-element model with explicit feature interaction, the FmFM model performs best in both datasets, proving that modeling the interaction between features in different domains as a matrix is ​​beneficial to improving model performance.

[0052] 2) Compared with DNN, the accuracy of the DenseDNN model proposed in this paper is improved by 0.29% and 0.70% in the AUC indicators and reduced by 0.57% and 0.84% ​​in the Criteo and Avazu datasets, respectively, which proves that DNN performs better after feature fusion.

[0053] Table 2 shows the performance results of the DDNNFMFM model and compares the DDNNFMFM model with several state-of-the-art models with similar structures.

[0054] Table 2 Comparison of comprehensive models on Criteo and Avazu datasets

[0055]

[0056] It can be seen that:

[0057] 1) The FmFM model is not only the shallow model with the best performance, but even in the Criteo dataset, its performance is better than the classic deep learning models DeepFM and xDeepFM, which once again proves the advantage of the domain matrix in the FmFM model. It is also lighter and faster than the two parallel models DeepFM and xDeepFM.

[0058] 2) The DDNNFMFM model performs best among all models based on embedded neural networks. In the Criteo and Avazu datasets, compared with the classic suboptimal model DeepLight, the AUC index of the DDNNFMFM model increased by 0.21% and 0.41% respectively, and the LogLoss index decreased by 0.36% and 0.35% respectively.

Claims

1. A click rate prediction method based on deep feature fusion, characterized by The steps include: S1. Get the historical data, heat encode the data into high-dimensional sparse features composed of 0 and 1, and obtain the basic features; S2, construct feature vectors of basic features through the embedding layer, and the constructed vectors are used as model training input; S3, through the shallow model FmFM and the deep model DenseDNN, learn explicit and implicit feature interactions at the same time; The specific process is: through the shallow model FmFM and the deep model DenseDNN, the explicit and implicit features are learned to interact and obtain more implicit features; Based on the low-dimensional dense feature vector, as the input of the feature interaction layer in the model, multivariate cross features are obtained. The feature interaction is divided into two parts: S31, the embedding vector passes through the FmFM model, using the idea of ​​domain matrix to efficiently learn explicit feature interactions; The FmFM model is developed from the FwFM model. The scalar describing the interaction strength between different domain features in the latter is replaced by a two-dimensional matrix M, which improves the degree of freedom and expression ability of the model. Its formula is: Among them, x i , x j is the i-th feature and the j-th feature, v i , v j is the embedding vector of feature i and feature j, F(i), F(j) is the domain to which feature i and feature j belong, M F(i),F(j) That is, the weight matrix related to the domain where feature i and feature j are located, w i is the weight parameter, w0 is the bias term; The process of FmFM interaction term is through the interaction pair (v i , v j ) and (v i , v k ) is calculated, and features i, j, and k come from three different domains respectively; it is divided into three steps: 1) EmbeddingLookup: Look up v from the embedding table i , v j , v k Three embedding vectors; 2) Transformation: v i and the matrix M F(i),F(j) , M F(i),F(k) Calculate separately to obtain two sets of corresponding intermediate vectors v i , F(j) and v i , F(k); 3) Dotproduct: obtain the final interaction term through simple click calculation; S32, the embedding vector passes through the DenseDNN network using the feature fusion method to learn implicit feature interactions; DenseDNN changes the input of each layer of the DNN network into the sum of the outputs of all previous layers, and the output of each layer is: where x i represents the output of the i-th layer, W (n) and b n is the training parameter; S4. After concatenating the results of the FmFM model and the DenseDNN network, the output result is obtained through the Sigmoid function, and the AUC and LogLoss indicators are used to judge the accuracy of the model prediction.

2. The method for predicting click rate based on deep feature fusion according to claim 1, characterized in that The specific process of the above step S1 is: get the historical data, pre-process the data to obtain basic features; extract user data from the user's historical click data, fill in the missing parts in the data set, and normalize the numerical features.

3. The method for predicting click rate based on deep feature fusion according to claim 2 is characterized in that The specific process of the above step S2 is as follows: constructing a feature vector as the input for model training; converting the basic features into a sparse matrix after one-hot encoding, and transforming it into a dense matrix through linear transformation, and using the parameters learned by the embedding layer to characterize the internal relationship of the dense matrix; constructing a feature vector is to associate independent vectors and convert the data from high-dimensional sparse to low-dimensional dense feature vectors.

4. The method for predicting click rate based on deep feature fusion according to claim 1, characterized in that The specific process of the above step S4 is: obtain the output result, concatenate the results of the FmFM model and the DenseDNN network, and then obtain the output result through the Sigmoid function; output the result and judge the model accuracy through the binary cross entropy loss function Logloss and AUC; Logloss defines a binary cross entropy loss function to evaluate the accuracy of the model. The performance of the model is evaluated by ranking all ads with expected CTR: Where N is the number of samples, for the i-th sample, y (i) is its true label, and y (i) ∈[0,1], the predicted probability is and AUC is defined as the area under the ROC curve, with a value between 0 and 1. The size of AUC is positively correlated with the performance of the CTR prediction model. Before introducing AUC, the concept of confusion matrix is ​​introduced. In the confusion matrix, there are four concepts: true, false, positive, and negative. The correct prediction is true, the wrong is false, the prediction category is 1 is positive, and the prediction category is 0 is negative. The concepts that need to be explained are as follows: (1) TP: predicted as a positive sample, and actually also a positive sample; (2) FP: The predicted sample is positive, but the actual sample is negative; (3) FN: predicted as a negative sample, but actually a positive sample; (4) TN: predicted as a negative sample, and actually also a negative sample; (5) TPR: True positive rate, which is the ratio of TP to (TP + FN); (6) FPR: false positive rate, which is the ratio of FP to (FP+TN); The calculation steps of AUC are as follows: a) solve the values ​​of true positive rate and false positive rate through confusion matrix to obtain coordinate point pairs; b) the curve formed by different coordinate point pairs is the ROC curve; c) AUC is the area under the ROC curve; When the AUC value is closer to 1.0, the authenticity of the method is higher and it is a close to perfect classifier; when the AUC value is equal to 0.5, it proves that the model has no predictive value; when the AUC value is less than 0.5, it proves that the effect is worse than random guessing.

Images