Deep cross-estimation model, method, device and storage medium for click rate

Abstract

The present invention discloses a deep cross-estimation model, method, device and storage medium of click-through rate. The deep cross-estimation model of click-through rate consists of an input layer (Input Layer), an FM layer, a DCA layer and an output layer (Output Layer). The input layer represents the input information with sparse high-dimensional features as a low-dimensional dense vector, and then sends it to the FM layer and the DCA layer respectively; the FM layer cross-mines low-order cross-features that never appear or rarely appear in the training data through first-order features and second-order features; the DCA layer uses a scaled dot product attention mechanism to design a cross-attention layer, distinguishes the importance of high-order cross-features, and designs a deep neural network to achieve modeling of high-order cross-features; the output layer is responsible for outputting the estimation result. The parallel structure of the present invention can effectively learn low-order and high-order cross-features at the same time, can distinguish the importance of different high-order cross-features and eliminate redundant cross-features, improve the estimation accuracy of the click-through rate, and is of great significance for screening information that meets user needs from massive information.

Description

Technical Field

[0001] The present invention belongs to the technical field of information recommendation, and relates to a deep cross-estimation model, method, device and storage medium of click rate, which are used to screen out information that meets user needs from massive information. Background Art

[0002] With the rapid development of computer technology, the Internet has generated a large amount of information, but the problem of information overload has also emerged. How to filter out information that meets user needs from massive information is a hot topic, and personalized recommendations have become very important. Click-through rate estimation is an important part of the recommendation system. Its purpose is to predict the probability that a user will respond to a given item, such as predicting the probability that a user will click on an advertisement or purchase a product.

[0003] The early solution to the problem of click-through rate prediction was simply to use a linear model to weight the sum of different features. Each feature was used separately without considering the interaction between features, such as logistic regression (LR) (see "NUSINOVICIS, THAM YC, YAN MYC, et al. Logistic regression was as good as machine learning for predicting major chronic diseases [J]. Journal of clinical epidemiology, 2020, 122: 56-69."), SGD regression, etc. The advantages of the LR model are that the model is simple and interpretable, but the disadvantage is that it can only perform simple feature weighting and cannot perform feature crossover. TAOZ et al. used the FM model for click-through rate prediction. FM takes into account the combination of second-order features, and the performance of the model is better than that of the linear model.

[0004] With the rapid development of deep learning, applying deep learning to the field of click-through rate prediction is a research trend. Scholars have begun to use deep neural networks (DNNs) to model more expressive high-order cross features. For example, KIM et al. used the Wide&Deep model that combines the LR model with memory capabilities and the DNN model with pan-China capabilities to analyze input data. GUO et al. from Huawei Noah's Ark Laboratory replaced the FM that performs feature cross-pollination with the linear part in the Wide&Deep model and proposed the DeepFM (deep factorization machines) model. Later, in order to enhance the model feature crossing capability, the DCN model that can perform feature crossing and linearly increase the network complexity, and the xDeepFM model that retains the advantages of Cross and introduces the Field vector-wise idea (see "LIAN J, ZHOU X, ZHANG F, et al. xdeepfm: Combining explicit and implicit feature interaction for recommender systems [C] / / Proceedings of the 24th ACM SIGKDD international conference on on knowledge discovery & data mining. 2018: 1754-1763.") were proposed one after another. Inspired by the Transformer in the NLP field, Song et al. (see "SONG W, SHI C, XIAO Z, et al. Autoint: Automatic feature interaction learning via self-attentive neural networks [C] / / Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019: 1161-1170") designed a multi-head self-attention mechanism Interacting Layer with residual connections, which explicitly acquires cross-features and realizes automatic learning of high-order feature cross-features. Feng et al. also directly applied this multi-head self-attention mechanism in DSIN for click-through rate estimation.

[0005] Although the above click rate estimation method has achieved good results, it has the following problems:

[0006] (1) The FM model uses one-hot encoding for categorical features, but performs poorly on numerical features. It only learns second-order cross features and does not consider high-order feature crosses, resulting in the loss of some correlation information between features.

[0007] (2) Realizing complex feature crossover solely through the superimposed attention mechanism may limit the performance of the model. Using only DNN to perform implicit feature crossover on high-order features may introduce redundant information. The noise caused by redundant information will increase the difficulty of model training and reduce the learning efficiency in feature crossover. There is no denoising module.

[0008] In order to solve the above problems existing in the model and achieve high-precision prediction of click-through rate, the present invention proposes a deep cross-attention network (DAPN). It integrates the scaled dot product attention mechanism and multi-layer perceptron into the Cross Network architecture to achieve effective learning of high-order nonlinear cross features. At the same time, it uses FM to cross-model low-order features in pairs, and fully learns first-order and second-order cross features. DAPN uses a parallel structure to share the same input, which improves model performance while reducing the cost of manually designed feature engineering, and obtains better prediction results. Summary of the invention

[0009] In order to solve the above problems, the present invention provides a deep cross-estimation model of click-through rate, which can distinguish the importance of different high-order cross-features and eliminate redundant cross-features, thereby improving the estimation accuracy of click-through rate, which is of great significance for screening out information that meets user needs from massive information.

[0010] The second objective of the present invention is to provide a deep cross-estimation method for click-through rate.

[0011] A third object of the present invention is to provide an electronic device.

[0012] A fourth object of the present invention is to provide a computer storage medium.

[0013] The technical solution adopted by the present invention is a deep cross-estimation model of click rate, comprising: an input layer (Input Layer), an FM layer, a DCA layer and an output layer (Output Layer);

[0014] The input layer represents the input information with sparse high-dimensional features as a low-dimensional dense vector and then sends it to the FM and DCA layers respectively;

[0015] The FM layer cross-mines low-order cross features that never appear or rarely appear in the training data through first-order features and second-order features;

[0016] The DCA layer uses the scaled dot product attention mechanism to design a cross attention layer, distinguishes the importance of high-order cross features, and designs a deep neural network to model high-order cross features;

[0017] The output layer is responsible for outputting the estimated results.

[0018] Furthermore, the input layer is the first stage of the model, responsible for representing the sparse high-dimensional features of the input as low-dimensional dense Embedding vectors as shared inputs of the FM layer and the DCA layer;

[0019] The input features in the input layer are mostly sparse category features (Sparse Feature), and the input layer is responsible for processing, encoding and dimensionality reduction of input data;

[0020] The data processing, encoding and dimensionality reduction process is to first divide the features of the same type into the same domain (field), then use one-hot encoding to convert the categorical features into sparse high-dimensional features (SparseFeature) represented in binary form, and then use Embedding encoding to convert the high-dimensional sparse feature vector into a low-dimensional dense real-number Embedding vector.

[0021] Furthermore, the FM layer is a factorization machine for learning feature interactions;

[0022] The input of the FM layer is the Embedding vector of all features;

[0023] In addition to using the linear relationship between features as the first-order feature intersection, the FM layer also uses the inner product of pairwise feature vectors as the second-order feature intersection, so as to better learn the feature intersections that never appear or rarely appear in the training data. In order to reduce the number of feature intersection weights, matrix decomposition is used to decompose the feature intersection weight matrix into two vectors.

[0024] Furthermore, the DCA layer explores more explicit high-order feature interactions by deepening the attention network and designs a cross-attention layer using the scaled dot-product attention mechanism;

[0025] The cross attention layer, the output of the lth layer is used as the input of the l+1th layer and the feature vector Z is obtained by the scaled dot product attention mechanism. (l+1) , and then do the inner product calculation with the embedding vector of the input layer, and finally reduce the dimension by average pooling to get the output result X of the l+1th layer (l+1) ;

[0026] The cross attention layer uses a connection layer to concatenate the outputs of all cross attention layers after deep feature crossover, and then inputs them into a multi-layer perceptron to further explore implicit feature crossover;

[0027] The input of the DCA layer is the same as the input of the FM layer, which is the Embedding vector of all features.

[0028] Furthermore, the output layer converts the low-order feature information y obtained by the FM layer into FM And the high-order combined feature information y output by the DCA layer DCA Add the sum and then input it into the Sigmoid function to calculate the output result.

[0029] A deep cross-estimation method for click rate, image enhancement is performed in the following steps:

[0030] Step S1: The information of the click rate to be estimated is sent to the deep cross-estimation model of the click rate. The input layer represents the input high-dimensional sparse feature (Sparse Feature) as a low-dimensional dense Embedding vector;

[0031] Step S2, the FM layer uses the linear relationship between features of the Embedding vector provided by the input layer as the first-order feature cross, and uses the inner product of each pair of feature vectors as the second-order feature cross;

[0032] Step S3: The DCA layer obtains the feature vector by scaling the dot product attention mechanism on the input of the cross attention layer, performs inner product calculation on the embedding vector provided by the input layer, and then realizes dimensionality reduction through average pooling.

[0033] Step S4: The output layer adds the low-order feature information output by the FM layer and the high-order combined feature information output by the DCA layer, and calculates the estimated result using the Sigmoid function.

[0034] Furthermore, in the input layer, all feature vectors of the same domain share the same encoding matrix, and its mathematical model is:

[0035] e embed,j =W embed,j x j

[0036] Where: e embed,j ∈R d*m (j=1,2,...,k) is the embedding vector corresponding to the jth domain, x j ∈R v*m is the one-hot vector of all features in the jth domain, W embed,j ∈R d*vis the Embedding encoding matrix corresponding to the jth domain, v is the number of words, d is the Embedding vector dimension, and m is the number of feature vectors in the jth domain;

[0037] In the input layer, the output after Embedding encoding is the set of Embedding vectors of all domains, expressed as,

[0038] E=[e embed,1 ,e embed,2 ,...,e embed,k ]

[0039] Where: E∈R d*n is the set of Embedding vectors of all domains, e embed,i ∈R d*m (i=1,2,...,k) represents the embedding vector of the i-th domain, d is the embedding vector dimension, n is the number of all input features, m is the number of features in the i-th domain, and k is the number of feature domains;

[0040] The mathematical model of the FM layer is,

[0041]

[0042] Where: w0∈R d is the global bias of the model, w i ∈R d is the weight of the i-th feature, n is the number of all features, x embed,i ,x embed,j ∈R d is the embedding vector of the i-th feature and the j-th feature, V i ,V j is the potential feature vector of the i-th feature and the j-th feature,<a,b> Indicates the inner product operation of vectors a and b. <V i ,V j > is the cross weight parameter between the i-th feature and the j-th feature, and d is the Embedding vector dimension;

[0043] The mathematical model of the cross attention network of the DCA layer is,

[0044] y DCA =MLP(Concat(X (1) ,X (2) ,...,X (l+1) ))

[0045] Z (l+1) =DP-Att(X (l) W Q ,X(l) W K ,X (l) W V )

[0046]

[0047] Where: X (i) ∈R d*n (i=1,2,...,l+1) is the output of the i-th cross attention layer, Z (i) ∈R d*n (i=1,2,...,l+1) is the output of the scaled dot product attention mechanism in the i-th cross attention layer, E∈R d*n is the input Embedding feature vector, W Q ∈R d*dq is the query transformation matrix, W K ∈R d*dk is the keyword transformation matrix, W V ∈R d*dv is the value transformation matrix, set dq = dk = dv, d is the Embedding vector dimension, and n is the number of all input features;

[0048] The mathematical model of the output layer is:

[0049]

[0050] Sigmoid(Y)=1(1+exp(-Y))

[0051] Where:y FM is the output of the FM layer, y DCA is the output of the DCA layer.

[0052] Furthermore, the deep cross-estimation method of the click-through rate must be implemented with the help of a deep cross-estimation model of the click-through rate trained under the constraint of the loss function;

[0053] The loss function is used to measure the gap between the model prediction value and the actual value, and the calculation formula is,

[0054]

[0055] Where: It represents the model's predicted user click probability, y represents the user's actual click result, which is 0 or 1, and N is the total number of training samples.

[0056] An electronic device, characterized in that the method described above is used to achieve the prediction of click rate.

[0057] A computer storage medium, characterized in that at least one program instruction is stored in the storage medium, and the at least one program instruction is loaded and executed by a processor to implement the click rate estimation method as described above.

[0058] The beneficial effect of the embodiment of the present invention is that it integrates the scaled dot product attention mechanism and the multi-layer perceptron into the Cross Network architecture to achieve effective learning of high-order nonlinear cross features. At the same time, it uses FM to perform pairwise cross modeling of low-order features to fully learn the first-order and second-order cross features. DAPN uses a parallel structure to share the same input, which improves model performance while reducing the cost of manually designed feature engineering, and obtains better prediction results. The present invention can distinguish the importance of different high-order cross features and eliminate redundant cross features, thereby improving the accuracy of click-through rate estimation, which is of great significance for screening information that meets user needs from massive information. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a structural diagram of a deep cross-estimation model of click rate according to an embodiment of the present invention.

[0061] Figure 2 It is a structural schematic diagram of a cross-attention layer in a DCA layer of a deep cross-estimation model of click rate according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Embodiment 1,

[0064] A deep cross-estimation model of click rate, whose structure is as follows Figure 1 As shown, it includes: input layer (InputLayer), FM layer, DCA layer and output layer (Output Layer);

[0065] The input layer is the first stage of the model, which is responsible for representing the input sparse high-dimensional features as low-dimensional dense Embedding vectors as shared inputs for the FM layer and the DCA layer. Most of the input features in the input layer are sparse categorical features (Sparse Feature). The input layer is responsible for processing, encoding, and dimensionality reduction of the input data. The data processing, encoding, and dimensionality reduction process is to first divide the features of the same type into the same domain (field), then use one-hot encoding to convert the categorical features into sparse high-dimensional features (Sparse Feature) represented in binary, and then use Embedding encoding to convert the high-dimensional sparse feature vectors into low-dimensional dense real Embedding vectors.

[0066] The FM layer is a factor decomposition machine used to learn feature interactions. It mines low-order cross features that never appear or rarely appear in the training data through the intersection of first-order features and second-order features. The input of the FM layer is the Embedding vector of all features. In addition to using the linear relationship between features as the first-order feature intersection, the FM layer also uses the inner product of two feature vectors as the second-order feature intersection to better learn feature intersections that never appear or rarely appear in the training data. In order to reduce the number of feature intersection weights, matrix decomposition is used to decompose the feature intersection weight matrix into two vectors.

[0067] like Figure 2 As shown,

[0068] The DCA layer uses the scaled dot product attention mechanism to design the cross attention layer, distinguishes the importance of high-order cross features, and designs a deep neural network to model high-order cross features; the DCA layer explores more explicit high-order feature intersections by deepening the attention network, and uses the scaled dot product attention mechanism to design the cross attention layer;

[0069] Cross attention layer, the output of the lth layer is used as the input of the l+1th layer through the scaled dot product attention mechanism to obtain the feature vector Z (l+1) , and then do the inner product calculation with the embedding vector of the input layer, and finally reduce the dimension by average pooling to get the output result X of the l+1th layer (l+1) ; The cross-attention layer uses a connection layer to concatenate the outputs of all cross-attention layers after deep feature cross-attention, and then inputs them into the multi-layer perceptron to further explore implicit feature cross-attention;

[0070] The input of the DCA layer is the same as the input of the FM layer, which is the Embedding vector of all features.

[0071] The output layer is responsible for outputting the estimated results. It converts the low-order feature information y obtained by the FM layer into FMAnd the high-order combined feature information y output by the DCA layer DCA Add the sum and then input it into the Sigmoid function to calculate the output result.

[0072] Embodiment 2,

[0073] A deep cross-estimation method for click rate, image enhancement is performed in the following steps:

[0074] Step S1: The information of the click rate to be estimated is sent to the deep cross-estimation model of the click rate. The input layer represents the input high-dimensional sparse feature (Sparse Feature) as a low-dimensional dense Embedding vector;

[0075] In the input layer, all feature vectors of the same domain share the same encoding matrix, and its mathematical model is:

[0076] e embed,j =W embed,j x j

[0077] Where: e embed,j ∈R d*m (j=1,2,...,k) is the embedding vector corresponding to the jth domain, x j ∈R v*m is the one-hot vector of all features in the jth domain, W embed,j ∈R d*v is the Embedding encoding matrix corresponding to the jth domain, v is the number of words, d is the Embedding vector dimension, and m is the number of feature vectors in the jth domain;

[0078] In the input layer, the output after Embedding encoding is the set of Embedding vectors of all domains, expressed as,

[0079] E=[e embed,1 ,e embed ,2,...,e embed,k ]

[0080] Where: E∈R d*n is the set of Embedding vectors of all domains, e embed,i ∈R d*m (i=1,2,...,k) represents the embedding vector of the i-th domain, d is the embedding vector dimension, n is the number of all input features, m is the number of features in the i-th domain, and k is the number of feature domains;

[0081] Step S2, the FM layer uses the linear relationship between features of the Embedding vector provided by the input layer as the first-order feature cross, and uses the inner product of each pair of feature vectors as the second-order feature cross;

[0082] The mathematical model of the FM layer is:

[0083]

[0084] Where: w0∈R d is the global bias of the model, w i ∈R d is the weight of the i-th feature, n is the number of all features, x embed,i ,x embed,j ∈R d is the embedding vector of the i-th feature and the j-th feature, V i ,V j is the potential feature vector of the i-th feature and the j-th feature,<a,b> Indicates the inner product operation of vectors a and b. <V i ,V j > is the cross weight parameter between the i-th feature and the j-th feature, and d is the Embedding vector dimension;

[0085] Step S3: The DCA layer obtains the feature vector by scaling the dot product attention mechanism on the input of the cross attention layer, performs inner product calculation on the embedding vector provided by the input layer, and then realizes dimensionality reduction through average pooling.

[0086] The mathematical model of the cross attention network of the DCA layer is,

[0087] y DCA =MLP(Concat(X (1) ,X (2) ,...,X (l+1) ))

[0088] Z (l+1) =DP-Att(X (l) W Q ,X (l) W K ,X (l) W V )

[0089]

[0090] Where: X (i) ∈R d*n (i=1,2,...,l+1) is the output of the i-th cross attention layer, Z (i) ∈R d*n(i=1,2,...,l+1) is the output of the scaled dot product attention mechanism in the i-th cross attention layer, E∈R d*n is the input Embedding feature vector, W Q ∈R d*dq is the query transformation matrix, W K ∈R d*dk is the keyword transformation matrix, W V ∈R d*dv is the value transformation matrix, set dq = dk = dv, d is the Embedding vector dimension, and n is the number of all input features;

[0091] Step S4: The output layer adds the low-order feature information output by the FM layer and the high-order combined feature information output by the DCA layer, and calculates the estimated result using the Sigmoid function.

[0092] The mathematical model of the output layer is:

[0093]

[0094] Sigmoid(Y)=1(1+exp(-Y))

[0095] Where:y FM is the output of the FM layer, y DCA is the output of the DCA layer.

[0096] The deep cross-estimation method of click-through rate must be implemented with the help of the deep cross-estimation model of click-through rate trained under the constraint of loss function;

[0097] The loss function is used to measure the gap between the model's predicted value and the actual value, and the calculation formula is,

[0098]

[0099] Where: It represents the model's predicted user click probability, y represents the user's actual click result, which is 0 or 1, and N is the total number of training samples.

[0100] In order to verify the effectiveness of the deep cross-attention model DAPN of the click-through rate of the embodiment of the present invention, five click-through rate prediction models, including LR, DCN, AutoInt, PNN (see "QU Y, FANG B, ZHANG W, et al. Product-based neural networks for user response prediction over multi-field categorical data [J]. ACM Transactions on Information Systems, 2018, 37 (1): 1-35"), and DCAP (see "CHEN ZK, ZHONG FT, CHEN ZM, et al. DCAP: Deep Cross Attentional Product Network for User Response Prediction [C] / / Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021: 221-230"), are selected as baselines for comparison with the DAPN model.

[0101] The experiments were conducted using two public datasets, MovieLens-1m and Avazu. The MovieLens-1m dataset contains 1 million user data from 6,000 users on 4,000 movies, and the Avazu dataset contains 40 million real click log data. The datasets were randomly shuffled and divided into training dataset and test dataset in a ratio of 9:1. AUC and Logloss were used as evaluation indicators of the model. The smaller the Logloss value, the higher the accuracy of the model, and the larger the AUC value, the better the performance of the click rate prediction model.

[0102] To ensure the stability and reliability of the experimental results, the final result is the average of 10 training results. The AUC index and Logloss loss of each model on the MovieLens-1m and Avazu datasets are shown in Table 1.

[0103] Table 1 Performance of each model on two public datasets

[0104]

[0105] As can be seen from Table 1, the LR model that only considers the linear relationship between features performs the worst compared to other models, indicating that there is a mutual influence between features, and appropriate cross-combination of features can improve model performance. On two public data sets, the AUC index of the DAPN model of the present invention is improved by 0.16% and 0.25% respectively compared with the LR model, and the Logloss index is improved by 0.62% and 1.6%.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A deep cross-estimation model of click-through rate, characterized in that: include: Input layer, FM layer, DCA layer and output layer; The input layer represents the input information with sparse high-dimensional features as a low-dimensional dense vector and then sends it to the FM and DCA layers respectively; The FM layer cross-mines low-order cross features that never appear or rarely appear in the training data through first-order features and second-order features; The DCA layer uses the scaled dot product attention mechanism to design a cross attention layer, distinguishes the importance of high-order cross features, and designs a deep neural network to model high-order cross features; The DCA layer explores more explicit high-order feature crosses by deepening the attention network and designs a cross-attention layer using the scaled dot product attention mechanism; The cross attention layer, the output of the lth layer is used as the input of the l+1th layer and the feature vector Z is obtained by the scaled dot product attention mechanism. (l+1) , and then do the inner product calculation with the embedding vector of the input layer, and finally reduce the dimension by average pooling to get the output result X of the l+1th layer (l+1) ; The cross attention layer uses a connection layer to concatenate the outputs of all cross attention layers after deep feature crossover, and then inputs them into a multi-layer perceptron to further explore implicit feature crossover; The input of the DCA layer is the same as the input of the FM layer, which is the Embedding vector of all features; The output layer converts the low-order feature information y obtained by the FM layer into FM And the high-order combined feature information y output by the DCA layer DCA Add the sum and then input it into the Sigmoid function to calculate the output result; The output layer is responsible for outputting the estimated results.

2. A deep cross-estimation model of click-through rate according to claim 1, characterized in that: The input layer is the first stage of the model, responsible for representing the sparse high-dimensional features of the input as low-dimensional dense Embedding vectors as shared inputs for the FM layer and the DCA layer. The input features in the input layer are mostly sparse category features (Sparse Feature), and the input layer is responsible for processing, encoding and dimensionality reduction of input data; The data processing, encoding and dimensionality reduction process is to first divide the features of the same type into the same field, then use one-hot encoding to convert the categorical features into sparse high-dimensional features (Sparse Feature) represented in binary form, and then use Embedding encoding to convert the high-dimensional sparse feature vector into a low-dimensional dense real-number Embedding vector.

3. The deep cross-estimation model of click-through rate according to claim 1, characterized in that: The FM layer is a factorization machine used to learn feature interactions; The input of the FM layer is the Embedding vector of all features; In addition to using the linear relationship between features as the first-order feature intersection, the FM layer also uses the inner product of pairwise feature vectors as the second-order feature intersection, so as to better learn the feature intersections that never appear or rarely appear in the training data. In order to reduce the number of feature intersection weights, matrix decomposition is used to decompose the feature intersection weight matrix into two vectors.

4. A deep cross-estimation method for click-through rate, characterized in that: To perform image enhancement, follow these steps: Step S1: The information of the click rate to be estimated is sent to the deep cross-estimation model of the click rate. The input layer represents the input high-dimensional sparse feature (Sparse Feature) as a low-dimensional dense Embedding vector; In the input layer, all feature vectors of the same domain share the same encoding matrix, and its mathematical model is: have been embed,j =W embed,j x j Where: e embed,j ∈R d*m (j=1,2,...,k) is the embedding vector corresponding to the jth domain, x j ∈R v*m is the one-hot vector of all features in the jth domain, W embed,j ∈R d*v is the Embedding encoding matrix corresponding to the jth domain, v is the number of words, d is the Embedding vector dimension, and m is the number of feature vectors in the jth domain; In the input layer, the output after Embedding encoding is the set of Embedding vectors of all domains, expressed as, And=[and embed,1 ,And embed,2 ,...,And embed,k ] Where: E∈R d*n is the set of Embedding vectors of all domains, e embed,i ∈R d*m (i=1,2,...,k) represents the embedding vector of the ith domain, d is the embedding vector dimension, n is the number of all input features, m is the number of features in the ith domain, and k is the number of feature domains; Step S2, the FM layer uses the linear relationship between features of the Embedding vector provided by the input layer as the first-order feature cross, and uses the inner product of each pair of feature vectors as the second-order feature cross; The mathematical model of the FM layer is, Where: w0∈R d is the global bias of the model, w i ∈R d is the weight of the i-th feature, n is the number of all features, x embed,i ,x embed,j ∈R d is the embedding vector of the i-th feature and the j-th feature, V i ,V j is the potential feature vector of the i-th feature and the j-th feature,<a,b> Indicates the inner product operation of vectors a and b. <V i ,V j > is the cross weight parameter between the i-th feature and the j-th feature, and d is the Embedding vector dimension; Step S3: The DCA layer obtains the feature vector by scaling the dot product attention mechanism for the input of the cross attention layer, performs inner product calculation with the embedding vector provided by the input layer, and then realizes dimensionality reduction through average pooling. The mathematical model of the cross attention network of the DCA layer is, y DCA =MLP(Concat(X (1) ,X (2) ,...,X (l+1) )) Z (l+1) =DP_Att(X (l) W Q ,X (l) W K ,X (l) W V ) Where: X (i) ∈R d*n (i=1,2,...,l+1) is the output of the i-th cross attention layer, Z (i) ∈R d*n (i=1,2,...,l+1) is the output of the scaled dot product attention mechanism in the i-th cross attention layer, E∈R d*n is the input Embedding feature vector, W Q ∈R d*dq is the query transformation matrix, W K ∈R d*dk is the keyword transformation matrix, W V ∈R d*dv is the value transformation matrix, set dq = dk = dv, d is the Embedding vector dimension, and n is the number of all input features; Step S4: the output layer adds the low-order feature information output by the FM layer and the high-order combined feature information output by the DCA layer, and calculates the estimated result using the Sigmoid function; The mathematical model of the output layer is: Sigmoid(Y)=1 / (1+exp(-Y)) Where:y FM is the output of the FM layer, y DCA is the output of the DCA layer.

5. A method for deep cross-estimation of click rate according to claim 4, characterized in that: The deep cross prediction method of the click rate must be implemented with the help of a deep cross prediction model of the click rate trained under the constraint of the loss function; The loss function is used to measure the gap between the model prediction value and the actual value, and the calculation formula is, Where: It represents the model's predicted user click probability, y represents the user's actual click result, which is 0 or 1, and N is the total number of training samples.

6. An electronic device, characterized in that: The click rate is estimated by using the method described in any one of claims 4-5.

7. A computer storage medium, characterized in that: The storage medium stores at least one program instruction, and the at least one program instruction is loaded and executed by the processor to implement the click rate prediction method as described in any one of claims 4-5.

Images