A click-through rate prediction method based on multi-attention mechanism fusion feature enhancement

Abstract

The present invention discloses a click rate prediction method based on multi-attention mechanism fusion feature enhancement, belonging to the field of machine learning technology, the method comprises: obtaining historical click instance sample data, and preprocessing the obtained data to construct a sample data set; wherein the click instance sample data comprises sample feature information and label value; combining multi-head self-attention mechanism and SENet network to construct a click rate prediction model; using the sample data set to train the constructed click rate prediction model; using the trained click rate prediction model to predict the click rate; regularly evaluating the performance of the click rate prediction model, by comparing the actual click situation with the model click rate prediction result, adjusting the model parameters or structure, so as to continuously optimize the prediction accuracy. The present invention can significantly improve the accuracy and efficiency of click rate prediction in online advertising and recommendation systems.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a click rate prediction method based on multi-attention mechanism fusion feature enhancement. Background Art

[0002] Click-through rate (CTR) refers to the ratio of the number of times an ad is clicked to the number of times it is displayed, and is used to measure the popularity of an ad. A good CTR prediction algorithm can capture the user's interest points at any time and accurately recommend the items or information that the user is most interested in at the moment, meet user needs, and improve the quality and efficiency of item recommendations.

[0003] Traditional CTR prediction methods, such as logistic regression and factorization machines (FM), have achieved certain results in the early stage, but they have limitations in dealing with complex interactions between features in high-dimensional sparse data. In recent years, deep learning methods have been widely used in click-through rate prediction due to their powerful feature expression capabilities. For example, Wide&Deep, DeepFM, DCN (Deep&Cross Network) and other models have explored different feature interaction methods by combining shallow models with deep models and achieved good results. However, the feature representations in these models are fixed, and the importance of different features in the model is not considered, which limits the expression ability of the model. To address this problem, some researchers have begun to consider using attention mechanisms to improve the click-through rate prediction effect. For example, IFM (Input-aware Factorization Machine), DIFM (Dual Input-aware Factorization Machine), FiBiNet (Feature Importance and Bilinear feature Interaction NETwork), FiBiNet++, etc. The FRNet (Feature Refinement Network) model proposed by Fudan University in 2022 learns the context-aware information of features and adjusts the feature weights in different instances based on the correlation information between features and the complete context information. In general, click-through rate prediction models based on attention mechanisms have become a research hotspot and cutting-edge direction, and various models continue to emerge. These methods have made certain breakthroughs in improving the expressiveness and performance of the models.

[0004] Early CTR models mainly considered low-order interactions between features. For example, the FM model models the feature interaction part by factorization, mapping high-dimensional sparse features to low-dimensional dense feature space. However, the FM model can only capture second-order feature interactions and has limited modeling capabilities for high-order feature interactions. To overcome this limitation, Google proposed the Wide&Deep model in 2016, which combines linear models and deep models to simultaneously consider shallow and deep feature interactions. Specifically, the Wide part uses a generalized linear model with cross terms, and the Deep part uses a multi-layer neural network. This model performs well in the field of CTR prediction. Stanford University and Google proposed the DCN (Deep&CrossNetwork) model in 2017, which adds a Cross Network layer to Wide&Deep to better learn the cross relationship between features. The Cross Network layer crosses the input features pairwise and performs multi-level feature cross learning, allowing the model to better capture the nonlinear relationship between features. Also in 2017, Guo et al. proposed the DeepFM model, which combines the advantages of FM and deep neural networks (DNN). FM can learn the second-order interactions between features, while deep neural networks can learn high-order interactions. The combination of these two methods can better learn the cross-relationship between features.

[0005] Traditional CTR models usually treat each feature as equivalent, ignoring the differences in the importance of different features. To solve this problem, some models use attention mechanisms to strengthen features. For example, the IFM model (Input-aware Factorization Machine) proposed by Tsinghua University introduces an interactive perception mechanism to effectively learn the importance information of features in different scenarios. Based on IFM, the DIFM (Dual Input-aware Factorization Machine) model introduces Transformer to learn vector-level interaction information and combines it with bit-level feature representation to learn feature weight information. The FiBiNet (Feature Importanceand Bilinear feature Interaction NETwork) proposed by Sina's team in 2019 and the improved version FiBiNet++ model proposed in 2022 learn feature importance information by introducing SENet (Squeeze-and-Excitation Networks) modules. These models are able to learn fixed weight representations at the bit level or vector level and perform linear weight assignments, but none of them take into account the differences in the importance of each feature in different contexts for adjustment.

[0006] The FRNet model uses a different approach to adjust weights by building contextual information of features so that the same feature has different representations in different instances. In addition, FRNet also designs a complementary gating unit that combines the original feature representation with the enhanced feature representation. This method enhances the expressive power of the input features and improves the effectiveness of the model feature representation while retaining the original feature expression. However, there are two problems with this approach. First, the attention representation is too single, which limits the expressive power of the features. Second, the lack of adjustable contextual information representation makes it impossible to meet the needs of various complex scenarios when mining contextual information.

[0007] In summary, the current CTR prediction model is not sufficient in feature expression, cannot handle the complex interactions between features, and lacks adjustable contextual information mining capabilities, which greatly limits the accuracy of click-through rate prediction. Summary of the invention

[0008] The present invention provides a click-through rate prediction method based on multi-attention mechanism fusion feature enhancement to solve the technical problems that the current CTR prediction model is not sufficient in feature expression, cannot handle complex interactions between features, and lacks adjustable context information mining capabilities, which limits the accuracy of click-through rate prediction.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0010] On the one hand, the present invention provides a click rate prediction method based on multi-attention mechanism fusion feature enhancement, and the click rate prediction method based on multi-attention mechanism fusion feature enhancement comprises:

[0011] Acquire historical click instance sample data, and preprocess the historical click instance sample data to construct a sample data set; wherein the click instance sample data includes sample feature information and label values;

[0012] Combining the multi-head self-attention mechanism and SENet network, a click-through rate prediction model is constructed;

[0013] Using the sample data set to train the constructed click rate prediction model;

[0014] Use the trained click rate prediction model to predict click rate;

[0015] Regularly evaluate the performance of the click-through rate prediction model, and adjust the model parameters or structure by comparing the actual click situation with the click-through rate prediction results output by the model to continuously optimize the prediction accuracy.

[0016] Furthermore, the historical click instance sample data is preprocessed to construct a sample data set, including:

[0017] Preprocessing the acquired historical click instance sample data; wherein the preprocessing process includes: cleaning the acquired historical click instance sample data; normalizing the numerical features in the cleaned historical click instance sample data;

[0018] According to the ratio of 8:1:1, the preprocessed historical click instance sample data is divided into a training set, a validation set and a test set; wherein the training set is used to train the click rate prediction model; the validation set is used to verify the trained model; and the test set is used to test the trained model.

[0019] Furthermore, the click rate prediction model includes: a feature input layer, a feature embedding layer, a feature enhancement layer, a feature interaction layer and a prediction layer; wherein,

[0020] The feature input layer is used to process the sample feature information input into the click rate prediction model to obtain a high-dimensional sparse feature vector, and input the high-dimensional sparse feature vector into the feature embedding layer;

[0021] The feature embedding layer is used to convert the high-dimensional sparse feature vector into a low-dimensional dense vector as original feature information, and input the original feature information into the feature enhancement layer;

[0022] The feature enhancement layer is used to enhance the original feature information to obtain enhanced feature information, and input the enhanced feature information into the feature interaction layer;

[0023] The feature interaction layer is used to obtain low-order interaction information and high-order interaction information between enhanced feature information output by the feature enhancement layer, and input the low-order interaction information and high-order interaction information into the prediction layer;

[0024] The prediction layer is used to combine the low-order interaction information and the high-order interaction information output by the feature interaction layer, perform final click-through rate prediction, and output the click-through rate prediction result.

[0025] Furthermore, the feature input layer processes the sample feature information by:

[0026] One-hot encoding is performed on the categorical features in the sample feature information.

[0027] Furthermore, the feature enhancement layer includes a first information extraction unit, a second information extraction unit and a gating unit; wherein,

[0028] The first information extraction unit takes the original feature information as input, is used to obtain the preliminary enhanced feature corresponding to the original feature information, and inputs the preliminary enhanced feature into the gating unit;

[0029] The second information extraction unit takes the original feature information as input, is used to obtain a gating matrix corresponding to the original feature information, and inputs the gating matrix into the gating unit;

[0030] The gating unit takes the preliminary enhanced feature, the gating matrix and the initial feature information as inputs, and is used to adaptively combine the original feature information with the preliminary enhanced feature based on the gating matrix using a complementary selection gate unit to obtain enhanced feature information.

[0031] Furthermore, the information extraction unit includes: a multi-head self-attention module, a SENet network, a context information extraction module and an integration module; wherein,

[0032] The multi-head self-attention module takes the original feature information as input, and the processing result of the original feature information is input into the integration module; the SENet network takes the original feature information as input, and the processing result of the original feature information is input into the context information extraction module; the context information extraction module is a multi-layer perceptron network, and the output of the multi-layer perceptron network is input into the integration module, and the output of the integration module is input into the gating unit;

[0033] The output of the integration module is represented as follows:

[0034] O=O veC ☉O bit

[0035] Wherein, O represents the output of the integrated module; vec represents the output of the multi-head self-attention module; bit represents the output of the context information extraction module; ⊙ represents the element-wise product of two vectors.

[0036] Furthermore, the calculation formula of the enhanced feature information is as follows:

[0037] E r =E⊙σ(W b )+E rp ⊙(1-σ(W b ))

[0038] Among them, E r represents enhanced feature information; E represents original feature information; E rp represents the preliminary enhanced features output by the first information extraction unit; σ represents the sigmoid activation function, W b Represents the gating matrix output by the second information extraction unit.

[0039] Furthermore, the feature interaction layer includes a FM model and a DNN model; wherein,

[0040] The FM model uses the enhanced feature information output by the feature enhancement layer as input, is used to obtain low-order interaction information between the enhanced feature information, and inputs the low-order interaction information into the prediction layer;

[0041] The DNN model uses the enhanced feature information output by the feature enhancement layer as input, is used to obtain high-order interaction information between the enhanced feature information, and inputs the high-order interaction information into the prediction layer.

[0042] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0043] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the above method.

[0044] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0045] This invention innovatively combines the multi-head self-attention mechanism to effectively capture the complex interactive information between features; at the same time, by introducing the SENet attention module, it realizes the adaptive adjustment of the importance of features in different contexts. The fusion of these two attention mechanisms not only improves the efficiency and accuracy of feature expression, but also avoids the increase of a large number of training parameters, which can significantly improve the accuracy of click-through rate prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a structural schematic diagram of a click rate prediction model constructed by a click rate prediction method based on multi-attention mechanism fusion feature enhancement provided by an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the network structure of the feature enhancement layer provided by an embodiment of the present invention;

[0049] Figure 3 It is a system block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0051] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0052] Moreover, in the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0053] First embodiment

[0054] For the prediction of click-through rate of online advertisements and recommendation systems, this embodiment provides a click-through rate prediction method based on multi-attention mechanism fusion feature enhancement, the goal of which is to use advanced deep learning models to improve the accuracy of predicting the possibility of users clicking on specific content or advertisements. The method can be implemented by an electronic device, which can be a terminal or a server. The execution process of the method includes the following parts:

[0055] 1. Data collection and preprocessing.

[0056] 1. Data collection: Comprehensively collect historical click instance sample data from online advertising platforms;

[0057] Among them, the click instance sample data collected in this embodiment includes sample feature information and label values; the sample feature information is mainly divided into three categories: sample user behavior feature information, target object (advertisement) feature information and contextual environment data. Sample user behavior feature information: This category of data is intended to capture the user's interaction history with the platform, including but not limited to user browsing history, user search keyword records, user stay time on the platform, click-through rate in the past period of time, etc. This information helps to understand the user's interest preferences and behavior patterns. Target object (advertisement) feature information: This category of data focuses on the attributes of the advertisement itself, including keywords in the advertisement content, advertiser industry classification, time and frequency of advertisement release, visual design elements of the advertisement (such as color, layout), and past performance of the advertisement (such as historical click-through rate). Contextual environment data: This includes environmental information related to the current user interaction moment, such as the time (timestamp) when the user clicks the advertisement, the user's geographical location, the weather conditions at the time, the type of web page or application being browsed, the device type at the time (such as mobile phone, tablet or desktop) and the operating system, etc. The label value is used to indicate whether the sample user has interactive behavior with the target object.

[0058] 2. Data cleaning: Clean the collected data, remove records with a large number of missing values, reasonably fill in slightly missing data points, and properly handle outliers.

[0059] 3. Feature preprocessing: Normalize numerical features and convert categorical features through one-hot encoding to reduce model complexity and improve training efficiency.

[0060] 4. Dataset division: The dataset is divided into training set, validation set and test set according to the ratio of 8:1:1 for subsequent model training and performance evaluation.

[0061] 2. Combine the multi-head self-attention mechanism and SENet network to build a click rate prediction model;

[0062] The click rate prediction model MAFRFM constructed in this embodiment is as follows: Figure 1 As shown in Figure 1, it mainly includes the following key components: feature input layer, feature embedding layer, feature enhancement layer, feature interaction layer and prediction layer. The following is a detailed description of each module.

[0063] 1. Feature input layer

[0064] Responsible for processing input features. In large-scale recommendation systems or CTR prediction problems, input features are generally divided into two types: numerical features and categorical features. Numerical features are represented by numerical quantities, such as height, age, price, etc., and can usually be used directly for network learning. However, most features in CTR prediction tasks are categorical features, that is, features represented by classifications or labels, such as "gender = male", "nationality = China", etc. This type of feature cannot usually be compared directly and requires one-hot encoding. For example, for the gender feature, if the sample is "male", the corresponding vector is [1, 0]; if it is "female", the corresponding vector is [0, 1].

[0065] 2. Feature Embedding Layer

[0066] Responsible for converting high-dimensional sparse feature vectors into low-dimensional dense vectors. Since categorical features need to be One-hot encoded, when the categorical features have multiple values, One-hot encoding will cause the feature dimension to explode and increase the network calculation complexity. Therefore, it is necessary to use a feature embedding layer to compress the high-dimensional sparse feature vector into a low-dimensional dense vector. The resulting feature vector is shown in formula (1):

[0067] E=[E1,E2,...,E n ] (1)

[0068] In the formula, n represents the number of domains, E i Represents the embedding vector of the i-th feature.

[0069] 3. Feature Enhancement Layer

[0070] The feature expression is enhanced by MAFRNet (Multi-Attention Feature Refinement Network). MAFRNet mainly consists of two parts: information extraction unit and gating unit. The network structure is as follows Figure 2 As shown in the figure, the information extraction unit is used to capture the cross-relationship between features and feature context information, and the gate control unit uses a complementary selection gate unit to adaptively combine the original features with the enhanced features extracted by the information.

[0071] 3.1 Information Extraction Unit

[0072] The information extraction unit in MAFRNet is mainly composed of four parts: multi-head self-attention module, SENet module, context information extraction module and integration module.

[0073] 1) Multi-head self-attention module: The multi-head self-attention model is a neural network structure based on the attention mechanism, which can be used to capture key features and interaction information in the input sequence. In this model, the attention weight is used to calculate the relative importance of each input element with other elements, so as to perform weighted aggregation and feature extraction on the input sequence. By analyzing and visualizing the attention weights, the correlation between features can be obtained, that is, which feature interactions contribute more or less to the output. The formula is as follows:

[0074] Q, K, V = EW Q , E.W. K , E.W. V (2)

[0075] Attention(Q,K,V)=Soft Max(QK T )V (3)

[0076] head i =Ahention(Q, K, V) (4)

[0077] o vec =Concat(head1, head2,..., head h )W o (5)

[0078] Among them, Q, K, V are query vector, key vector and value vector respectively, and W Q , W K , W V is the transformation matrix, head i represents the output of the i-th head, h is the number of heads, W O is the weight matrix of the linear transformation.

[0079] This embodiment introduces a multi-head self-attention mechanism to learn multiple attention weights in parallel, so that the model can focus on different feature combinations to more fully capture the interactive information between features.

[0080] 2) SENet module: The contextual information in each instance is implicit in all features. Therefore, it is necessary to ensure that all features contribute to the contextual information in each instance. By introducing the SENet attention module, the importance of different features in the contextual information is adjusted to enhance the effect of contextual information mining. SENet can help us model feature importance. It can dynamically increase the weight of important features and reduce the weight of irrelevant features. SENet mainly includes four steps: Squeeze, Excitation, Re-weight and Fuse.

[0081] The purpose of Squeeze is to rescale the original embedded features into a new compressed vector. For each feature vector, select the max, min or mean value to represent the information of the feature. Its expression is as follows:

[0082] Z i =[Z max , Z min , Z mean ] (6)

[0083] Z=[Z1,Z2,...,Z n ] (7)

[0084] Among them, Z max , Z min , Z mean Represents the maximum, minimum and average values ​​of the eigenvectors, Z i Represents the compressed feature vector.

[0085] The role of Excitation is to use compressed vectors to learn feature weights. Here, two fully connected layers with activation functions are used to obtain weight information A. The calculation formula is as follows:

[0086] A=ReLU(W2ReLU(W1Z)) (8)

[0087] Among them, ReLU is the activation function, is a training parameter, and r is a hyperparameter representing the reduction ratio.

[0088] The Re-Weight step adjusts the embedding vector according to the obtained weight vector to obtain the feature vector E with weight information. w :

[0089]

[0090] This embodiment adds a SENet network to the context information extraction unit and automatically adjusts the weight according to the importance of the feature to improve the accuracy of the context information and obtain richer context information.

[0091] 3) Context information extraction module

[0092] The context information extraction module is used to mine the relationship between feature contexts and to distinguish the importance of features in different contexts. This paper uses the MLP (Multilayer Perceptron) module to mine context information. Each layer of the MLP module is as follows:

[0093] h l =ReLU(W l h l-1 +b l ) (10)

[0094] Among them, h l and h l-1 are l layers and l-1 layers of hidden layers respectively, W l , b l is the learnable parameter of the lth layer, and the output of the last layer is the bit-level weight O with context information bit .

[0095] 4) Integrated modules

[0096] After obtaining the context information, the context information vector is directly used to weight the feature representation. The calculation is as follows:

[0097] O=O vec ⊙O bit (11)

[0098] Among them, O vec , O bit are the outputs of the multi-head self-attention module and the context information extraction module respectively, and ⊙ is the element-wise product of the two vectors.

[0099] 3.2 Gate Control Unit

[0100] In the gated unit, a complementary gating mechanism is designed to control the information flow and select important information from the enhanced features and original features with bit-level weights. The gated unit has three different inputs from three channels: 1) enhanced feature representation; 2) weight matrix; 3) original feature representation. The calculation formula is as follows:

[0101] E r =E⊙σ(W b )+E r ⊙(1-σ(W b )) (12)

[0102] Among them, E represents the original feature, E rrepresents the enhanced features obtained by the information extraction unit, σ is the sigmoid activation function, and W b is the gating matrix obtained by another information extraction unit.

[0103] 4. Feature Interaction Layer

[0104] It includes low-order interactions and high-order interactions. Low-order interactions enhance the memory ability of the model, and high-order interactions enhance the generalization ability of the model. This model uses the FM model for low-order interactions and the DNN model for high-order interactions.

[0105] The FM model is a classic low-order cross model that can model the interaction between features. Specifically, for a sample with n features, the FM model can be expressed as:

[0106]

[0107] Among them, x i represents the i-th eigenvector, w0 is the bias term, and w i represents the weight of the i-th feature, w i represents the latent vector of the i-th feature, <v i ,v j > represents the vector inner product. The first two terms in the model represent the first-order linear relationship of the features, and the last term represents the second-order cross relationship of the features.

[0108] As a representative of deep learning models, DNN can more effectively model and process high-dimensional, nonlinear data. In the click-through rate prediction task, DNN captures high-order interaction information between features, thereby improving the model's expressiveness. The calculation formula for each layer is as follows:

[0109] h l =σ(w l h l-1 +b l ) (14)

[0110] Among them, h l and h l-1 are l layers and l-1 layers of hidden layers respectively, w l , b l is the learnable parameter of the lth layer, and σ represents the nonlinear activation function. Each hidden layer converts the feature representation of the previous layer into a higher-level abstract feature representation through nonlinear transformation, thereby achieving the purpose of feature extraction.

[0111] 5. Feature prediction layer

[0112] The outputs of the two parts of the interaction layer are combined and the final click-through rate prediction is made.

[0113]

[0114] in, For the prediction results, yfm is the output of the low-order interaction part, and ydnn is the output of the high-order interaction part.

[0115] Based on the above, specifically, this embodiment constructs a neural network based on the MAFRFM model under the PyTorch framework for click-through rate prediction. Specifically, the embedding dimension is set to 16 in the embedding layer, the number of heads of the multi-head self-attention mechanism is set to 4 in the feature enhancement layer, and the reduction ratio of the SENet module is set to 3. In the feature cross layer, the DNN module is set to a 3-layer structure, each layer contains 200 neurons.

[0116] 3. Model training and optimization.

[0117] The constructed click-through rate prediction model is trained using the sample data set; the Adam (Adaptive Moment Estimation) optimizer is used, AUC (Area under Curve) and Logloss are used as evaluation indicators, and the model is fully trained using a large amount of historical data. The number of iterations is set to 1000, the batch size is 1024, and the learning rate is set to 0.001. The gradient descent method is used to update the parameters, and the early stopping method and regularization techniques are used to prevent overfitting. During the training process, the model performance is evaluated by the cross-validation method to select the optimal model parameters.

[0118] 4. Output and application.

[0119] After the model training is completed, for a given user and ad combination, the model will output the predicted click-through rate value. Ads are sorted according to the predicted click-through rate, and ads with higher predicted click-through rates are displayed first.

[0120] 5. Evaluation and Iteration

[0121] Regularly evaluate model performance using newly collected data, compare actual clicks with predicted results, and adjust model parameters or structure accordingly to continuously optimize prediction accuracy.

[0122] In summary, this embodiment provides a new type of click-through rate prediction model, namely the MAFRFM (Multi-Attention Feature Refinement Factorization Machine) model, which realizes in-depth learning of complex interactions between features and dynamic adjustment of feature importance by integrating the multi-head self-attention mechanism and the SENet network, thereby significantly improving the accuracy and generalization ability of CTR prediction. The MAFRFM model can not only capture the interactive information between features, but also automatically adjust the feature weights according to the context and optimize the feature expression. This can significantly improve the accuracy and efficiency of click-through rate prediction in online advertising and recommendation systems.

[0123] Second embodiment

[0124] This embodiment provides an electronic device, such as Figure 3 As shown, the electronic device includes: a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment. In addition, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0125] Next, combine Figure 3 The following is a detailed introduction to the various components of the electronic device:

[0126] Among them, the processor is the control center of the electronic device, and the electronic device may include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may be a processor or a general term for multiple processing elements. For example, the processor is one or more central processing units (CPUs), or other general-purpose processors, application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor may execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0127] In a specific implementation, as an embodiment, the processor may include one or more CPUs, such as Figure 3 The CPU0 and CPU1 shown in the figure are, of course, only exemplary.

[0128] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0129] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Figure 3 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0130] The transceiver may include a receiver and a transmitter ( Figure 3 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Figure 3 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0131] In addition, it should be noted that Figure 3 The structure of the electronic device shown in the figure does not constitute a limitation on the device, and the actual device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment above can refer to the technical effects described in the first embodiment above, so they are not repeated here.

[0132] Third embodiment

[0133] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.

[0134] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.

[0135] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only an association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.

[0138] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0139] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0140] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0141] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0142] Finally, it should be noted that the above is a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for those skilled in the art, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A click rate prediction method based on multi-attention mechanism fusion feature enhancement, characterized in that: The click rate prediction method based on multi-attention mechanism fusion feature enhancement includes: Acquire historical click instance sample data, and preprocess the historical click instance sample data to construct a sample data set; wherein the click instance sample data includes sample feature information and label values; Combining the multi-head self-attention mechanism and SENet network, a click-through rate prediction model is constructed; Using the sample data set to train the constructed click rate prediction model; Use the trained click rate prediction model to predict click rate; Regularly evaluate the performance of the click-through rate prediction model, and adjust the model parameters or structure by comparing the actual clicks with the click-through rate prediction results output by the model to continuously optimize the prediction accuracy; The click rate prediction model includes: a feature input layer, a feature embedding layer, a feature enhancement layer, a feature interaction layer and a prediction layer; wherein, The feature input layer is used to process the sample feature information input into the click rate prediction model to obtain a high-dimensional sparse feature vector, and input the high-dimensional sparse feature vector into the feature embedding layer; The feature embedding layer is used to convert the high-dimensional sparse feature vector into a low-dimensional dense vector as original feature information, and input the original feature information into the feature enhancement layer; The feature enhancement layer is used to enhance the original feature information to obtain enhanced feature information, and input the enhanced feature information into the feature interaction layer; The feature interaction layer is used to obtain low-order interaction information and high-order interaction information between enhanced feature information output by the feature enhancement layer, and input the low-order interaction information and high-order interaction information into the prediction layer; The prediction layer is used to combine the low-order interaction information and the high-order interaction information output by the feature interaction layer, perform final click-through rate prediction, and output the click-through rate prediction result; The feature enhancement layer includes a first information extraction unit, a second information extraction unit and a gating unit; wherein, The first information extraction unit takes the original feature information as input, is used to obtain the preliminary enhanced feature corresponding to the original feature information, and inputs the preliminary enhanced feature into the gating unit; The second information extraction unit takes the original feature information as input, is used to obtain a gating matrix corresponding to the original feature information, and inputs the gating matrix into the gating unit; The gating unit takes the preliminary enhanced feature, the gating matrix and the original feature information as input, and is used to adaptively combine the original feature information with the preliminary enhanced feature based on the gating matrix by using a complementary selection gate unit to obtain enhanced feature information; The first information extraction unit and the second information extraction unit have the same structure, both comprising: a multi-head self-attention module, a SENet network, a context information extraction module and an integration module; wherein, The multi-head self-attention module takes the original feature information as input, and the processing result of the original feature information is input into the integration module; the SENet network takes the original feature information as input, and the processing result of the original feature information is input into the context information extraction module; the context information extraction module is a multi-layer perceptron network, and the output of the multi-layer perceptron network is input into the integration module, and the output of the integration module is input into the gating unit; The output of the integration module is represented as follows: O=O rec ⊙O bit Wherein, O represents the output of the integrated module; vec represents the output of the multi-head self-attention module; bit represents the output of the context information extraction module; ⊙ represents the element-wise product of two vectors.

2. The click-through rate prediction method based on multi-attention mechanism fusion feature enhancement as claimed in claim 1, characterized in that: Preprocessing the historical click instance sample data to construct a sample data set includes: Preprocessing the acquired historical click instance sample data; wherein the preprocessing process includes: cleaning the acquired historical click instance sample data; normalizing the numerical features in the cleaned historical click instance sample data; According to the ratio of 8:1:1, the preprocessed historical click instance sample data is divided into a training set, a validation set and a test set; wherein the training set is used to train the click rate prediction model; the validation set is used to verify the trained model; and the test set is used to test the trained model.

3. The click-through rate prediction method based on multi-attention mechanism fusion feature enhancement as claimed in claim 1, characterized in that: The process of processing sample feature information by the feature input layer includes: One-hot encoding is performed on the categorical features in the sample feature information.

4. The click-through rate prediction method based on multi-attention mechanism fusion feature enhancement as claimed in claim 1, characterized in that: The calculation formula of the enhanced feature information is as follows: E r E⊙σ(W b )+E rp ⊙(1-σ(W b )) Among them, E r represents enhanced feature information; E represents original feature information; E rp represents the preliminary enhanced features output by the first information extraction unit; σ represents the sigmoid activation function, W b Represents the gating matrix output by the second information extraction unit.

5. The click-through rate prediction method based on multi-attention mechanism fusion feature enhancement as claimed in claim 1, characterized in that: The feature interaction layer includes a FM model and a DNN model; wherein, The FM model uses the enhanced feature information output by the feature enhancement layer as input, is used to obtain low-order interaction information between the enhanced feature information, and inputs the low-order interaction information into the prediction layer; The DNN model uses the enhanced feature information output by the feature enhancement layer as input, is used to obtain high-order interaction information between the enhanced feature information, and inputs the high-order interaction information into the prediction layer.

Images