Behavior prediction method and device, equipment, storage medium and product

Abstract

The invention discloses a behavior prediction method and device, equipment, a storage medium and a product, and relates to the technical field of data processing, the behavior prediction method comprises the following steps: obtaining user behavior data, and inputting the user behavior data to a preset TabNet model, the preset TabNet model comprising an attention module, a feature weight learning module and an operation module; performing attention extraction on the user behavior data through an attention module to obtain an attention weight; performing feature extraction on the user behavior data through a feature weight learning module to obtain a feature importance weight; and determining a user behavior prediction result through an operation module according to the attention weight and the feature importance weight. Compared with an existing mode of carrying out behavior prediction through a Tabnet model, the method has the advantages that the feature weight learning module can learn the importance of each feature, so that the model focuses on features with more information, and the behavior prediction efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to behavior prediction methods, devices, equipment, storage media and products. Background Art

[0002] As operators gradually strengthen their digital transformation efforts, their data collection, storage, processing and application capabilities in all aspects have also been improved. They have accumulated a large amount of user data with wide fields, strong continuity and high accuracy, covering users' basic information, online behavior information, social information, location information, etc. Relying on these data, the financial industry has been widely used to explore potential customers, develop new products and guide new directions.

[0003] In the typical binary classification problem of credit default prediction, the use of various algorithms is very common. Tabnet was proposed by Google in 2019. It is a novel neural network structure designed specifically for tabular data. It uses the idea of ​​sequential attention to imitate the process of decision trees. Each decision step will contribute its own decision result in the final decision process. The model summarizes the weighted decisions of all decision steps as the final output. Summary of the invention

[0004] The main purpose of this application is to provide a behavior prediction method, device, equipment, storage medium and product, aiming to solve the technical problem of low efficiency of existing behavior prediction.

[0005] To achieve the above objectives, the present application proposes a behavior prediction method, which includes:

[0006] Acquire user behavior data, and input the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module;

[0007] Extracting attention from the user behavior data through the attention module to obtain an attention weight;

[0008] Extracting features from the user behavior data through the feature weight learning module to obtain feature importance weights;

[0009] The user behavior prediction result is determined through the calculation module according to the attention weight and the feature importance weight.

[0010] Optionally, the feature weight learning module includes a fully connected layer, a global average pooling layer and an activation layer;

[0011] The step of extracting features from the user behavior data through the feature weight learning module to obtain feature importance weights includes:

[0012] Compressing the user behavior data through the fully connected layer to obtain a compression result;

[0013] Performing average pooling on the compression result based on the global average pooling layer to obtain feature perception importance;

[0014] The feature abstraction extraction is performed on the feature perception importance according to the activation layer to obtain the feature importance weight.

[0015] Optionally, the preset TabNet model also includes a feature filtering module and a feature transformation module;

[0016] The step of extracting attention from the user behavior data through the attention module to obtain an attention weight comprises:

[0017] Performing feature extraction on the user behavior data by the feature filtering module to obtain sparse weights;

[0018] According to the sparse weight, strong feature data is filtered through the feature transformation module to obtain target features;

[0019] The attention module extracts attention from the target feature to obtain an attention weight.

[0020] Optionally, the feature filtering module includes a data abstraction layer, a data fusion layer, a data dimension reduction layer and a weight pre-representation layer;

[0021] The step of purifying the user behavior data by the feature filtering module to obtain sparse weights includes:

[0022] Perform multi-angle data abstraction on the user behavior data through the data abstraction layer to obtain multi-angle data abstraction results;

[0023] fusing the multi-angle data abstraction results according to the data fusion layer to obtain a fusion result;

[0024] Performing data dimension reduction on the fusion result based on the data dimension reduction layer to obtain the target structure;

[0025] The weight representation of the target structure is realized through the weight pre-representation layer to obtain sparse weights.

[0026] Optionally, the step of inputting the user behavior data into a preset TabNet model includes:

[0027] Classifying the user behavior data to obtain a classification result;

[0028] Inputting the classification result into a preset gradient boosting decision model to obtain the original feature importance weight and classification probability output by the preset gradient boosting decision model;

[0029] Determine input data according to the user behavior data, the original feature importance weight and the classification probability, and input the input data into a preset TabNet model.

[0030] Optionally, the step of classifying the user behavior data to obtain a classification result includes:

[0031] dividing the user behavior data into categorical data and continuous data;

[0032] Determine mixed data according to the categorical data and the continuous data;

[0033] Performing feature domain division on the categorical data based on the subject information of the categorical data to obtain a feature domain division result;

[0034] A classification result is determined according to the feature domain division result, the mixed data, the categorical data and the continuous data.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a behavior prediction device, which includes:

[0036] An acquisition module, used to acquire user behavior data, and input the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module;

[0037] An attention extraction module, used to extract attention from the user behavior data through the attention module to obtain an attention weight;

[0038] A feature extraction module, used to extract features from the user behavior data through the feature weight learning module to obtain feature importance weights;

[0039] A prediction module is used to determine the user behavior prediction result through the operation module according to the attention weight and the feature importance weight.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a behavior prediction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the behavior prediction method described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior prediction method described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the behavior prediction method described above are implemented.

[0043] This application obtains user behavior data, and inputs the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module; the attention module extracts attention from the user behavior data to obtain an attention weight; the feature weight learning module extracts features from the user behavior data to obtain feature importance weights; and the calculation module determines the user behavior prediction result based on the attention weights and the feature importance weights. Compared with the existing method of predicting behavior through the Tabnet model, the feature weight learning module of the above method of this application can learn the importance of each feature, allowing the model to focus on more informative features and improve the efficiency of behavior prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A flow chart of the first embodiment of the behavior prediction method of the present application;

[0047] Figure 2 A schematic diagram of an attention module provided in Example 1 of the behavior prediction method of this application;

[0048] Figure 3 A schematic diagram of the improved structure of the Attentive Transformer provided in Example 1 of the behavior prediction method of this application;

[0049] Figure 4 A schematic diagram of source data feature division provided in Example 1 of the behavior prediction method of this application;

[0050] Figure 5 A schematic diagram of GBDT feature encoding provided in Example 1 of the behavior prediction method of this application;

[0051] Figure 6 A flow chart of the second embodiment of the behavior prediction method of the present application;

[0052] Figure 7 A schematic diagram of the structure of a feature filtering module provided in Example 2 of the behavior prediction method of this application;

[0053] Figure 8 A schematic diagram of the improved Feature Transformer structure provided in Example 2 of the behavior prediction method of this application;

[0054] Fig. 9 This is a schematic diagram of the module structure of the behavior prediction device according to an embodiment of the present application;

[0055] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the behavior prediction method in the embodiment of the present application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] The main solution of the embodiment of the present application is: obtaining user behavior data, inputting the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module and a calculation module; performing attention extraction on the user behavior data through the attention module to obtain attention weights; performing feature extraction on the user behavior data through the feature weight learning module to obtain feature importance weights; and determining user behavior prediction results through the calculation module based on the attention weights and the feature importance weights.

[0060] Compared with the existing method of predicting behavior through the Tabnet model, the feature weight learning module of the above method of the present application can learn the importance of each feature, so that the model focuses on more informative features and improves the efficiency of behavior prediction.

[0061] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a behavior prediction device, etc. The following takes the behavior prediction device as an example to illustrate this embodiment and the following embodiments.

[0062] Based on this, the present application embodiment provides a behavior prediction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the behavior prediction method of the present application.

[0063] In this embodiment, the behavior prediction method includes steps S10 to S40:

[0064] Step S10, obtaining user behavior data, and inputting the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module;

[0065] It should be noted that the user behavior data may include the user's personal information, consumption behavior, communication behavior, location data, App usage information, etc. The preset TabNet model may be a model obtained by improving the Tabnet neural network structure. The Tabnet neural network structure was proposed by Google in 2019 and is a novel neural network structure designed specifically for tabular data. It uses the idea of ​​sequential attention to imitate the process of the decision tree. Each decision step will contribute its own decision result in the final decision process. The model summarizes the weighted decisions of all decision steps as the final output. Among them, one of the core modules is the Feature transformer. This module is responsible for performing complex computational processing on the input features in order to extract more useful information. The structure of this layer is more complex than the traditional fully connected (FC) layer. It contains a gated linear unit (GLU), which is a structure that adds a gating mechanism to the fully connected layer. Through the gating mechanism, the information flow can be adjusted according to the dynamic characteristics of the input features, thereby effectively capturing and combining nonlinear relationships in the features. Another core module of TabNet is the Attentive transformer. This module is responsible for learning the importance distribution of features. By learning the importance of each feature, different weights are assigned to different input features, similar to the attention mechanism, helping the model focus on more informative features. In addition, this module also gives the TabNet model a certain degree of interpretability, because it reveals the contribution of different features to model predictions, which helps to understand the decision-making process of the model, which is particularly valuable when processing tabular data. However, the original design of TabNet focuses on the interpretability of the model, and therefore deliberately avoids the automatic derivation of deep features. In particular, in the existing implementation, the embedding dimension used by categorical variables is set to 1, the purpose of which is to avoid the feature black box caused by high-dimensional vector embedding. However, this also greatly limits the performance accuracy of TabNet, and it is necessary to consider combining the characteristics of the algorithm structure for effective feature engineering. In the Feature Transformer module of TabNet, data features undergo a complex abstraction process. This module repeatedly uses the gated linear unit (GLU) gating mechanism and residual connection, imitating the feature selection method of the tree model and allowing flexible information transmission. This design prevents overfitting and enhances the expressiveness of the model. However, as the model structure deepens, information transmission may be lost. Therefore, it is necessary to improve the residual connection structure and adjust the data processing method of the gating module to maintain the efficiency of information flow and further improve the model performance. Another core module of TabNet, Attentive Transformer, implements instance-based feature selection, which is called instance-wise in the original paper, that is, feature selection can be performed separately for each input.This approach easily leads to overfitting of the model to the training data, thereby reducing its generalization ability; in addition, the original intention of the design of the Priorscales mechanism is to ensure that the sub-models generated in each step are as different as possible, which helps the model explore more feature representations. However, the initial Prior scales is set to 1 and each batch will be reset, which will cause the convergence of subsequent steps to become slow. Therefore, in this embodiment, the TabNet model is optimized to obtain a preset TabNet model. Including: In view of the incomplete decision-making of the flexible transfer simulation tree model of the GLU module in the Feature Transformer, and the data transmission loss problem caused by the simple residual connection, the Hardsigmoid function is used to directly set the parameters whose absolute values ​​are outside the threshold to 0 or 1, so as to implement a more robust feature screening strategy. In addition, the dense connection method is used to enhance the network's ability to extract and utilize features; the innovative FAT Block module is added to improve the model's feature mining ability; in response to the problems of generalization and the initial value setting of Prior scales in the Attentive Transformer module, a feature weight learning module is added to realize the attention weight transfer of global features. In addition, the feature importance output in GBDT is set as the initial value of Priorsscales to give prior knowledge to accelerate model convergence. In view of the fact that the data set contains many categorical variables and TabNet faces the contradiction between high-dimensional embedding and model interpretability, it is proposed to use GBDT to encode and convert data features before entering the model, and a method of traversing GBDT subtree nodes is proposed to give interpretability to the encoding vector of each dimension, so as to enhance the richness of data features and improve model accuracy.

[0066] Step S20, extracting attention from the user behavior data through the attention module to obtain an attention weight;

[0067] It should be noted that the attention module can be the Attentive Transformer in the TabNet model. Figure 2 , Figure 2 Schematic diagram of the attention module provided for the first embodiment of the behavior prediction method of the present application; the attention module calculates the mask layer of the current step based on the result of the previous step.

[0068] Among them, the sparsemax layer can be understood as a sparse version of softmax; the structure of this layer is represented by the formula:

[0069] M[i] = Sparsemax(P[i-1]·h i (a[i-1]))

[0070] Among them, M[i] represents the result of the Attentive Transformer layer in the i-th step. Because the entire TabNet structure is a repeated stacking process, i-1 represents the previous step. Sparsemax represents the calculation method for sparseizing its internal matrix. h represents the FC+BN layer; a[i-1] is the result of the previous step and the input of this layer. Mask means masking and filtering, which is a functional description of the Attentive Transformer layer. Through the calculation of the Attentive Transformer layer, a matrix that can filter and screen the original data feature matrix is ​​output. The core of this selective matrix is ​​implemented using the sparsemax layer. Among them, a[i-1] is divided by the split layer in the previous step, representing the FC+BN layer, and P[i] is the prior scales item, which is in the form of:

[0071]

[0072] Among them, γ represents a hyperparameter. P[i] is used to indicate the degree of application of a feature in the previous step. Intuitively, if a feature is used many times in the previous step, it should not be selected by the model again. Therefore, this item is used to reduce the weight of such features. M[i] can be understood as the model's attention weight distribution for batch samples at the current step. It is worth noting that the attention weight of the output of this layer is different for different samples. This feature is called instance-wise.

[0073] At the same time, as the step gradually deepens, the subsequent feature screening will become sparse, forcing the model to use meaningless features to fit the data, forcing the model to overfit the noise of the training set, and reducing the generalization performance. In addition, in each batch sample training, the prior scales are initialized to 1, that is, all features have the same importance, which will also delay the convergence of the model. Therefore, in the improvement of this module, GBDT is used in this implementation to give prior knowledge to the feature importance generated in the feature encoding process to accelerate the convergence of the model. The GBDT model is a preset gradient boosting decision model used to perform derivative encoding on the input data.

[0074] Step S30, extracting features from the user behavior data through the feature weight learning module to obtain feature importance weights;

[0075] It should be noted that the feature extraction of the user behavior data by the feature weight learning module to obtain the feature importance weight may be inputting the user behavior data into the feature weight learning module to obtain the feature importance weight output by the feature weight learning module.

[0076] Furthermore, the feature weight learning module includes a fully connected layer, a global average pooling layer and an activation layer;

[0077] The step of extracting features from the user behavior data through the feature weight learning module to obtain feature importance weights includes:

[0078] Compressing the user behavior data through the fully connected layer to obtain a compression result;

[0079] Performing average pooling on the compression result based on the global average pooling layer to obtain feature perception importance;

[0080] The feature abstraction extraction is performed on the feature perception importance according to the activation layer to obtain the feature importance weight.

[0081] It should be noted that you can refer to Figure 3 , Figure 3 Schematic diagram of the improved structure of Attentive Transformer provided in Example 1 of the behavior prediction method of this application; the fully connected layer included in the feature weight learning module is Figure 3 The FC+Relu structure in the above example, the global average pooling layer is AVG global average pooling, and the activation layer is Figure 3 The FC+Sigmoid structure in the example is as follows: according to the result of the previous step, the path is divided into two parts. The first path still maintains the original model structure, namely the attention module. The second part is a newly added structure, namely the feature weight learning module. In the feature weight learning module, the FC+Relu structure is used to abstract and compress the data features to maintain the number of features. Then, the global average pooling is performed to obtain the perceived importance of each feature. The FC+Sigmoid structure is used to further abstract the result into the feature importance weight for the feature column.

[0082] Step S40: determining a user behavior prediction result through the calculation module according to the attention weight and the feature importance weight.

[0083] It should be noted that, please refer to Figure 3After obtaining the attention weight and feature importance weight according to the original attention module and the newly added feature weight learning module, after instance-wise of the original structure (i.e., the attention module), the attention weight output by the attention module is multiplied by the feature importance weight generated by the feature weight learning module, so that the preset TabNet model always remembers important features. The overall structure of the improved Attentive Transformer is expressed as:

[0084] D(a[i-1])=σ(W 2 Avg(δ(W 1 a[i-1])))

[0085] M[i] = Sparsemax(P[i-1]·h i (a[i-1]))·D(a[i-1])

[0086] Among them, a[i-1] is divided by the split layer in the previous step, and passes through W 1 The matrix (model learnable parameters) is used for feature abstraction, the δ layer is used for nonlinear activation, and the avg layer is used for global pooling. 2 The matrix (model learnable parameters) is abstracted and finally passed through the Sigmoid function to realize parameter weighting and compress it to between 0 and 1; D(a[i-1]) is the result after the above compression and restoration, and it is multiplied with the result of the original attention module to obtain the final output.

[0087] Furthermore, when processing categorical features, neural networks often use the embedding method to convert categorical features into continuous, dense vectors to improve the model's ability and effectiveness in processing categorical data; however, high-dimensional vector representations bring about the unexplainability of dimensions, which is contrary to TabNet's attempt to establish feature interpretability in neural networks; therefore, this embodiment establishes a GBDT model on the source data, and establishes multiple tree models for data features in different domains, and understands the generation process of the tree model as a multi-dimensional combination of features. Leaf nodes are used to uniquely represent the path from the root node to the leaf, completing the derivative encoding of the data features, and giving the meaning of the path to the new code. Please refer to Figure 4 , Figure 4 A schematic diagram of source data feature division provided for the first embodiment of the behavior prediction method of the present application; specifically, the step of inputting the user behavior data into the preset TabNet model includes:

[0088] Classifying the user behavior data to obtain a classification result;

[0089] Inputting the classification result into a preset gradient boosting decision model to obtain the original feature importance weight and classification probability output by the preset gradient boosting decision model;

[0090] Determine input data according to the user behavior data, the original feature importance weight and the classification probability, and input the input data into a preset TabNet model.

[0091] It should be noted that the classification of the user behavior data to obtain the classification result can be to distinguish the massive data collected about the user according to the category type and the continuous type to obtain the classification result. The preset gradient boosting decision model can be a GBDT algorithm model. The preset gradient boosting decision model will output the corresponding feature importance (using gain, such as the Gini index) in the process of establishing the model in each feature domain (non-mixed feature domain), and use the min-max normalization method to compress the importance of all original features to between 0 and 1, and regard it as the importance weight of the original feature; at the same time, the new feature vector derived after GBDT encoding regards the classification probability of each leaf node as the importance weight of the dimension, that is, the classification probability. The input data is determined according to the user behavior data, the original feature importance weight and the classification probability, and the input data is input into the preset TabNet model. It can be that the user behavior data, the original feature importance weight and the classification probability are all used as input, and input into the preset TabNet model for behavior prediction. Specifically, the massive source data features collected are classified into categorical and continuous types. The categorical features are further classified into different topics. For example, based on business logic, the topics include the user's natural portrait features (occupation), consumption behavior features (package preferences), etc., and then mixed random sampling is performed on categorical features and continuous features to form mixed domain features. Finally, the features of different domains are encoded based on the GBDT algorithm. Figure 5 , Figure 5GBDT feature encoding schematic diagram provided for the first embodiment of the behavior prediction method of this application; the original features of a certain feature domain are constructed into 3 subtrees by the algorithm, each subtree has 4 leaf nodes, after a training sample is input, it falls into the 3rd leaf node of "subtree 1" successively, then the feature vector is [0, 0, 1, 0], the 1st leaf node of "subtree 2", the feature vector is [1, 0, 0, 0], the 4th leaf node of "subtree 3", the feature vector is [0, 0, 0, 1], and finally all feature vectors are connected to form the final feature vector [0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1]. In order to show the actual meaning of the final feature encoding dimension, the feature division combination from the root node to the nearest parent node of the leaf node is regarded as the actual meaning of the dimension, for example: "Gender_0 & Income>5000", while retaining the classification probability in the leaf node (for classification scenarios) for the subsequent TabNet structural feature weight initialization calculation.

[0092] In the specific implementation, when predicting whether a user's consumer loan will be overdue, forward-looking predictions can be made through the operator's massive data. Collect 380 dimensions, including consumer basic identity indicators, basic communication indicators, location indicators, app usage indicators, etc., including 60 categorical features. Divide the homogeneous categorical features into 3 feature domains. Some indicators are shown in Table 1-Sample Data. At the same time, categorical features and continuous features (which may include: the number of times various apps are used, traffic, duration, frequency, voice, SMS, traffic, broadband and other consumption amounts, etc.) are randomly mixed to form 2 mixed feature domains. The remaining continuous features are divided into 4 continuous feature domains according to their major categories (which can be consumption behavior, communication behavior, location data, App usage), totaling 9 feature domains.

[0093] Table 1 - Sample data

[0094]

[0095] The above 9 feature domains are respectively constructed using the GBDT model. Since the deeper the tree model, the higher the order of feature cross-derivation, but it is also more prone to overfitting, the important parameters are limited, the tree depth is 3, and the remaining main parameters are set as shown in Table 2-Parameter Setting Table. The rest of the tree models remain the same.

[0096] Table 2-Parameter Setting Table

[0097]

[0098]

[0099] After the model is built, the splitting process from the root node to the leaf node of the model is recorded through the method described above to form a new feature dimension, and the importance of the original features and the node probability of the new features in the model building process are saved and calculated into weights for subsequent initialization of the prior scales value of the improved TabNet. After the encoding of the source data is completed, the source data and the new feature data are spliced ​​and input into the improved TabNet model. The parameter settings are as shown in Table 3-Parameter Setting Table:

[0100] Table 3-Parameter setting table

[0101]

[0102] Comparison of the effect of improved TabNet and original TabNet: Substitute the prepared data into the improved preset TabNet model and the original TabNet for modeling. The two models use the same parameter configuration in the same structure. As shown in Table 4-Effect Comparison Table, the model AUC value increased from 75.34 to 78.47, and the KS value increased from 54.75 to 58.72, indicating the effectiveness of the improved TabNet model.

[0103] Table 4-Effect Comparison Table

[0104]

[0105]

[0106] This embodiment obtains user behavior data and inputs the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module; the attention module extracts attention from the user behavior data to obtain an attention weight; the feature weight learning module extracts features from the user behavior data to obtain a feature importance weight; and the calculation module determines the user behavior prediction result based on the attention weight and the feature importance weight. Compared with the existing method of predicting behavior through the Tabnet model, the feature weight learning module in the above method of this embodiment can learn the importance of each feature, so that the model focuses on more informative features and improves the efficiency of behavior prediction.

[0107] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 , Figure 6The flowchart of the second embodiment of the behavior prediction method of the present application is provided. The preset TabNet model also includes a feature filtering module and a feature transformation module. The step S20 also includes the following steps:

[0108] Step S201: purifying the user behavior data by the feature filtering module to obtain sparse weights;

[0109] It should be noted that Feature Transformer is the core feature processing abstract module of the TabNet model, that is, the feature transformation module. The original structure is difficult to obtain effective feature information only by stacking FC layers. At the same time, in order to simplify and purify the original information, this embodiment adds a Feature Attention-Block (FAT Block) module at the beginning, that is, the feature filtering module. The feature filtering module is used to purify the user behavior data to obtain sparse weights, thereby retaining data with strong features.

[0110] Furthermore, the feature filtering module includes a data abstraction layer, a data fusion layer, a data dimension reduction layer, and a weight pre-representation layer;

[0111] The step of purifying the user behavior data by the feature filtering module to obtain sparse weights includes:

[0112] Perform multi-angle data abstraction on the user behavior data through the data abstraction layer to obtain multi-angle data abstraction results;

[0113] fusing the multi-angle data abstraction results according to the data fusion layer to obtain a fusion result;

[0114] Performing data dimension reduction on the fusion result based on the data dimension reduction layer to obtain the target structure;

[0115] The weight representation of the target structure is realized through the weight pre-representation layer to obtain sparse weights.

[0116] It should be noted that you can refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of the feature filtering module provided in Example 2 of the behavior prediction method of this application. The data abstraction layer of the feature filtering module is Figure 7In the FC layer, the data fusion layer is Concat, the data dimension reduction layer is Global Pooling layer, and the weight pre-representation layer is SparseMax. Specifically, the feature filtering module copies K copies of the input data, first performs multi-angle data abstraction through the FC layer, and then Concat fuses the data. The data will be analogous to the multi-channels in the image; then it is reduced to a 1*K structure through the Global Pooling layer, and passes through the FC layer, ReLU activation layer, and FC layer in turn. The dimensionality changes are 1*K / r, 1*K / r, and 1*K in turn; at this time, the weight is pre-represented and passed through the SparseMax layer to achieve sparse k-dimensional weight representation, that is, sparse weight.

[0117] Step S202: filtering strong feature data through the feature transformation module according to the sparse weight to obtain target features;

[0118] It should be noted that this embodiment improves the Feature Transformer module (i.e., feature transformation module) in the TabNet model. Figure 8 , Figure 8 The schematic diagram of the improved FeatureTransformer structure provided for the second embodiment of the behavior prediction method of this application shows that the change of the gated activation function in the Feature Transformer will cause the data to be sparse during the transmission process. Add connections to the input data at the junction of the shared layer (shared across decision steps) and the independent layer (decision step dependent), and add connections to the two residual connections of the independent layer at the same time; this module processing improves the information flow and gradient feedback, thereby reducing the gradient disappearance; at the same time, it enables subsequent processing to perceive the most original data information, enhances the reuse of features, and promotes deeper information interaction between independent layers and shared layers.

[0119] The strong feature data is filtered through the feature transformation module according to the sparse weights to obtain the target feature, which can be a feature-level multiplication operation of the initially extracted feature data (i.e., the feature data flowing into the feature transformation module) and the sparse weights to achieve data retention of strong features, and pass these strong information, i.e., the target features, to the rest of the entire module through dense connections. Figure 7 .

[0120] Furthermore, the original feature conversion processing module of the TabNet model can be regarded as a stack of FC+BN+GLU (FC+BN represents the fully connected layer and the normalized layer respectively) and a simple residual connection, where GLU is a gating mechanism, and its calculation method is as follows:

[0121]

[0122] Among them, X is used to represent the data features input to TabNet after GBDT pre-processing, where x represents the matrix element of X, and W, B, V, and C are used to represent learnable parameters. The Sigmoid function compresses the value to between (0, 1) to achieve flexible feature screening.

[0123] To address the incomplete decision-making of the flexible transfer simulation tree model of the GLU module in the Feature Transformer and the data transmission loss problem caused by the simple residual connection, this embodiment replaces the Sigmoid function with the HardSigmoid function, and the calculation formula of the function is as follows:

[0124]

[0125] The threshold of the original function is set to ±3. Here, because the upstream data will pass through the BN layer, the threshold is changed to ±1. Compared with the original function, this function implements strong feature screening, which is beneficial to avoid overfitting of the model and is more in line with the strategy of the tree model in the feature screening process. In addition, the HardSigmoid function avoids the operation of exponential functions, which helps to speed up the operation when a large number of GLU layer connection operations are required.

[0126] Step S203: extracting attention from the target feature through the attention module to obtain an attention weight.

[0127] It should be noted that the attention module is used to extract the attention of the target feature and obtain the attention weight, which can refer to the above-mentioned step S20, and this embodiment will not be repeated here.

[0128] This embodiment uses the feature filtering module to purify the user behavior data to obtain sparse weights; according to the sparse weights, the feature transformation module is used to filter strong feature data to obtain target features; and the attention module is used to extract attention from the target features to obtain attention weights. In this embodiment, the Hardsigmo id function is used to directly set parameters with absolute values ​​outside the threshold to 0 or 1 in order to implement a more robust feature screening strategy, in order to address the incomplete decision-making of the flexible transfer simulation tree model of the GLU module in FeatureTransformer and the data transmission loss caused by simple residual connections. In addition, a dense connection method is used to enhance the network's ability to extract and utilize features; the innovative FAT Block module is added to enhance the model's feature mining capabilities.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the behavior prediction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a behavior prediction device, please refer to Fig. 9 , the behavior prediction device comprises:

[0131] An acquisition module 10, used to acquire user behavior data, and input the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module;

[0132] An attention extraction module 20, configured to extract attention from the user behavior data through the attention module to obtain an attention weight;

[0133] A feature extraction module 30, configured to extract features from the user behavior data through the feature weight learning module to obtain feature importance weights;

[0134] The prediction module 40 is used to determine the user behavior prediction result through the calculation module according to the attention weight and the feature importance weight.

[0135] This embodiment obtains user behavior data and inputs the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module; the attention module extracts attention from the user behavior data to obtain an attention weight; the feature weight learning module extracts features from the user behavior data to obtain a feature importance weight; and the calculation module determines the user behavior prediction result based on the attention weight and the feature importance weight. Compared with the existing method of predicting behavior through the Tabnet model, the feature weight learning module in the above method of this embodiment can learn the importance of each feature, so that the model focuses on more informative features and improves the efficiency of behavior prediction.

[0136] The behavior prediction device provided by the present application adopts the behavior prediction method in the above embodiment, which can solve the technical problem of low efficiency of existing behavior prediction. Compared with the prior art, the beneficial effects of the behavior prediction device provided by the present application are the same as the beneficial effects of the behavior prediction method provided by the above embodiment, and the other technical features in the behavior prediction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0137] The present application provides a behavior prediction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the behavior prediction method in the above-mentioned embodiment one.

[0138] Reference below Fig.10 , which shows a schematic diagram of the structure of a behavior prediction device suitable for implementing the embodiment of the present application. The behavior prediction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The behavior prediction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0139] like Fig.10 As shown, the behavior prediction device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the behavior prediction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the behavior prediction device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a behavior prediction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0140] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0141] The behavior prediction device provided by the present application adopts the behavior prediction method in the above embodiment, which can solve the technical problem of low efficiency of existing behavior prediction. Compared with the prior art, the beneficial effects of the behavior prediction device provided by the present application are the same as the beneficial effects of the behavior prediction method provided by the above embodiment, and the other technical features in the behavior prediction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0144] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the behavior prediction method in the above-mentioned embodiment.

[0145] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0146] The computer-readable storage medium may be included in the behavior prediction device; or may exist independently without being assembled into the behavior prediction device.

[0147] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the behavior prediction device, the behavior prediction device executes the behavior prediction method.

[0148] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0151] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned behavior prediction method, and can solve the technical problem that the existing behavior prediction efficiency is low. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the behavior prediction method provided in the above-mentioned embodiment, and will not be repeated here.

[0152] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned behavior prediction method when executed by a processor.

[0153] The computer program product provided by this application can solve the technical problem of low efficiency of existing behavior prediction. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the behavior prediction method provided by the above embodiment, which will not be repeated here.

[0154] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A behavior prediction method, characterized in that: The behavior prediction method comprises the following steps: Acquire user behavior data, and input the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module; Extracting attention from the user behavior data through the attention module to obtain an attention weight; Extracting features from the user behavior data through the feature weight learning module to obtain feature importance weights; The user behavior prediction result is determined through the calculation module according to the attention weight and the feature importance weight.

2. The behavior prediction method according to claim 1, characterized in that: The feature weight learning module includes a fully connected layer, a global average pooling layer and an activation layer; The step of extracting features from the user behavior data through the feature weight learning module to obtain feature importance weights includes: Compressing the user behavior data through the fully connected layer to obtain a compression result; Performing average pooling on the compression result based on the global average pooling layer to obtain feature perception importance; The feature abstraction extraction is performed on the feature perception importance according to the activation layer to obtain the feature importance weight.

3. The behavior prediction method according to claim 1, characterized in that: The preset TabNet model also includes a feature filtering module and a feature transformation module; The step of extracting attention from the user behavior data through the attention module to obtain an attention weight comprises: Performing feature extraction on the user behavior data by the feature filtering module to obtain sparse weights; According to the sparse weight, strong feature data is filtered through the feature transformation module to obtain target features; The attention module extracts attention from the target feature to obtain an attention weight.

4. The behavior prediction method according to claim 3, characterized in that: The feature filtering module includes a data abstraction layer, a data fusion layer, a data dimension reduction layer and a weight pre-representation layer; The step of purifying the user behavior data by the feature filtering module to obtain sparse weights includes: Perform multi-angle data abstraction on the user behavior data through the data abstraction layer to obtain multi-angle data abstraction results; fusing the multi-angle data abstraction results according to the data fusion layer to obtain a fusion result; Performing data dimension reduction on the fusion result based on the data dimension reduction layer to obtain the target structure; The weight representation of the target structure is realized through the weight pre-representation layer to obtain sparse weights.

5. The behavior prediction method according to any one of claims 1 to 4, characterized in that: The step of inputting the user behavior data into a preset TabNet model comprises: Classifying the user behavior data to obtain a classification result; Inputting the classification result into a preset gradient boosting decision model to obtain the original feature importance weight and classification probability output by the preset gradient boosting decision model; Determine input data according to the user behavior data, the original feature importance weight and the classification probability, and input the input data into a preset TabNet model.

6. The behavior prediction method according to claim 5, characterized in that: The step of classifying the user behavior data to obtain a classification result includes: dividing the user behavior data into categorical data and continuous data; Determine mixed data according to the categorical data and the continuous data; Performing feature domain division on the categorical data based on the subject information of the categorical data to obtain a feature domain division result; A classification result is determined according to the feature domain division result, the mixed data, the categorical data and the continuous data.

7. A behavior prediction device, characterized in that: The behavior prediction device comprises: An acquisition module, used to acquire user behavior data, and input the user behavior data into a preset TabNet model, wherein the preset TabNet model includes an attention module, a feature weight learning module, and a calculation module; An attention extraction module, used to extract attention from the user behavior data through the attention module to obtain an attention weight; A feature extraction module, used to extract features from the user behavior data through the feature weight learning module to obtain feature importance weights; A prediction module is used to determine the user behavior prediction result through the operation module according to the attention weight and the feature importance weight.

8. A behavior prediction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the behavior prediction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the behavior prediction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the behavior prediction method according to any one of claims 1 to 6 are implemented.

Images