Internet card user churn prediction method and system based on user portrait

Abstract

The present invention discloses a method and system for predicting user churn of an Internet card based on user portraits. By analyzing the identity characteristics and behavioral characteristics of Internet card users related to churn, the key portrait data dimensions and key time series behavioral data dimensions of Internet card users are determined; the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card user behavior, and the key time series behavioral data dimensions include the number of abnormal behavior days that characterizes the abnormal behavior of Internet card users; key portrait data of corresponding dimensions of different users and key time series behavioral data of different periods are obtained from historical data to construct a training data set, and the constructed deep learning model is trained using the training data set, and then the trained deep learning model is used to predict user churn. When selecting the characteristic dimensions of training data, the present invention adds the active entropy and abnormal behavior days of users to reflect the churn behavior rules of different users, so that the prediction accuracy of the deep learning model obtained by training is higher.

Description

Technical Field

[0001] The present invention relates to the technical field of user churn prediction, and in particular to a method and system for predicting user churn of an Internet card based on user portraits. Background Art

[0002] With the rapid development of modern information technology and communication technology, users in the field of mobile communications can enjoy high-quality services at a lower price. Market competition is becoming increasingly fierce, demand is gradually saturated, and the loss of existing users is becoming increasingly serious. Among them, user loss refers to users terminating their service contracts with enterprises or switching to services provided by other companies. Garter's survey data shows that the cost of developing a new user is 4 to 5 times the cost of maintaining an old user. Other studies have shown that if a company reduces the user churn rate by 5%, it can increase profits by 25% to 85%. At present, the cost of developing a new user is getting higher and higher, so controlling user churn is a key issue related to the future survival and development of major communication operators. Retaining existing users through user churn prediction has become the primary goal of many mobile communication companies. As part of the customer relationship management (CRM) method, user churn prediction can effectively help companies reduce customer churn, which is of great significance to companies increasing revenue and improving competitiveness.

[0003] Due to the rapid development of deep learning in recent years, prediction algorithms based on deep learning models have also begun to emerge. Some use multi-layer feedforward networks (FFN) to predict user churn, while others use recurrent neural network (RNN) models for processing. When constructing training sets, traditional user churn prediction methods only consider the general behavioral differences of all users before and after churn, and do not consider the differences in the behavioral patterns of different users themselves, resulting in the construction of training sets that cannot well reflect the churn behavioral patterns of different users, which in turn leads to poor prediction accuracy of the neural network model trained with the training set.

[0004] Therefore, how to solve the problem that the existing user churn prediction methods do not take into account the differences in the behavioral patterns of different users when constructing the training set, resulting in the constructed training set not being able to well reflect the churn behavior patterns of different users, and then resulting in poor prediction accuracy of the neural network model trained with the training set has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The present invention provides an Internet card user churn prediction method and system based on user portraits, which are used to solve the technical problem of poor accuracy of existing user churn prediction methods.

[0006] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0007] A method for predicting Internet card user churn based on user portraits comprises the following steps:

[0008] Analyze the identity characteristics and behavioral characteristics of Internet card users related to churn, and determine the key portrait data dimensions and key time series behavioral data dimensions of Internet card users; wherein the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card users' behaviors, and the key time series behavioral data dimensions include the number of days of abnormal behaviors that characterizes abnormal behaviors of Internet card users;

[0009] Acquire key portrait data of different dimensions corresponding to different users and key time series behavior data of different periods from historical data to construct a training data set, and annotate the user churn categories corresponding to the training data in the training data set; construct a deep learning model, and use the annotated training data in the training data set to train the deep learning model to obtain a trained deep learning model;

[0010] Obtain the key portrait data and key time series behavior data of the corresponding dimensions of the user to be predicted, input the key portrait data and key time series behavior data of the corresponding dimensions of the user to be predicted into the trained deep learning model, and obtain the churn category of the user to be predicted.

[0011] Preferably, the key portrait data dimensions include: gender, age, mobile terminal model, mobile terminal price, mobile terminal usage time, balance, package ID, package level, second card slot to which operator belongs, online month, number of single suspensions in the past three months, number of double suspensions in the past three months, active entropy based on monthly upstream traffic, active entropy based on monthly downstream traffic, active entropy based on monthly online time, and active entropy based on the number of monthly online records;

[0012] The key time series behavior data dimensions include: daily downlink traffic and daily uplink traffic, daily Internet access time and, daily number of Internet access records and, whether the daily traffic is abnormal, daily number of calling calls, daily calling call duration, daily number of calling numbers, daily number of called calls, daily called call duration, daily number of called numbers and the number of base stations connected to daily calls.

[0013] Preferably, the acquisition of the activity entropy based on the monthly upstream traffic, the activity entropy based on the monthly downstream traffic, the activity entropy based on the monthly online time, and the activity entropy based on the number of monthly online records is achieved by the following steps:

[0014] For any dimension of data in monthly upstream traffic, monthly downstream traffic, monthly online time, and monthly number of online records, perform the following steps:

[0015] Convert the data of the dimension into a time series behavior sequence =[ , ,…, ,…, ], where d represents the day of the month, Indicates the statistical value of the user on the dimensional data on the i-th day;

[0016] Sequence the temporal behavior of the dimension data Divide it into k buckets of equal width, and calculate the activity entropy of the dimensional data according to the following formula:

[0017]

[0018] in, Represents the temporal behavior sequence of the dimension data The proportion of values ​​falling in the i-th bucket, k represents the number of buckets, Represents the temporal behavior sequence of the dimension data Length, is the indicator function, when hour =1, otherwise =0.

[0019] Preferably, the daily determination of whether the flow rate is abnormal is achieved by the following steps:

[0020] For any dimension of data in monthly upstream traffic, monthly downstream traffic, monthly online time, and monthly number of online records, perform the following steps:

[0021] Convert the data of the dimension into a time series behavior sequence =[ , ,…, ,…, ], where n represents the number of days in the month, Indicates the statistical value of the user on the dimensional data on the i-th day;

[0022] Calculate the sequence of temporal behaviors The first-order forward difference sequence of =[ , ,…, ,…, ],in, , Represents the first-order forward difference sequence The i-th difference value;

[0023] For the first-order forward difference sequence Any difference value in , judge the Is it greater than +1.5IQR or less -1.5IQR, if stated Greater than +1.5IQR or less -1.5IQR, then judge There are abnormal traffic on the corresponding days.

[0024] in, is the first-order forward difference sequence The upper quartile of is the first-order forward difference sequence The lower quartile, IQR is the interquartile range, which is and difference.

[0025] Preferably, the deep learning model is a deep learning model based on a multi-head self-attention mechanism, comprising a feature extraction layer, an embedding layer, an attention layer, a pre-classification layer and a classifier, wherein the embedding layer, the attention layer, the pre-classification layer and the classifier are connected in sequence, and the classifier is also connected to the feature extraction layer;

[0026] The feature extraction layer is used to extract a reduced-dimensional user portrait key feature matrix from the input key portrait data, and output the key feature matrix to the classifier;

[0027] The embedding layer is used to convert the input key temporal behavior data and its corresponding category tags into an embedded feature block containing a block embedding vector, a category embedding vector and a position embedding vector, and output the embedded feature block to the attention layer; the category embedding vector is used to learn and mark the target category of each block embedding vector; the position embedding vector is used to mark the positional relationship between each block embedding vector;

[0028] The attention layer is used to extract the target category of each block embedding vector of the embedded feature block, and perform self-similarity calculation on the target category of each extracted block embedding vector according to the position relationship between each block embedding vector marked by the position embedding vector to obtain the category embedding vector of the embedded feature block; and output the category embedding vector of the embedded feature block to the pre-classification layer;

[0029] The pre-classification layer is used to pre-classify the category embedding vector of the embedded feature block and output the user churn probability value to the classification layer;

[0030] The classification layer is used to determine the churn category of the user according to the key feature matrix of the user portrait and the user churn probability value.

[0031] Preferably, the key temporal behavior data is input into the embedding layer in the form of a matrix, and the embedding layer converts the input key temporal behavior data and its corresponding category labels into an embedding feature block including a block embedding vector, a category embedding vector and a position embedding vector, comprising the following steps:

[0032] Assume that the matrix form of the key timing behavior data is the timing behavior feature matrix , where N is the characteristic dimension of the time series behavior feature matrix, and L is the length of each time series feature; the time series behavior feature matrix Cut into pieces of size The matrix blocks are divided into time series behavior feature matrix ;

[0033] The segmented time series behavior feature matrix is ​​embedded into E through linear transformation Mapped to a dimension of size D, the embedded representation is obtained ;

[0034] In the embedded representation Add category embedding vectors ∈ and the position embedding vector , the output of the embedding layer is:

[0035]

[0036] in, is the output of the embedding layer, and For the embedding representation Middle block embedding vectors, , block embedding vector The corresponding position embedding is , the category embedding vector ∈ is a learnable embedding vector with no obvious semantic information at the beginning.

[0037] Preferably, the attention layer is a multi-head self-attention layer, which extracts the target category of each block embedding vector of the embedded feature block, and performs self-similarity calculation on the target category of each extracted block embedding vector according to the position relationship between each block embedding vector marked by the position embedding vector to obtain the category embedding vector of the embedded feature block, including the following steps:

[0038] Calculate the similarity between each block embedding vector and other block embedding vectors and give corresponding weights:

[0039]

[0040]

[0041] in, and are three different parameter matrices, obtained by random initialization, with dimensions of (M+1)×(M+1). , , are the outputs of the embedding layer. The query, key and value matrices; A query vector The dimension of is M+1; To adopt the parameter matrix and The calculated feature submatrix is ​​used by the model to learn relevant information in different representation subspaces; is the serial number of the self-attention block in the multi-head self-attention layer, =1,2,3...,N, where N is the total number of self-attention blocks.

[0042] Compute the query matrix with the transpose of the key matrix The inner product between them is the fraction, divided by Normalize, then use the softmax activation function on the score, and then multiply the value matrix , and obtain the characteristic submatrix , splicing , multiplied by the projection matrix get , as a new sequence feature:

[0043]

[0044] is the projection matrix, the dimension is (N×D)×D, The dimension is (M+1)×D.

[0045] The new sequence features are output to the fully connected neural network in the attention layer, and then L layers are stacked, and finally the category embedding vector of the Lth layer is output, where L is determined according to the amount of data and experimental results, and the value range is 3~10.

[0046] Preferably, the pre-classification layer pre-classifies the category embedding vector of the embedded feature block and outputs the user churn probability value to the classification layer, which is achieved by the following formula:

[0047]

[0048] in, The category embedding vector of the Lth layer output by the multi-head self-attention layer:

[0049] Preferably, the classification layer determines the churn category of the user according to the key feature matrix of the user portrait and the user churn probability value, comprising the following steps:

[0050] After feeding the user portrait features and the user churn probability value output by the pre-classification to the classifier multi-layer perceptron MLP, training is performed to output the churn probability value ;

[0051] Then, according to the threshold filtering, the indicator variable I of whether the user will churn in the next month is output. The threshold filtering formula is:

[0052]

[0053] Among them, 0 means that the user will not churn next month, and 1 means that the user will churn next month.

[0054] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method in the above claim are implemented when the processor executes the computer program.

[0055] The present invention has the following beneficial effects:

[0056] 1. The method and system for predicting churn of Internet card users based on user portraits in the present invention determine the key portrait data dimensions and key time series behavior data dimensions of Internet card users by analyzing the identity characteristics and behavior characteristics related to churn of Internet card users; the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card user behavior, and the key time series behavior data dimensions include the number of abnormal behavior days that characterizes the abnormal behavior of Internet card users; obtain key portrait data of corresponding dimensions of different users and key time series behavior data of different periods from historical data to construct a training data set, and annotate the user churn categories corresponding to the training data in the training data set; and use the annotated training data in the training data set to train the constructed deep learning model, and then use the trained deep learning model to predict user churn. When selecting the characteristic dimensions of the training data, the present invention adds the active entropy and abnormal behavior days of users to reflect the churn behavior rules of different users, so that the prediction accuracy of the deep learning model trained with the training data is higher.

[0057] 2. In the preferred embodiment, the present invention constructs and converts the temporal behavior feature matrix into multiple block embedding vectors, adds the initial category embedding vector and the position embedding vector, extracts the target category of each block embedding vector of the embedded feature block, and calculates the self-similarity of each extracted block embedding vector in addition to the position relationship between each block embedding vector marked by the position embedding vector to obtain the category embedding vector of the embedded feature block; pre-classifies the category embedding vector of the embedded feature block, outputs the user churn probability value, extracts the key feature matrix of the user portrait from the user portrait feature matrix, and determines the churn category of the user according to the key feature matrix of the user portrait and the user churn probability value. Compared with the prior art, it can accurately and comprehensively capture the self-similarity between various temporal behavior features of users, and realizes high-precision user churn prediction in combination with the key feature matrix of the user portrait.

[0058] 3. In the preferred embodiment, the present invention uses a category label vector without inductive bias to represent the user churn probability sequence corresponding to the temporal behavior feature matrix, and then splices the low-dimensional embedding sequence, position sequence and learnable category label vector together, and learns the self-similarity of the spliced ​​sequence through a transformation layer using a multi-head self-attention mechanism, and then outputs the churn probability obtained according to the user's temporal behavior feature matrix through a lightweight classifier, and then splices it together with the dimensionality reduction features obtained by principal component analysis to feed the multi-layer perceptron used for classification, and finally outputs whether the user will churn, which can not only improve the accuracy of user churn prediction, but also improve the user prediction speed.

[0059] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0061] Figure 1 This is a diagram of the deep learning model architecture used in the Internet card user churn prediction method based on user portrait in the preferred embodiment of the present invention.

[0062] Figure 2 It is a comparison chart of the AUC performance of the Internet card user churn prediction method based on user portrait in the preferred embodiment of the present invention and other Internet card user churn prediction methods;

[0063] Figure 3It is a PR-AUC performance comparison diagram of the Internet card user churn prediction method based on user portrait in the preferred embodiment of the present invention and other Internet card user churn prediction methods;

[0064] Figure 4 It is an F-Score performance comparison diagram of the Internet card user churn prediction method based on user portrait in the preferred embodiment of the present invention and other Internet card user churn prediction methods;

[0065] Figure 5 It is a comparison chart of the model prediction performance before and after the active entropy is extracted between the Internet card user churn prediction method based on user portrait in the preferred embodiment of the present invention and other Internet card user churn prediction methods;

[0066] Figure 6 It is a comparison chart of the model prediction performance before and after the abnormal days are extracted between the Internet card user churn prediction method based on user portrait in the preferred embodiment of the present invention and other Internet card user churn prediction methods;

[0067] Figure 7 It is a flow chart of a method for predicting Internet card user churn based on user portrait in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0068] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0069] Embodiment 1:

[0070] like Figure 7 As shown, this embodiment discloses a method for predicting Internet card user churn based on user portraits, including the following steps:

[0071] Analyze the identity characteristics and behavioral characteristics of Internet card users related to churn, and determine the key portrait data dimensions and key time series behavioral data dimensions of Internet card users; wherein the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card users' behaviors, and the key time series behavioral data dimensions include the number of days of abnormal behaviors that characterizes abnormal behaviors of Internet card users;

[0072] Acquire key portrait data of different dimensions corresponding to different users and key time series behavior data of different periods from historical data to construct a training data set, and annotate the user churn categories corresponding to the training data in the training data set; construct a deep learning model, and use the annotated training data in the training data set to train the deep learning model to obtain a trained deep learning model;

[0073] Obtain the key portrait data and key time series behavior data of the corresponding dimensions of the user to be predicted, input the key portrait data and key time series behavior data of the corresponding dimensions of the user to be predicted into the trained deep learning model, and obtain the churn category of the user to be predicted.

[0074] In addition, in this embodiment, a computer system is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method in the above claim when executing the computer program.

[0075] The user portrait-based Internet card user churn prediction method and system in the present invention determines the key portrait data dimensions and key time series behavior data dimensions of Internet card users by analyzing the identity characteristics and behavior characteristics related to churn of Internet card users; the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card user behavior, and the key time series behavior data dimensions include the number of abnormal behavior days that characterizes the abnormal behavior of Internet card users; key portrait data of corresponding dimensions of different users and key time series behavior data of different periods are obtained from historical data to construct a training data set, and the user churn categories corresponding to the training data in the training data set are labeled; and the constructed deep learning model is trained using the labeled training data in the training data set, and then the trained deep learning model is used to predict user churn. When selecting the characteristic dimensions of the training data, the present invention adds the active entropy and abnormal behavior days of the users to reflect the churn behavior rules of different users, so that the prediction accuracy of the deep learning model trained with the training data is higher.

[0076] Embodiment 2:

[0077] Embodiment 2 In view of the key issues such as the loss and maintenance of existing users of current communication operators, a method for predicting the loss of Internet card users based on user portraits is proposed. First, the user attributes, CDR (call detail records), traffic data and other data collected by the operator are cleaned, and the target user groups required by the communication operator are extracted for each monthly data set. According to the loss judgment rule, each user is labeled whether it has been lost. Then, feature extraction is performed based on the data of each dimension of the user, which is mainly divided into four aspects, namely, the user's personal information, package and expense information, call detail records and traffic usage behavior. In particular, for the latter two, features are extracted according to formulas or algorithms to maximize the difference between normal users and lost users. In the second step, the user features and labels extracted based on the first step are used, and then the deep neural network based on the multi-head self-attention mechanism designed by the present invention is used to train and predict whether the existing users will be lost in the next month. The third step is to launch the model. The hyperparameters of the deep neural network and the trained weights and biases are fixed. Every new month, the operations of the first and second steps are repeated for the data set of that month, and then directly predicted by the solidified deep neural network.

[0078] In this embodiment, the method for predicting Internet card user churn based on user portraits specifically includes the following steps:

[0079] 1. Data cleaning and feature engineering

[0080] Use Spark SQL or Hive SQL to clean the user attributes, CDRs (call detail records), and traffic data collected by operators through their own OSS and BSS. Extract the target user groups required by the communication operators from the data set of each month, filter out abnormal data caused by data recording, format conversion, import and export, and then use Python to fill in the missing values ​​of the data of each dimension of the target user group. Then, filter out abnormal users based on domain knowledge, and then label each user as whether they have churned based on the churn determination rules.

[0081] Based on the data of each dimension of the user, feature extraction is performed. For sequence data such as call detail records and traffic usage behavior, the present invention proposes active entropy to characterize the uncertainty and activity of Internet card user behavior, and abnormal behavior days to characterize the abnormal behavior of Internet card users. The following are the specific calculation steps of these two features.

[0082] 1.1 Active entropy:

[0083] 1) Static statistics:

[0084] Statistics are collected for each user's daily traffic and / number of online records / online time / call time / number of call records, so each user has some behavior sequences, the number of which is equal to the number of features counted. Each behavior sequence can be expressed as [ , ,…, ,…, ], where d represents the day of the month, It represents the statistical value of a certain feature of the user on the i-th day. That is, the daily traffic generated every day in the month is used to construct the monthly traffic behavior sequence of the month (referred to as the monthly traffic), the daily traffic generated every day in the month is used to construct the monthly traffic behavior sequence of the month (referred to as the monthly traffic), the daily number of online records generated every day in the month is used to construct the number of online records behavior sequence of the month (referred to as the monthly call time), the daily online time generated every day in the month is used to construct the online time behavior sequence of the month (referred to as the monthly online time), the daily call time generated every day in the month is used to construct the call time behavior sequence of the month (referred to as the monthly call time), and the daily call records generated every day in the month are used to construct the number of call records behavior sequence of the month (referred to as the monthly call records).

[0085] 2) Bucketing:

[0086] The user's behavior sequence data for one month Divide it into k buckets of equal width, then the values ​​of the behavior sequence will be scattered in these k buckets, and the width of each bucket ( ) is equal to:

[0087]

[0088] in, and Represents the behavior sequence The maximum and minimum values ​​in the bucket are { + , +2 ,…, +(k-1) }, is the bucket width.

[0089] 3) Activity entropy calculation:

[0090] Based on 1) the situation of equal-distance bucketing, the entropy of this probability distribution can be calculated. ie active entropy It can be defined as:

[0091]

[0092] in, represents the proportion (probability) of the behavior sequence value falling into the i-th bucket, k represents the number of buckets, Representing a behavior sequence Length.

[0093] If the active entropy of a time series is large, it means that the values ​​of this time series are more evenly distributed in [ , ]; if the active entropy of a time series is small, it means that the values ​​of this time series are concentrated in a certain section.

[0094] 1.2 Days of abnormal behavior:

[0095] 1) Static statistics:

[0096] Statistics are collected for each user’s daily upstream and downstream traffic, online time, and number of online records. Therefore, each user has some behavior sequences, the number of which is equal to the number of statistical features. Each behavior sequence can be expressed as [ , ,…, ,…, ], where n represents the day of the month, Indicates the statistical value of a certain feature of the user on the i-th day.

[0097] 2) Forward difference calculation:

[0098] For a behavior sequence Calculate the first-order forward difference, so each user will get a first-order forward difference of a behavior sequence of length n-1 [ , ,…, ,…, ].in The calculation method is:

[0099]

[0100] 3) Anomaly Detection:

[0101] Outliers are defined as values ​​greater than +1.5IQR or less -1.5IQR value. It is the upper quartile, which means that 1 / 4 of all observations are larger than it. is the lower quartile, indicating that 1 / 4 of all observed values ​​are smaller than it. IQR is the interquartile range, which is and , which contains half of the observations.

[0102] According to the above outlier identification criteria, the outliers in the first-order forward difference of the behavior sequence are detected, and the number of outliers is used as the feature of the abnormal days.

[0103] 2. Build and train the model

[0104] The user labels and features extracted in the first and second steps are called data set D. The present invention performs n-fold cross-validation (for example, n=10) on data set D to verify the robustness of the model in different data partitions. Specifically, data set D is randomly divided into n sub-data sets of almost equal size, and a total of n model trainings are performed. During the i-th training, the i-th sub-data set is used as a test set, and the remaining sub-data sets are used as training sets to feed the deep neural network based on the multi-head self-attention mechanism designed by the present invention.

[0105] In this embodiment, if Figure 1 As shown, the deep neural network based on the multi-head self-attention mechanism includes a feature extraction layer, an embedding layer, an attention layer, a pre-classification layer and a classifier, wherein the embedding layer, the attention layer, the pre-classification layer and the classifier are connected, and the classifier is also connected to the feature extraction layer. The feature extraction layer is used to extract the reduced-dimensional user portrait key feature matrix from the user portrait feature matrix (referred to as the user portrait matrix in the figure), and output the key feature matrix to the classifier. The embedding layer is used to convert the temporal behavior feature matrix composed of the time series 3D tensor and its corresponding category labels into an embedded feature block containing a block embedding vector, a category embedding vector and a position embedding vector, and output the embedded feature block to the attention layer. The attention layer is used to extract the target category of each block embedding vector of the embedded feature block, and perform self-similarity calculation on the target category of each extracted block embedding vector according to the positional relationship between the block embedding vectors marked by the position embedding vector to obtain the category embedding vector of the embedded feature block; and output the category embedding vector of the embedded feature block to the pre-classification layer. The pre-classification layer is used to pre-classify the category embedding vector of the embedded feature block and output the user churn probability value to the classification layer; the classification layer is used to determine the churn category of the user according to the key feature matrix of the user portrait and the user churn probability value.

[0106] Among them, in this embodiment, the user features of the deep neural network based on the multi-head self-attention mechanism can be divided into two types: one is the user portrait feature matrix composed of the key portrait data of each user (2D tensor), such as age, gender, package amount, balance, network duration, active entropy, etc. The other is a 3D tensor of time series features composed of each user's key time series behavior data. ,For example: daily uplink traffic and, daily call duration, the number of base stations connected to daily calls, etc., among which the specific user characteristics and categories are shown in Table 1:

[0107] Table 1 Key time series behavior data and key time series behavior data dimension table

[0108]

[0109] 2.1 Feature Extraction Layer

[0110] In order to make full use of the hundreds of features of users, assuming that there is no order relationship between features, the present invention constructs a feature extraction layer to speed up model training and convergence while retaining the most important feature components, that is, to retain the original information to the greatest extent. Centralization, where User portrait feature matrix No. The user's portrait feature column vector, , n is the characteristic dimension of the user portrait feature matrix, m is the number of users, and the user portrait feature vector after centering each dimension of the user is calculated using the following formula to obtain the centralized feature vector of the i-th user :

[0111]

[0112] Assemble the user-centered feature vectors of all users into a user-centered feature matrix , calculate the covariance matrix of the user-centered feature matrix , and the covariance Perform eigenvalue decomposition and take the eigenvectors corresponding to the largest k eigenvalues Composing the projection matrix , multiply the projection matrix by the user feature matrix to obtain the key feature matrix extracted by the feature extraction layer:

[0113]

[0114] In order to better utilize the time series features with the richest data volume and finest granularity among the features, the present invention captures the potential relationship between the sequence features and whether the user is offline by adding an embedding layer, an attention layer and a pre-classification layer.

[0115] 2.2 Embedding Layer

[0116] The embedding layer is divided into block embedding, category embedding and position embedding. Block embedding transforms the matrix of multiple time series features of each user into a series of 1D block embedding representations. The time series tensor of each user input is recorded as , where N is the number of time series features (N=12), L is the length of each time series feature (L=30), and the time series tensor of each user is reshaped by the reshape operation Divide into size The block (P=4,Q=3) can be divided into blocks (M=30), we get , and then embed E through linear transformation Mapped to a dimension of size D (D=12), the embedded representation is obtained .

[0117] Assume that the time series tensor is divided into 30 blocks, that is, M=30, and there are 30 vectors input into the attention layer, but for these 30 vectors, no matter which vector is used for classification prediction, it is not suitable. Therefore, the present invention adds a category label vector, which is a learnable embedding vector, and inputs it into the attention layer together with the other 30 block embedding vectors, and finally takes the first vector as the category prediction result. Because compared with other existing vectors, this vector without obvious semantic information will more fairly integrate the semantic information of each vector, thereby better representing the semantics of all time series. This additional vector can be understood as the category information sought by the other 30 blocks, and can learn the loss state represented by different sequences. The specific approach is to randomly initialize a category label vector during training. ∈ (Each number is randomly selected from a normal distribution with a mean of 0 and a standard deviation of 1) as the first sequence feature and as the feature used for classification in the pre-classification layer.

[0118] In order to prevent the model from changing the prediction results by disrupting the order of blocks, the embedding layer also represents the blocks through trainable 1D position embedding and Block The positional relationship between (i, j∈[0,M)), that is, embedding the 2D block into a 1D sequence ,piece The corresponding position embedding is The output of the final embedding layer is

[0119]

[0120] 2.3 Attention Layer

[0121] In order to better capture the potential relationship between long-distance sequence features, the present invention uses a multi-head self-attention layer as a feature extractor to calculate the similarity between each sequence feature and other sequence features and give corresponding weights:

[0122]

[0123]

[0124] in, and are three different parameter matrices, obtained by random initialization, with dimensions of (M+1)×(M+1). , , are the outputs of the embedding layer. The query, key and value matrices; A query vector The dimension of is M+1; To adopt the parameter matrix and The calculated feature submatrix is ​​used by the model to learn relevant information in different representation subspaces; is the serial number of the self-attention block in the multi-head self-attention layer, =1,2,3, the total number of self-attention blocks is 3.

[0125] Compute the query matrix with the transpose of the key matrix The inner product between them is the fraction, divided by Normalize, then use the softmax activation function on the score, and then multiply the value matrix , and obtain the characteristic submatrix , splicing , multiplied by the projection matrix get , as a new sequence feature:

[0126]

[0127] Then it is output to the fully connected neural network in the attention layer, and then L layers are stacked, and finally the category label vector of the Lth layer is output. L is set to 3~10 in the example, specifically 6, which is an adjustable parameter and is fine-tuned according to the amount of data and experimental results.

[0128] 2.4 Pre-classification layer

[0129] The attention layer outputs the L-th layer category label vector to the pre-classification layer for pre-classification. The pre-classification layer outputs the off-grid probability value p inferred based on the time series features:

[0130]

[0131] 2.5 Classification Layer

[0132] The classification layer concatenates the off-grid probability value p inferred based on time series features and the feature matrix extracted by the feature extraction layer After feeding the classifier multi-layer perceptron MLP, it is trained to output the off-grid probability value , and then filter according to the threshold to output the indicator variable I of whether the user will churn in the next month.

[0133]

[0134] Among them, 0 means that the user will not churn next month, and 1 means that the user will churn next month.

[0135] 3. Verification and Evaluation

[0136] On the test set, predict whether these Internet card users will churn in the next month, and evaluate the prediction results using multiple evaluation indicators such as AUC, PR-AUC, and F-score. Finally, the average of the n evaluation indicators is taken as the final model evaluation result.

[0137] The test was conducted on a real data set with about 100,000 Internet card users in two months provided by a telecommunications operator. The test results are as follows Figure 2-5 As shown, we can find that the user churn prediction method based on the attention mechanism proposed in the present invention has a more obvious improvement compared with the prediction performance of other machine learning and deep learning models. For example, the AUC index is 35% higher than that of linear discriminant analysis, the PR-AUC index is 50% higher than that of decision tree, and the F-Score index is 21% higher than that of lightweight gradient boosting machine.

[0138] In order to show that the active entropy and abnormal days extracted by the present invention have a good improvement effect on all models, the present invention conducted an ablation experiment by removing the corresponding features. Figure 5 It shows that the addition of active entropy improves all models by 1%~1.5%. Figure 6 It shows that the addition of abnormal days improves all models by 1% to 5%.

[0139] Model launch and performance comparison

[0140] The weights, biases and hyperparameters trained by the fixed deep neural network are repeated for the data of each new month. Then, the solidified deep neural network is used to directly predict whether the latest Internet card users will churn in the next month.

[0141] In summary, the user portrait-based Internet card user churn prediction method and system in the present invention determines the key portrait data dimensions and key time series behavior data dimensions of Internet card users by analyzing the identity characteristics and behavior characteristics of Internet card users related to churn; the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card user behavior, and the key time series behavior data dimensions include the number of abnormal behavior days that characterizes the abnormal behavior of Internet card users; key portrait data of corresponding dimensions of different users and key time series behavior data of different periods are obtained from historical data to construct a training data set, and the user churn categories corresponding to the training data in the training data set are labeled; and the constructed deep learning model is trained using the labeled training data in the training data set, and then the trained deep learning model is used to predict user churn. When selecting the characteristic dimensions of the training data, the present invention adds the active entropy and abnormal behavior days of the users to reflect the churn behavior rules of different users, so that the prediction accuracy of the deep learning model trained with the training data is higher.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for predicting Internet card user churn based on user portrait, characterized in that: The following steps are involved: Analyze the identity characteristics and behavioral characteristics of Internet card users related to churn, and determine the key portrait data dimensions and key time series behavioral data dimensions of Internet card users; wherein the key portrait data dimensions include active entropy that characterizes the uncertainty and activity of Internet card users' behaviors, and the key time series behavioral data dimensions include the number of days of abnormal behaviors that characterizes abnormal behaviors of Internet card users; Acquire key portrait data of different dimensions corresponding to different users and key time series behavior data of different periods from historical data to construct a training data set, and annotate the user churn categories corresponding to the training data in the training data set; construct a deep learning model, and use the annotated training data in the training data set to train the deep learning model to obtain a trained deep learning model; Obtain key portrait data and key time series behavior data of the corresponding dimensions of the user to be predicted, input the key portrait data and key time series behavior data of the corresponding dimensions of the user to be predicted into the trained deep learning model, and obtain the churn category of the user to be predicted; The key portrait data dimensions include: gender, age, mobile terminal model, mobile terminal price, mobile terminal usage time, balance, package ID, package level, second card slot operator, online months, single-time suspension times in the past three months, double-time suspension times in the past three months, active entropy based on monthly upstream traffic, active entropy based on monthly downstream traffic, active entropy based on monthly online time, and active entropy based on the number of monthly online records; The key time series behavior data dimensions include: daily downlink traffic and daily uplink traffic, daily Internet access time and, daily Internet access record number and, whether the daily traffic is abnormal, daily number of calling calls, daily calling call duration, daily number of calling numbers, daily number of called calls, daily called call duration, daily number of called numbers and the number of base stations connected to daily calls; The deep learning model is a deep learning model based on a multi-head self-attention mechanism, comprising a feature extraction layer, an embedding layer, an attention layer, a pre-classification layer and a classifier, wherein the embedding layer, the attention layer, the pre-classification layer and the classifier are connected in sequence, and the classifier is also connected to the feature extraction layer; The feature extraction layer is used to extract a reduced-dimensional user portrait key feature matrix from the input key portrait data, and output the key feature matrix to the classifier; The embedding layer is used to convert the input key temporal behavior data and its corresponding category tags into an embedded feature block containing a block embedding vector, a category embedding vector and a position embedding vector, and output the embedded feature block to the attention layer; the category embedding vector is used to learn and mark the target category of each block embedding vector; the position embedding vector is used to mark the positional relationship between each block embedding vector; The attention layer is used to extract the target category of each block embedding vector of the embedded feature block, and perform self-similarity calculation on each extracted block embedding vector by adding the position relationship between each block embedding vector marked by the position embedding vector to obtain the category embedding vector of the embedded feature block; and output the category embedding vector of the embedded feature block to the pre-classification layer; The pre-classification layer is used to pre-classify the category embedding vector of the embedded feature block and output the user churn probability value to the classification layer; The classification layer is used to determine the churn category of the user according to the key feature matrix of the user portrait and the user churn probability value.

2. The method for predicting Internet card user churn based on user portrait according to claim 1, characterized in that: The acquisition of the activity entropy based on the monthly upstream traffic, the activity entropy based on the monthly downstream traffic, the activity entropy based on the monthly online time, and the activity entropy based on the number of monthly online records is achieved by the following steps: For any dimension of data in monthly upstream traffic, monthly downstream traffic, monthly online time, and monthly number of online records, perform the following steps: Convert the data of the dimension into a time series behavior sequence Where d represents the number of days in the month, t i Indicates the statistical value of the user on the dimensional data on the i-th day; Sequence the temporal behavior of the dimension data Divide it into k buckets of equal width, and calculate the activity entropy of the dimensional data according to the following formula: Among them, p i Represents the temporal behavior sequence of the dimension data The proportion of values ​​falling in the i-th bucket, k represents the number of buckets, Represents the temporal behavior sequence of the dimension data length, I is the indicator function, when p i When >0, I=1, otherwise I=0.

3. The method for predicting Internet card user churn based on user portrait according to claim 1, characterized in that: The daily traffic abnormality is determined by the following steps: For any dimension of data in monthly upstream traffic, monthly downstream traffic, monthly online time, and monthly number of online records, perform the following steps: Convert the data of the dimension into a time series behavior sequence Where n is the number of days in the month, t i Indicates the statistical value of the user on the dimensional data on the i-th day; Calculate the sequence of temporal behaviors The first-order forward difference sequence of Among them, d i =t i+1 -t i , d i Represents the first-order forward difference sequence The i-th difference value; For the first-order forward difference sequence Any difference value d in i , determine the d i Is it greater than Q U +1.5IQR or less than Q L -1.5IQR, if the d i Greater than Q U +1.5IQR or less than Q L -1.5IQR, then judge d i There are abnormal traffic on the corresponding days, among which, Q U is the first-order forward difference sequence The upper quartile, Q L is the first-order forward difference sequence The lower quartile, IQR is the interquartile range, Q U With Q L difference.

4. The method for predicting Internet card user churn based on user portrait according to claim 3 is characterized in that: The key temporal behavior data is input into the embedding layer in the form of a matrix, and the embedding layer converts the input key temporal behavior data and its corresponding category labels into an embedding feature block including a block embedding vector, a category embedding vector and a position embedding vector, including the following steps: Assume that the matrix form of the key timing behavior data is the timing behavior feature matrix X∈R N×L , where N is the characteristic dimension of the time series behavior feature matrix, and L is the length of each time series feature; the time series behavior feature matrix X∈R N×L Divide into matrix blocks of size P×Q, and obtain the time series behavior feature matrix X∈R after segmentation M×(P×Q) ; The segmented time series behavior feature matrix X∈R is embedded into E through linear transformation M×(P×Q) Mapped to a dimension of size D, we get the embedding representation X′=XE∈R M×D ; In the embedding representation X′=XE∈R M×D Add the category embedding vector x c ∈R D and the position embedding vector E p ∈R (M+1)×D , the output of the embedding layer is: y0=[x c ;x 1 E;x 2 E;...;x M E]+E p ,E∈R (P×Q)×D ,E p ∈R (M+1)×D Among them, y0 is the output of the embedding layer, and y0∈R (M+1)×D , X i′ Denote the embedding X′=XE∈R M×D The i-th block embedding vector in , i = 1, 2, 3, ..., M, the block embedding vector x i′ =x i The position embedding corresponding to E is The category embedding vector x c ∈R D is a learnable embedding vector with no obvious semantic information at the beginning.

5. The method for predicting Internet card user churn based on user portrait according to claim 4 is characterized in that: The attention layer is a multi-head self-attention layer, which extracts the target category of each block embedding vector of the embedded feature block, and performs self-similarity calculation on the target category of each extracted block embedding vector according to the position relationship between each block embedding vector marked by the position embedding vector to obtain the category embedding vector of the embedded feature block, including the following steps: Calculate the similarity between each block embedding vector and other block embedding vectors and give corresponding weights: Q i =W i Q y0,K i =W i K y0,V i =W i V y0,i=1,...,N Among them, W i Q , W i K and W i V There are three different parameter matrices, which are obtained by random initialization and have dimensions of (M+1)×(M+1). i , K i 、V i They are the query, key and value matrices of the output y0 of the embedding layer respectively; is a query vector q i The dimension of Head is M+1; i To use the i-th parameter matrix W i Q , W i K and W i V The calculated feature submatrix is ​​used for the model to learn relevant information in different representation subspaces; i is the serial number of the self-attention block in the multi-head self-attention layer, i=1, 2, 3..., N, N is the total number of self-attention blocks; Calculate the query matrix Q i with the transpose of the key matrix The inner product between them is the fraction, divided by Normalize, then use the softmax activation function on the score, and then multiply the value matrix V i , get the characteristic submatrix Head i , splicing Head i , multiplied by the projection matrix W O Get MultiHead(y0) as a new sequence feature: MultiHead(y0)=Concat(Head1,...,Head N )W O W O is the projection matrix, the dimension is (N×D)×D, the dimension of MultiHead(y0) is (M+1)×D; The new sequence features are output to the fully connected neural network in the attention layer, and then L layers are stacked, and finally the category embedding vector of the Lth layer is output, where L is determined according to the amount of data and experimental results, and the value range is 3 to 10.

6. The method for predicting Internet card user churn based on user portrait according to claim 5, characterized in that: The pre-classification layer pre-classifies the category embedding vector of the embedded feature block and outputs the user churn probability value to the classification layer, which is achieved by the following formula: p=LayerNorm(MultiHead 0 ) Among them, MultiHead 0 It is the category embedding vector of the Lth layer output by the multi-head self-attention layer.

7. The method for predicting Internet card user churn based on user portrait according to claim 6, characterized in that: The classification layer determines the churn category of the user according to the key feature matrix of the user portrait and the user churn probability value, including the following steps: After feeding the user portrait features and the user churn probability value output by pre-classification to the classifier multi-layer perceptron MLP, training is performed to output the churn probability value p′; Then, according to the threshold filtering, the indicator variable I of whether the user will churn in the next month is output. The threshold filtering formula is: Among them, 0 means that the user will not churn next month, and 1 means that the user will churn next month.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images