Cellular user app usage data synthesis method based on large language model

Abstract

The application belongs to the technical field of data synthesis, and particularly relates to a cellular user App use data synthesis method based on a large language model, which is divided into the following steps: (1) a text encoding method, which encodes real table data into text sequences to convert App table data into text sequence representation; (2) pre-training fine-tuning, which fine-tunes a pre-trained generative large language model using a text data set; and (3) App data sampling synthesis, which samples and generates text sequence data using the fine-tuned pre-trained generative large language model, converts the text sequence data into table data, and obtains a synthesized table data set. The application has the following beneficial effects: the method allows users to control the data generation process through various input prompt settings, users can flexibly define conditions to realize controllable synthesis of App use data according to their own needs, and the quality of the synthesized data and the availability in downstream applications are higher.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data synthesis, and particularly relates to a cellular user App usage data synthesis method based on a large language model (LLM). BACKGROUND

[0002] Recent research data shows that the global mobile application market continues to grow, and as of 2023, there are one billion global smartphone users. The widespread use of Apps has become an integral part of modern life, from social media, online shopping to daily work and entertainment, people increasingly rely on various Apps to meet their needs. With the popularity of smartphones and the rapid development of mobile Internet, the number of App users and their activities and behavior data on Apps have shown explosive growth, including user communication behavior, traffic usage, and application preferences. Therefore, the richness, diversity, and real-time nature of App data have great value for application development and user experience improvement, such as user behavior analysis, App function improvement, decision support, targeted advertising, and user personalized recommendations.

[0003] Although the scale of these data is large, there are still some key challenges in research and practical application development. First, the acquisition and use of real user data pose privacy and security problems, which limit the use of real data by researchers and developers. Second, the cost of collecting and processing large-scale user data is often very high, especially for individual researchers or small teams. In addition, due to the sensitivity and limitations of data, the sharing of real data is limited. In order to solve these problems, people have begun to explore the use of synthetic data to replace real user data for research and application development. Synthetic data is artificial data synthesized from existing data through simulation and generation algorithms, with similar distribution and characteristics as real data. In recent years, with the development and application of large language models, data synthesis technology based on large language models has attracted widespread attention and research. SUMMARY

[0004] The present application aims to provide a cellular user App usage data synthesis method based on a large language model, which solves the challenges of the scarcity of publicly available cellular user App usage data and the complex correlation between user attributes and App traffic when synthesizing cellular user App usage data.

[0005] The present application solves the above challenges through the following technical solution: a cellular user App usage data synthesis method based on a large language model, which is divided into the following steps:

[0006] S1 Text encoding of cellular user App usage data

[0007] Standard pre-trained generative large language models usually take word sequences as input, and the present application proposes a text encoding method specific to a cellular user App usage dataset, which converts each row of data samples in the App table data into a text sequence representation, as follows:

[0008] S1.1 Given an original table dataset D real , which contains p user attribute feature columns (e.g. user ID, age, gender, etc.) and q App usage traffic feature columns (e.g. traffic consumption, etc.), each row represents the corresponding feature value. After preprocessing operations such as outlier filtering and data aggregation, the dataset D' real is obtained, which contains n rows of samples, where the user attribute feature columns are defined as The App usage traffic feature columns are

[0009] S1.2 In order to perform subsequent model training, feature conversion is first performed. Specifically, for each row of data samples in the dataset D' real , it is converted into a text sequence representation, as follows:

[0010] S1.2.1 Let denote the value of the jth user attribute feature of the ith row of data samples, where i ∈ {1, 2, …, n}, j ∈ {1, 2, …, p}, and the user attribute text encoding feature clause is represented as:

[0011]

[0012] S1.2.2 Let denote the set of kth user App usage traffic features of the ith row of data samples, where k ∈ {1, 2, …, q}, and the user App usage traffic text encoding feature clause is represented as:

[0013]

[0014] contains user attribute features and their corresponding values, contains App usage traffic features and their corresponding sets, and each row of data samples in D' real is converted into a text sequence representation t i :

[0015]

[0016] S1.3 Each row of data samples in the dataset D' real is converted into a text sequence representation t iAfter, and add the mark symbol "," respectively, the pseudo position information is introduced into the text sequence representation t i , to identify the position of each data sample in the text sequence representation t real . i .

[0017] In order to ensure the independence of the feature order, a random feature order permutation function Z(·) is designed to randomly arrange the text representation t i , as follows:

[0018] S1.3.1 For the input text sequence representation t , create a random permutation list perm of length p+q, which contains all random numbers from 1 to p+q.

[0019] S1.3.2 Define perm a as the first p elements of perm, and perm b as the last q elements of perm.

[0020] S1.3.3 The function Z(·) randomly permutes the input text sequence representation t i , denoted as Z(t i ):

[0021]

[0022] The text data set T = {Z(t i )} i is obtained after randomly permuting the text sequence representation t i＝1,2,…,n .

[0023] S2 Pre-training generative large language model fine-tuning

[0024] The pre-trained generative large language model is fine-tuned using the text data set T to obtain a large model suitable for App data synthesis. The present application uses the GPT2 model as the pre-trained generative large language model.

[0025] S2.1 Before fine-tuning, obtain the vocabulary W of the pre-trained generative large language model, which is given by the developer of the model during pre-training and is fixed in the parameters of the model. W contains all tokens required by the pre-trained generative large language model in the pre-training data.

[0026] S2.2 Encode each input text sequence t e T using a space-based tokenization method, i.e. divide the sentence by space, and each word is an independent token, to obtain the token sequence ttoken = [w1, w2,..., w l ], l denotes the encoded input text sequence t token ; for each token in t token : w1, w2,..., w l ∈ W. The text dataset T is encoded by row to obtain a dataset T token in the form of a token sequence composed of t token .

[0027] S2.3 Set the hyperparameters required for model fine-tuning, including learning rate β, training rounds epoch, and training batch size batch_size, for example, β = 5e-4, epoch = 25, batch_size = 16.

[0028] S2.4 Input the dataset T token to the pre-trained generative large language model by the training batch size batch_size for model fine-tuning training; specifically as follows:

[0029] S2.4.1 For every batch_size t token = [w1, w2,..., w token ] in the dataset T l , use the conditional autoregressive generation method for iterative training, including the following steps: 1) If there is no input, the model randomly generates w′1; 2) If the input is [w1], the model predicts and generates w′2; 3) If the input is [w1, w2], the model predicts and generates w′3, and so on; if the input is [w1, w2,..., w l-1 ], the model predicts and generates w′ l . In the above steps, w′1, w′2, w′3,..., w′ l represent the generated tokens and all come from the vocabulary W. The conditional autoregressive generation method can be represented as the product of the probability distribution of iteratively generating new tokens with real tokens as conditions:

[0030]

[0031] p(w′ k |w1, w2,..., w k-1 ) represents the probability distribution of inputting t token to the pre-trained generative large language model to iteratively train and generate w′ k by the conditional autoregressive generation method, so as to obtain the product p(t token ) of the probability distribution after iteratively training by the conditional autoregressive generation method by inputting t token to the pre-trained generative large language model.

[0032] S2.4.2 use one-hot encoding method to represent as high-dimensional vector p(w1), p(w2), …, p(w l l l token token

[0033] S2.5 constantly adjust the learning rate β, training rounds epoch, training batch size batch_size and other hyperparameters, through the iterative training of step S2.4, finally get the pre-trained generative large language model M used for App data synthesis after fine-tuning using the text data set T.

[0034] S3 cellular user App usage data synthesis

[0035] S3.1 by setting input prompts, using the fine-tuned model M to sample the feature joint distribution of the table data set D real to generate text sequence data, and converting the text sequence data into table data to obtain the synthesized cellular user App table data set. The feature joint distribution refers to the number distribution of user attribute feature rows and App usage traffic feature rows in D real , denoted by p(v1,…,v n ), where v i represents the i-th feature name. The input prompts include: (1) feature name prompts, input format is "[Feature] is", using model M to sample and generate text sequence data from p(v1,…,v n ) according to the feature name. (2) feature key-value prompts, input format is "[Feature i ] is [Values i ]", subscript i represents v n randomly selected from p(v1,…,v i ), and model M samples and generates text sequence data from the remaining feature joint distribution according to the above prompts.

[0036] S3.2 preprocess the input prompts and encode the input prompts into conditional token sequences [w1, w2, …, w k ​​​​​]input to the model M, k denotes the number of encoded tokens of the input prompt, and its output is logits distribution z of ω, ω denotes the next token to be generated. Specifically, a temperature parameter T > 0 is used to adjust the randomness of ω, and the probability distribution of ω is represented as:

[0037]

[0038] where z i denotes logits of logits distribution z that predict ω as each token in the vocabulary W, is the sum of exponent logits of logits distribution z that predict ω as each token in the vocabulary W, which serves as a normalization factor to ensure that the sum of probabilities of all tokens is 1.

[0039] S3.3 randomly samples the probability distribution p(ω|w1,w2,…,w k ) using the sample() function to select a token from the vocabulary W as the final generation result, obtaining the next token to be generated ω.

[0040] The sample() function is implemented in the following steps:

[0041] S3.3.1 For a given probability distribution p(ω|w1,w2,…,w k ), calculate the cumulative probability of each token by sequentially adding the probability distribution p(ω|w1,w2,…,w k ) in descending order to obtain an increasing cumulative probability distribution.

[0042] S3.3.2 Generate a random number r between 0 and 1.

[0043] S3.3.3 Find the token corresponding to the first probability value greater than the random number r from the cumulative probability distribution, and select the token as the final generation result.

[0044] S3.4 Add ω to the conditional token sequence [w1,w2,…,w k+1 ] as the k+1 token w k , and repeat steps S3.2 to S3.3 to continuously generate new tokens until the conditional token sequence [w1,w2,…,w k]Sampling the k+1th to the lth token, where l represents the number of tokens required to encode the input text sequence t in step S2.2. Finally, all tokens are spliced together to form the input prompt and the token sequence generated by the input prompt, obtaining the synthesized cellular user App usage text sequence data.

[0045] S3.5 Regular expression decoding is performed on the text sequence data to extract user attribute feature column names and App traffic usage feature column names and their corresponding values. According to the decoded features and values, the corresponding table data is constructed, and the synthesized cellular user App usage table data set is obtained.

[0046] The present application has the following beneficial effects:

[0047] (1) The present application provides a cellular user App usage data synthesis method and system based on a large language model. Compared with other data synthesis methods, this method allows users to control the data generation process through multiple input prompt settings. Users can define conditions flexibly to achieve controllable synthesis of cellular user App usage data according to their own needs.

[0048] (2) The present application encodes cellular user App usage data into text, fully utilizes the advantages of large models in text understanding, and generates cellular user App usage data with better context awareness based on training data with context information and semantic association. The quality of the synthesized data is higher and more usable in downstream applications. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is the overall framework diagram of the proposed method;

[0051] Figure 2 is an illustrative example diagram of the proposed method in the process of text encoding and data synthesis:

[0052] (a) is an illustrative example diagram of the proposed text encoding method

[0053] (b) is an illustrative example diagram of cellular user App usage data synthesis;

[0054] Figure 3 is a performance comparison diagram of evaluating the synthesized data on the overall distribution:

[0055] (a) is the JSD value histogram of user attribute features

[0056] (b) is the EMD value histogram of user App traffic features;

[0057] Figure 4 is the performance comparison chart of synthetic data on user classification tasks:

[0058] (a) is the accuracy rate histogram of user classification tasks

[0059] (b) is the AUC value histogram of user classification tasks

[0060] (c) is the F1 score histogram of user classification tasks;

[0061] Figure 5 is the Jaccard similarity histogram of App positioning tasks. DETAILED DESCRIPTION

[0062] The technical solutions in the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0063] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and for the same or similar concepts or processes, some embodiments may not be described again.

[0064] Glossary:

[0065] LLM: LLM refers to "Large Language Model", which means a large language model, usually containing tens of billions of parameters, capable of learning and generating more complex and accurate text.

[0066] Fine-tuning: Fine-tuning refers to further training of LLM using a specific task dataset based on pre-training. Through fine-tuning, LLM can adapt to the requirements of a specific task and improve performance on that task.

[0067] Pre-training: Pre-training refers to the initial training of LLM on large-scale text data. Pre-training usually uses unsupervised learning methods such as autoencoders or masked language models, enabling LLM to learn general features and statistical rules of language.

[0068] To solve the problem of the scarcity of publicly available cellular user App usage data and the existence of complex correlations between user attributes and App usage traffic in the presence of multiple data heterogeneous distributions when synthesizing real and diverse cellular user App usage data, the present application proposes a cellular user App usage data synthesis method based on a large language model, named LLMApp, as shown in Figure 1 The LLMApp consists of three main parts: (1) text encoding method. Text encoding is performed on real table data to convert App usage table data into text sequence representation; (2) pre-training fine-tuning. A pre-trained generative large language model is fine-tuned using a text dataset; (3) App data sampling synthesis. The fine-tuned pre-trained generative large language model is used to sample and generate text sequence data, and the text sequence data is converted into table data to obtain the synthesized table dataset. Specifically, the following steps are taken:

[0069] S1 cellular user App usage data text encoding

[0070] Standard pre-trained generative large language models usually take word sequences as input. The present application proposes a text encoding method specific to the cellular user App usage dataset, which converts each row of data sample in the App table data into text sequence representation, as follows:

[0071] S1.1 Given a table dataset D real , which contains p user attribute feature columns and q App usage traffic feature columns. In a specific embodiment of the present application, 3 user attribute feature columns are included: user ID column "uid", user age column "age", and user gender column "gender", and 4 App usage traffic feature columns: App usage name column "app_label3", App consumption traffic column "app_consume_traffic", App usage days column "app_use_days", and App usage times column "app_use_times".

[0072] Then, D real is de-duplicated by row according to the user ID column "uid" in the user attribute feature column to ensure that each user in D real contains only unique user attribute feature rows. For example, the user with user ID 922***305 in D realThere are 3 repeated user attribute feature rows "922***305, 19, 11", and only 1 row is left after deduplication. At the same time, the App usage traffic feature rows of the same user are aggregated to obtain the App usage traffic feature set. For example, the user with user ID 922***305 contains 3 App usage traffic feature rows: "pinduoduo, 67.02, 30, 2148", "mobile taobao, 30.53, 20, 569", and "iqiyi, 93.45, 28, 1356". After aggregation, 4 App usage traffic feature sets are obtained, which are "pinduoduo, mobile taobao, iqiyi", "67.02, 30.53, 93.45", and "30, 20, 28", "2148, 569, 1356", respectively. The App usage traffic feature set of all users is replaced by the new column to obtain a new table data set D' real There are n rows of data samples, and the user attribute feature column is defined as The App usage traffic feature column is

[0073] S1.2 For each row of data samples in the table data set D' real , it is converted into a text sequence representation, and the specific implementation process is as follows:

[0074] S1.2.1 Let denote the value of the jth user attribute feature of the ith row of data samples, where i∈{1, 2, …, n}, j∈{1, 2, …, p}, and the user attribute text encoding feature clause is denoted as:

[0075]

[0076] S1.2.2 Let denote the kth user App usage traffic feature set of the ith row of data samples, where k∈{1, 2, …, q}, and the user App usage traffic text encoding feature clause is denoted as:

[0077]

[0078] contains user attribute features and their corresponding values, contains App usage traffic features and their corresponding sets, and each row of data samples of D' real is converted into a text sequence representation t i :

[0079]

[0080] Figure 2(a) is an illustrative example of the proposed text encoding method. The App is converted into a data table format into the sequence "uid is 922***305, age is 19, gender is 11, app_label3 is Pinduoduo, Mobile Taobao, iQiyi, app_consume_traffic is 67.02, 30.53, 93.45, app_use_days is 30, 20, 28, app_use_times is 2148, 569, 1356". While word order is crucial in standard natural language sequences, there is no preferred order among the features in this invention.

[0081] S1.3 The dataset D′ is processed through step S1.2. real Each line of data sample in the dataset is converted into a text sequence representation t. i and in and By adding commas to each element, pseudo-positional information is introduced into the text sequence representation t. i In the middle, to identify dataset D′ real Each line of data sample is represented in the text sequence t. i The position in the middle.

[0082] Furthermore, to ensure the independence of feature order, a random feature order permutation function Z(·) is designed to randomly arrange the text representation t. i The steps are as follows:

[0083] S1.3.1 Representation of the input text sequence Create a random permutation list perm of length p+q, containing all random numbers from 1 to p+q.

[0084] S1.3.2 will perm a Defined as the first p elements of perm, perm b Defined as the last q elements of perm.

[0085] The function Z(·) in S1.3.3 represents the input text sequence as t. i Perform random feature order permutation, denoted as Z(t) i ):

[0086]

[0087] Then, we obtain the text sequence representation t. i Text dataset T = {Z(t) after random feature order permutation i )} i＝1,2,…,n .

[0088] S2 pre-training generative large language model fine-tuning

[0089] The pre-training generative large language model is trained on a large amount of unlabeled data and has strong feature extraction and representation capabilities. However, these models cannot be directly applied to specific downstream tasks. Through fine-tuning, the model can better adapt to the data distribution and features of a specific task, thereby improving the performance of the model in synthesizing data.

[0090] The pre-training generative large language model is fine-tuned using a text data set T to obtain a large model suitable for App data synthesis. The GPT2 model is used as the pre-training generative large language model in the embodiments of the present application.

[0091] S2.1 Before fine-tuning, obtain the vocabulary W of the pre-training model, which is given by the developer of the model during pre-training and is fixed in the parameters of the model. W contains all the tokens required by the GPT2 model in the pre-training data, which is represented as follows:

[0092] W=[w1,w2,...,w N ]

[0093] Where w i ∈W represents the i-th token in the vocabulary W, and token is the smallest processing unit in the pre-training generative large language model, including words, subwords, characters or other smaller units. The present application uses word-level tokens, with each token corresponding to a word. For example, in the sentence "I love cats", each word "I", "love", "cats" can be an independent token. N represents the number of tokens in the vocabulary W.

[0094] S2.2 In order to fine-tune the GPT2 model using the text data set T, each row of input text sequence t is encoded into token sequence t token according to the following formula:

[0095] t token =TOKENIZE(t)=[w1,w2,...,w l ],t∈T

[0096] In this context, TOKENIZE(·) represents the process of tokenizing and encoding the input text sequence t. Tokenization is the process of dividing the input text sequence into individual tokens. Common tokenization methods include: space-based partitioning, character-based partitioning, and word-based partitioning. This invention uses space-based partitioning, that is, dividing the sentence into spaces, with each word as an independent token. For example, the sentence "I love cats" can be tokenized as [I,love,cats]. l represents the encoded input text sequence t. token The required number of tokens for t token Each token in the array: w1, w2, ..., w l ∈W. Encode the text dataset T row by row to obtain the result from t token The dataset T consists of a sequence of tokens. token .

[0097] S2.3 Set the hyperparameters required for model fine-tuning, including the learning rate β, the number of training epochs, and the batch size, for example, β = 5e-4, epoch = 25, batch size = 16.

[0098] S2.4 will use dataset T token The training batch size (batch_size) is input into the GPT2 model for fine-tuning training. For dataset T... token Each batch_size contains t token Follow these steps to fine-tune and train the model:

[0099] S2.4.1 In order to input t token To further fine-tune the GPT2 model to generate new token sequences, iterative training using a conditional autoregressive generation method is employed, including the following steps: 1) If there is no input, the model randomly generates w′1; 2) If the input is [w1], the model predicts and generates w′2; 3) If the input is [w1, w2], the model predicts and generates w′3, and so on, if the input is [w1, w2, ..., w l-1 The model predicts and generates w′. l .

[0100] In the above steps, w′1, w′2, w′3, ..., w′ l The generated tokens are all derived from the vocabulary W. The conditional autoregressive generation method can be represented as the product of the probability distributions of generating new tokens by using real tokens as conditions:

[0101]

[0102] where p(w'1) represents the probability distribution of the model randomly generating w'1 without input, p(w'2|w1) represents the probability distribution of the model predicting and generating w'2 under the condition of input [w1], p(w'3|w1, w2) represents the probability distribution of the model predicting and generating w'3 under the condition of input [w1, w2], and p(w'4|w1, w2, w3) represents the probability distribution of the model predicting and generating w'4 under the condition of input [w1, w2, w3]. The product p(t l |w1,w2,…,w l-1 ) of the probability distributions obtained after t l-1 inputs are input into the GPT2 model for conditional autoregressive generation mode training. l token token

[0103] S2.4.2 represents w1, w2,..., w l using one-hot encoding method as high-dimensional vectors p(w1), p(w2),..., p(w l ), calculates the product of p(w1), p(w2),..., p(w l ), and obtains the product p(real) of the real token high-dimensional vector of p(t token ):

[0104]

[0105] The cross-entropy loss L of p(t token ) and p(real) is calculated using the following formula:

[0106]

[0107] where p(real) i and p(t token ) i represent the probability values of p(t token ) and p(real) at the i-th position, respectively.

[0108] According to the cross-entropy loss L, the weight matrix θ of the model neural network layer is updated using the gradient descent algorithm, and the update rule of the gradient descent algorithm is represented as:

[0109]

[0110] where θ e is the model weight matrix θ after the e-th training round iteration, β is the learning rate, represents the gradient of the cross-entropy loss L with respect to the model weight matrix θ.

[0111] ​​​The cross-entropy loss L gradually decreases, and the performance of the model on synthetic data gradually improves.

[0112] S2.5 Through continuous adjustment of learning rate β, training epoch, training batch size, and other hyperparameters, the GPT2 model M for App data synthesis is finally obtained through iterative training in step S2.4 after fine-tuning using the text data set T.

[0113] S3 Cellular user App usage data synthesis

[0114] The fine-tuned model M is used to sample and generate text sequence data, and the text sequence data is converted into table data to obtain the synthesized cellular user App usage data set. Figure 2 The idea of using model M to sample and generate text sequence data is described in (b). Starting from the input prompt predefined by the given model, each token of the text sequence data is generated step by step until the desired length is reached.

[0115] S3.1 Through step S2, the fine-tuned model M using the text data set T is obtained, and the input prompt is set to use the model M to sample and generate text sequence data from the feature joint distribution of the table data set D real . The feature joint distribution refers to the number distribution of user attribute feature rows and App usage traffic feature rows in D real , denoted by p(v1,…,v n ), where v i represents the i-th feature name.

[0116] Preferably, the input prompt includes: (1) feature name prompt, input format is "[Feature] is", using model M to sample and generate text sequence data from p(v1,…,v n ) according to the feature name. (2) Feature key-value prompt, input format is "[Feature i ] is [Values i ]", subscript i represents v n randomly selected from p(v1,…,v i ), and model M samples and generates text sequence data from the remaining feature joint distribution according to the above prompt.

[0117] S3.2 Preprocess the input prompt, which is encoded into a conditional token sequence [w1,w2,…,w k ] input to model M, k represents the number of tokens after encoding the input prompt, and the logits distribution z of the output ω, ω represents the next token to be generated, and z represents the following:

[0118] z = M([w1, w2,..., w k ])

[0119] Specifically, a temperature parameter T > 0 is used to adjust the randomness of the generation of ω, and the probability distribution of ω is represented by normalizing the logits distribution z:

[0120]

[0121] where z i represents the logits in the logits distribution z that predict ω as each token in the vocabulary W, is the sum of the exponential logits in the logits distribution z that predict ω as each token in the vocabulary W, which serves as a normalization factor to ensure that the sum of the probabilities of all tokens is 1.

[0122] The above formula scales and normalizes the logits of each token in the logits distribution z by temperature T. Higher temperature makes the logits distribution more uniform, increases the probability of sampling tokens with low probability, and leads to more diverse output. Lower temperature concentrates the probability on the most likely tokens, making the sampling process more deterministic and leading to more predictable output.

[0123] S3.3 uses the sample() function to randomly sample the probability distribution p(ω|w1,w2,...,w k ) to select a token as the final generation result, and obtains the next token ω to be generated:

[0124] ω = sample(p(ω|w1,w2,...,w k ))

[0125] The sample() function is implemented in the following steps:

[0126] S3.3.1 For a given probability distribution p(ω|w1,w2,...,w k ), calculate the cumulative probability of each token by adding the probability distribution p(ω|w1,w2,...,w k ) in descending order to obtain an increasing cumulative probability distribution.

[0127] S3.3.2 Generate a random number r between 0 and 1.

[0128] S3.3.3 Find the token corresponding to the first probability value greater than the random number r in the cumulative probability distribution, and select the token as the final generation result.

[0129] S3.4 Take ω as the (k + 1)-th token w k+1 and add it to the conditional token sequence [w1, w2, …, w k to form a new conditional token sequence [w1, w2, …, w k , w k+1 . Repeat steps S3.2 to S3.3 to continuously generate new tokens until the (k + 1)-th to the l-th tokens are sampled from the conditional token sequence [w1, w2, …, w k . Denote the sampled token sequence as [w k+1 , w k+2 , …, w l , where l represents the number of tokens required to encode the input text sequence t in step S2.2. Concatenate [w1, w2, …, w k , w k+1 and [w k+1 , w k+2 , …, w l together to form the input prompt and the token sequence generated according to the input prompt, and obtain the synthesized text sequence data.

[0130] In the illustrative embodiment of Figure 2 (b), the model M is used to sample and generate text sequence data similar to ″uid is 922***305, age is 19, gender is 11, app_label3 is Pinduoduo, Mobile Taobao, iQIYI, app_consume_traffic is 67.02, 30.53, 93.45, app_use_days is 30, 20, 28, app_use_times is 2148, 569, 1356″ and ″uid is 748***346, age is 28, gender is 10, app_label3 is WeChat, Mobile Taobao, app_consume_traffic is 20.56, 50.63, app_use_days is 30, 13, app_use_times is 37462, 142″.

[0131] S3.5 The text sequence data is decoded by regular expression to obtain the user attribute feature column name and the App traffic use feature column name and the corresponding values. For example, for the text sequence data of "uid is 922***305, age is 19, gender is 11, app_label3 is PinDuoDuo, mobile Taobao, iQiyi, app_consume_traffic is 67.02, 30.53, 93.45, app_use_days is 30, 20, 28, app_use_times is 2148, 569, 1356", first, the user attribute feature column name and the App traffic use feature column "uid", "age", "gender", "app_label3", "app_consume_traffic", "app_use_days" and "app_use_times" are extracted, then the values corresponding to the App traffic use feature column name are extracted and separated into multiple values according to the separator "、", and the values corresponding to the user attribute column name are extracted and copied according to the number of values corresponding to the App traffic use feature column name, to obtain the extracted values: "922***305 1911 PinDuoDuo 67.02 30.53 30 2148", "922***305 19 11 mobile Taobao 30.53 20 569", "922***305 19 11 iQiyi 93.45 28 1356".

[0132] According to the user attribute feature column name and the App traffic use feature column name and the corresponding values, the corresponding table data is constructed, and the cellular user App use data set synthesized by the use model M is obtained.

[0133] In order to better illustrate the embodiment of the present application, the following experiments are carried out in the embodiment:

[0134] The embodiment of the present application is based on a cellular user App use data set of a large network operator, 26,000 table data of 6102 users using 1129 Apps in November 2020, and experiments are run on a machine equipped with 8 NVIDIA Tian VGPU, each GPU is equipped with 12 GB memory and 60 GB CPU memory, and the experiments use Python 3.8 and PyTorch 1.10 framework. In order to ensure the training efficiency, the training batch size is set to 16, the training round is set to 25, and the initial learning rate is set to 5e-4. These hyperparameters are determined through multiple experiments to achieve the best performance of the model. The specific experimental process is as follows:

[0135] I. Performance index

[0136] The present application uses different evaluation indicators to evaluate data fidelity, utility and privacy, as follows.

[0137] 1) In terms of evaluation of data fidelity, in order to evaluate the fidelity of synthetic data, Earth Mover Distance (EMD) (also known as Wasserstein-1 distance) is used to evaluate the distribution of synthetic data and real data on user App usage traffic, times and days, and Jensen-Shannon divergence (JSD) is used to evaluate the distribution of synthetic data and real data on user ID (uid), age, gender and App name. Since EMD of different data fields has different scales, EMD of each field is normalized to [0.1, 0.9]. Specifically, given two probability distributions P and Q of a set X, EMD is calculated as:

[0138]

[0139] where γ is a parameter representing the quality of transferring each element x in P to each element y in Q, and d(x, y) is the distance between elements x and y in the metric space. JSD is calculated as:

[0140]

[0141] where x represents an element in set X, and P(x) and Q(x) are the probabilities of x in distributions P and Q, respectively.

[0142] 2) In terms of evaluation of data utility, evaluation is performed through two typical downstream applications, i.e. user classification based on personalized recommendation and App positioning based on advertisement placement.

[0143] For user classification, three commonly used indicators are used: Accuracy, AUC and F1-Score, which are calculated as:

[0144]

[0145] where TP (True Positive) represents true positive, which is the number of samples correctly classified as positive by the model; FP (False Positive) represents false positive, which is the number of samples incorrectly classified as positive by the model; TN (True Negative) represents true negative, which is the number of samples correctly classified as negative by the model; FN (False Negative) represents false negative, which is the number of samples incorrectly classified as negative by the model. TPR (True Positive Rate) represents true positive rate, and the calculation formula is Precision is the ratio of the number of true positive samples to the total number of positive samples, and FPR (False Positive Rate) is the false positive rate, calculated as False positive rate is the ratio of the number of false positive samples to the total number of negative samples.

[0146] Precision (Precision) is the ratio of true positive samples predicted by the model to the total number of positive samples, indicating the prediction accuracy of the model, Recall (Recall) is the ratio of true positive samples correctly predicted by the model to the total number of positive samples, indicating the recognition ability of the model to positive samples,

[0147] For the downstream application of App positioning, the evaluation index used is Jaccard Similarity, an index for measuring the similarity between two sets, calculated as follows:

[0148]

[0149] where P and Q are two sets, and Jaccard Similarity is between 0 and 1, and the closer to 1 represents the higher similarity between the two sets.

[0150] 3) In terms of privacy evaluation, distance-based privacy metrics are used, and DCR is used to evaluate privacy, which measures the Euclidean Distance between any synthetic record data point and its closest real neighboring data point. For two points x and y with coordinates (x1, x2, … x n ) and (y1, y2, … y n ), the Euclidean distance is calculated as follows:

[0151]

[0152] II. Comparative experiment

[0153] The technical solutions of the embodiments of the present application and several most advanced data synthesis methods are compared and evaluated in terms of data fidelity, data utility and privacy performance, including the method GReaT based on large language model, the method TabDDPM based on diffusion model, the method STAN based on autoregressive network and the method CTGAN and NetShare based on generative adversarial network. The results are as follows:

[0154] (1) First, compare the data fidelity of the method LLMApp proposed in the present application with other baseline methods. Figure 3 (a) and Figure 3(b) The performance bar charts of JSD and EMD values on the cellular network App user dataset are given, and it can be observed that LLMApp outperforms several baselines on all metrics. For example, the EMD values of LLMApp, GReaT, TabDDPM, CTGAN, NetShare, and STAN on user App usage traffic are about 0.100, 0.321, 0.900, 0.271, 0.460, and 0.605, respectively, which means that LLMApp reduces the EMD by about 68.85%, 88.66%, 63.84%, 78.26%, and 83.47% compared to these baselines. In addition, LLMApp is the lowest in JSD value.

[0155] (2) In addition to data fidelity, data utility is a key evaluation angle, that is, whether the generated data is better than the baseline model in terms of performance for some downstream application, and the application of the application is evaluated from two downstream applications of personalized recommendation and advertisement placement.

[0156] 1) First, a user classification task is designed based on personalized recommendation, that is, relevant user attribute information (such as gender, age) and App usage data (such as App name, usage traffic, and frequency) are input into the classification model to determine which category of users the user belongs to, so as to predict user preferences and provide personalized recommendation services to users.

[0157] The application adopts four commonly used classification algorithms in machine learning: random forest, multilayer perceptron, extreme gradient boosting, and lightweight gradient boosting to realize the classification task of users. Figure 4 (a), Figure 4 (b), Figure 4 (c) are the bar charts of accuracy, AUC value, and F1 score of user classification task evaluation, respectively, and the LLMApp method achieves the best effect.

[0158] 2) The second downstream task is designed from the perspective of advertisement placement. Specifically, for an advertiser, the first consideration is the traffic consumption on the App, which is used as the basis for placing an advertisement on the App.

[0159] Therefore, the cellular user App usage data generated by the data synthesis model has the use of positioning the top n App in terms of traffic consumption among all users for the advertiser, and the need to be evaluated is whether the top n App in terms of traffic consumption in the synthesized data and the real data have relevance. As Figure 5 shown is the bar chart of the Jaccard similarity of App positioning task evaluation, and it can be seen that the data synthesized by the LLMApp method is closest to the real data.

[0160] (3) Finally, the privacy assessment, which uses the distance closest record (DCR) metric to assess the distance-based privacy protection capability of each model, mainly focuses on ensuring that the synthetic data remains high quality while avoiding leaking the original records in the dataset. Specifically, for each synthetic sample, the minimum Euclidean distance to the closest real record is calculated, and then DCR takes the average of these distances for all generated samples. Generally, the larger the DCR value, the worse the quality of the synthetic data, and the smaller the DCR value may mean a greater risk of leaking sensitive information from the training data. The model that achieves a balance between data fidelity (lower EMD) and data privacy (larger DCR) represents a better model. Therefore, the assessment of privacy protection is relative, depending on finding the appropriate balance between data fidelity and privacy. As shown in Table 1 is the DCR of the synthetic data using different methods, it can be seen that the DCR value of the LLMApp method is at a more appropriate size.

[0161] Table 1 DCR of synthetic data using different methods

[0162]

Claims

1. A large language model-based cellular user App usage data synthesis method, characterized by, The method comprises the following steps: S1, the cellular user App uses data text encoding Each row of data samples in the App table data is converted into a text sequence representation, as follows: S1.1 Given an original table dataset D real , which contains p user attribute feature columns and q App usage traffic feature columns, each row represents the corresponding feature value; after the outlier filtering and data aggregation preprocessing operations, the dataset D′ is obtained real , which contains n rows of samples, where the user attribute feature column is defined as The App usage traffic feature column is S1.2 For each data sample in the data set D' real convert it to a text sequence representation as follows: S1.2.1 Let represents the value of the jth user attribute feature of the ith data sample, where i ∈ {1, 2, …, n}, j ∈ {1, 2, …, p}, user attribute text encoding feature clause is represented as: S1.2.2 order represents the set of user App usage traffic features of the i-th row of data samples, where k e {1, 2, …, q}, user App usage traffic text encoding feature clause is represented as: contains the user attribute features and their corresponding values, contains the App usage traffic features and their corresponding sets, and D′ real converts each row of data samples into a text sequence representation ti: S1.3 converting each row of data samples in the data set D' into a text sequence representation t real i i real i ​​​​​​​ To guarantee the independence of the feature order, a random feature order permutation function Z(·) is designed to randomly arrange the text representation t i The steps are as follows: S1.3.1 for an input sequence of text Create a random permutation list perm of length p+q, containing all random numbers from 1 to p+q; S1.3.2 perm a the first p elements of perm, perm b the last q elements of perm; S1.3.3 The function Z(·) represents the input text sequence t i A random feature order permutation is performed, denoted as Z(t i ): Obtaining a text sequence representation t i Performing a random feature order permutation on the text dataset T = {Z(t i )} i＝1 , 2,…,n ; S2, pre-training generative large language model fine-tuning The pre-trained generative large language model is fine-tuned using the text data set T to obtain a large model suitable for App data synthesis, as follows: S2.1, before fine-tuning, obtain the vocabulary W of the pre-trained generative large language model, which is given by the developer of the model during pre-training and is fixed in the model's parameters; W contains all tokens required by the pre-trained generative large language model in the pre-training data; S2.2 Encode each row of input text sequence t e T using a space-based tokenization method, divide the sentence by space, and take each word as an independent token to obtain a token sequence t token = [w1, w2,..., w l ], l represents the number of tokens required after encoding the input text sequence t token ; for each token in t token : w1, w2,..., w l e W; The text data set T is encoded by row to obtain a data set T composed of token sequences in the form of t token token ;​ S2.3, set the hyperparameters required for model fine-tuning, including learning rate β, training rounds epoch and training batch size batch_size; S2.4 will use dataset T token The batch size (batch_size) is used to input data into a pre-trained generative large language model for fine-tuning training; the specific steps are as follows: S2.4.1 For dataset T token Each batch_size contains t token =[w1,w2,...,w l The training uses a conditional autoregressive generation method, which is represented as the product of the probability distributions of generating new tokens by using real tokens as conditions. p(w' | t, θ) = p(w' | w' -1, t, θ) p(w' -1 | t, θ) k |w1,w2,…,w k-1 ) represents the probability distribution of iteratively training the pre-trained generative large language model to generate w' in a conditional autoregressive generation manner with t token input, so as to obtain the product p(t k ) of the probability distribution of iteratively training the pre-trained generative large language model in a conditional autoregressive generation manner with t token input. token ) S2.4.2 w1, w2, …, w l are represented as high-dimensional vectors p(w1), p(w2), …, p(w l ), the product of p(w1), p(w2), …, p(w l ) is calculated to obtain the product p(real) of the real token high-dimensional vector of p(t token ), the cross-entropy loss of p(t token ) and p(real) is calculated, and the weight matrix of the model is updated according to the cross-entropy loss using a gradient descent algorithm; S2.5, continuously adjust the learning rate β, training rounds epoch, and training batch size batch_size hyperparameters, and finally obtain the pre-trained generative large language model M fine-tuned using the text data set T for App data synthesis through the iterative training of step S2.4; S3, cellular user App usage data synthesis S3.1 generating text sequence data using the fine-tuned model M on the table dataset D by setting input prompts, and converting the text sequence data into table data to obtain a synthetic cellular user App usage dataset; the feature joint distribution refers to the number distribution of user attribute feature rows and App usage traffic feature rows in D real , denoted as p(v1, …, v real ), where v n represents the i-th feature name; i ​ S3.2 Preprocess the input prompt, encode the input prompt into a conditional token sequence [w1, w2, …, w k ] into the model M, k represents the number of tokens after the input prompt is encoded, and its output is the logits distribution z of ω, ω represents the next token to be generated; a temperature parameter T > 0 is used to adjust the randomness of ω, and the probability distribution of ω is represented as: where z i denotes logits that predict ω as a token from the vocabulary W, is the sum of the exponential logits in the logits distribution z that predict ω as a token from the vocabulary W, which serves as a normalization factor to ensure that the probabilities of all tokens sum up to 1. S3.3 Randomly sample from the probability distribution p(ω | w1, w2,..., w k ) using the sample() function to select a token from the vocabulary W as the final generated result, resulting in the next token to be generated, ω; The sample() function is implemented as follows: S3.3.1 For a given probability distribution p(ω | w1, w2,..., wn), compute the cumulative probability of each token by sequentially adding the probability distribution p(ω | w1, w2,..., wn) in descending order to get an increasing cumulative probability distribution. k S3.3.2 For a given probability distribution p(ω | w1, w2,..., wn), compute the cumulative probability of each token by sequentially adding the probability distribution p(ω | w1, w2,..., wn) in descending order to get an increasing cumulative probability distribution. k S3.3.2 For a given probability distribution p(ω | w1, w2,..., wn), compute the cumulative probability of each token by sequentially adding the S3.3.2, generate a random number r between 0 and 1; S3.3.3, find the token corresponding to the first probability value greater than the random number r in the cumulative probability distribution, and select the token as the final generation result; S3.4 take ω as the k+1th token w k+1 is added to the conditional token sequence [w1, w2, …, w k ], steps S3.2 to S3.3 are repeatedly executed to continuously generate new tokens until the k+1th to lth tokens are sampled from the conditional token sequence [w1, w2, …, w k ], where l represents the number of tokens required to encode the input text sequence t in step S2.2; finally, all tokens are spliced together to form the input prompt and the token sequence generated according to the input prompt, to obtain the synthesized cellular user App usage text sequence data; S3.5, regular expression decoding of text sequence data to extract user attribute feature column names and App traffic usage feature column names and their corresponding values, and construct the corresponding table data according to the decoded features and values to obtain the synthesized cellular user App usage data set.

2. The method according to claim 1, wherein the method is based on a large language model. In S2, the GPT2 model is used as the pre-trained generative large language model.

3. The method of claim 1, wherein the method is based on a large language model. In S2.3, the learning rate β required for model fine-tuning is 5e-4, the training rounds epoch is 25, and the training batch size batch_size is 16.

4. The method of claim 1, wherein the method is based on a large language model. In S2.4.1, for the data set T token each batch_size t token = [w1, w2,..., w l ], using the conditional autoregressive generation method to iteratively train, including the following steps: 1) if there is no input, the model randomly generates w'1; 2) if the input is [w1], the model predicts and generates w'2; 3) if the input is [w1, w2], the model predicts and generates w'3, and so on; if the input is [w1, w2,..., w l-1 ], the model predicts and generates w' l .

5. The method of claim 1, wherein the method is based on a large language model. In S2.4.2, the cross-entropy loss L for p(t token ) and p(real) is calculated using the following equation: where p(real) i and p(t token ) i represent the probability values of p(t token ) and p(real) at the i-th position, respectively, and N represents the number of tokens in the vocabulary W.

6. The method of claim 1, wherein the method is a method of synthesizing cellular user App usage data based on a large language model. In S2.4.2, the update rule of the gradient descent algorithm is expressed as: where θ e is the model weight matrix θ after the e-th training round iteration, β is the learning rate, denotes the gradient of the cross-entropy loss L with respect to the model weight matrix θ.

7. The method according to claim 1, wherein the method is a method for synthesizing cellular user App usage data based on a large language model. In S3.1, the input prompt includes: (1) a feature name prompt, the input format is "[Feature] is", using the model M to sample from p(v1,…,v n ) according to the feature name and generate text sequence data; (2) a feature key value prompt, the input format is "[Feature i ] is [Values i ]", the subscript i represents v n i selected from p(v1,…,v i ), and the model M samples from the remaining feature joint distribution according to the above prompt and generates text sequence data.

Images