Cellular user App use data synthesis method based on large language model
Through the data synthesis method based on large language model, text encoding and model fine-tuning, the use data of cellular user apps with context-aware capabilities is generated, which solves the problems of data scarcity and heterogeneous distribution, and realizes high-quality data synthesis and flexible generation process.
Patent Information
- Application Number
- CN202410945664.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-15
AI Technical Summary
In the prior art, cellular user Apps have scarce data usage and there are multiple heterogeneous distributions of data in synthetic data. The complex correlation between user attributes and App traffic is difficult to solve, resulting in limited data sharing and usage.
Using a data synthesis method based on a large language model, the table data is converted into text sequence representation through text encoding, and fine-tuned using a pre-trained generative large language model to generate synthetic data with context-aware capabilities, realizing user attributes and App traffic characteristics joint distribution sampling.
It realizes high-quality synthesis of data used by cellular user apps, improves data controllability and the availability of downstream applications, solves the problems of data scarcity and heterogeneous distribution, provides flexible data generation process and higher data fidelity Spend.
Smart Images

Figure CN118890612A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data synthesis, and in particular to a method for synthesizing cellular user App usage data based on a large language model (LLM). Background Art
[0002] Recent research data shows that the global mobile application market continues to grow, and by 2023, there will be one billion smartphone users worldwide. The widespread use of apps has become an indispensable part of modern life. From social media and online shopping to daily work and entertainment, people are increasingly relying on a variety of apps to meet their needs. With the popularity of smartphones and the rapid development of mobile Internet, the number of app users and the user activities and behavior data on apps have exploded, including information such as user communication behavior, traffic usage, and application preferences. Therefore, the inherent advantages of app data, such as richness, diversity, and real-time, are of great value to application development and user experience improvement, such as user behavior analysis, app function improvement, decision support, targeted advertising, and user personalized recommendations.
[0003] Despite the large scale of this data, there are still some key challenges in research and practical application development. First, there are privacy and security issues in the acquisition and use of real user data, which limits the use of real data by researchers and developers. Second, the cost of collecting and processing large-scale user data is often very high, especially for individual researchers or small teams. In addition, due to the sensitivity and restrictiveness of the data, the sharing of real data faces restrictions. To address these issues, people have begun to explore the use of synthetic data instead of real user data for research and application development. Synthetic data refers to artificial data synthesized from existing data through simulation and generation algorithms, with distribution and characteristics similar to real data. In recent years, with the development and application of large language models, data synthesis technology based on large language models has attracted widespread attention and research. Summary of the invention
[0004] The present invention aims to provide a method for synthesizing cellular user App usage data based on a large language model to address the challenges of scarcity of publicly available cellular user App usage data, heterogeneous distribution of various data when synthesizing cellular user App usage data, and complex correlation between user attributes and App traffic.
[0005] The present invention solves the above challenges through the following technical solutions: A method for synthesizing cellular user App usage data based on a large language model, which is divided into the following steps:
[0006] S1 Cellular User App uses data text encoding
[0007] The standard pre-trained generative large language model usually takes a word sequence as input. The present invention proposes a text encoding method specific to the cellular user App usage dataset, which converts each row of data samples in the App table data into a text sequence representation, as follows:
[0008] S1.1 Given an original table dataset D real , which contains p user attribute feature columns (such as user ID, age, gender, etc.) and q App usage traffic feature columns (such as traffic consumption, etc.), and each row represents the corresponding feature value. After preprocessing operations such as outlier screening and data aggregation, the data set D′ is obtained. real , contains a total of n rows of samples, where the user attribute feature column is defined as App usage traffic characteristics are listed as
[0009] S1.2 In order to carry out subsequent model training, feature conversion is first performed. Specifically, for the dataset D′ real For each row of data samples in , convert it into a text sequence representation. The steps are as follows:
[0010] S1.2.1 Order Represents the value of the jth user attribute feature of the i-th row data sample, where i∈{1,2,…,n},j∈{1,2,…,p}, and the user attribute text encoding feature clause It is expressed as:
[0011]
[0012] S1.2.2 Order Represents the set of features of the kth user App usage flow of the i-th row data sample, where k∈{1,2,…,q}, and the user App usage flow text encoding feature clause It is expressed as:
[0013]
[0014] Contains user attribute features and their corresponding values. Contains App usage traffic characteristics and their corresponding sets, and D′ real Each row of data samples is converted into a text sequence representation t i :
[0015]
[0016] S1.3 Through step S1.2, the data set D′ real Each row of data samples in is converted into a text sequence representation t iAfter that, and The marker symbols ","," are added respectively, and the pseudo position information is introduced into the text sequence representation t i In the dataset D′ real Each row of data samples in the text sequence represents t i The position in.
[0017] In order to ensure the independence of feature order, a random feature order permutation function Z(·) is designed to randomly arrange the text representation t i , the steps are as follows:
[0018] S1.3.1 Representation of input text sequence Create a random permutation list perm of length p+q containing all random numbers from 1 to p+q.
[0019] S1.3.2 Perm a Defined as the first p elements of perm, perm b Defined as the last q elements of perm.
[0020] S1.3.3 Function Z(·) represents the input text sequence t i Perform random feature order permutation, expressed as Z(t i ):
[0021]
[0022] Get the text sequence representation t i The text dataset T after random feature order permutation is i )} i=1,2,…,n .
[0023] S2 Pre-trained Generative Large Language Model Fine-tuning
[0024] The pre-trained generative large language model is fine-tuned using the text dataset T to obtain a large model suitable for App data synthesis. The present invention uses the GPT2 model as the pre-trained generative large language model.
[0025] S2.1 Before fine-tuning, obtain the vocabulary W of the pre-trained generative large language model. The vocabulary W is given by the developer of the model during the pre-training process and is fixed in the model parameters. W contains all the tokens required by the pre-trained generative large language model in the pre-training data.
[0026] S2.2 Encode each line of input text sequence t∈T using a space-based tokenization method, that is, divide the sentence by space, and treat each word as an independent token to obtain a token sequence ttoken =[w1,w2,...,w l ], l represents the encoded input text sequence t token The number of tokens required; for t token Each token in: w1,w2,...,w l ∈W. Encode the text dataset T by row, and get token The token sequence data set T token .
[0027] S2.3 sets the hyperparameters required for model fine-tuning, including learning rate β, training round epoch, and training batch size batch_size, for example, β = 5e-4, epoch = 25, batch_size = 16.
[0028] S2.4 Dataset T token According to the training batch size batch_size, it is input into the pre-trained generative large language model for model fine-tuning training; the details are as follows:
[0029] S2.4.1 For dataset T token Each batch_size t token =[w1,w2,...,w l ], iterative training using conditional autoregressive generation, including the following steps: 1) If there is no input, the model randomly generates w′1; 2) If the input is [w1], the model predicts and generates w′2; 3) If the input is [w1,w2], the model predicts and generates w′3, and so on. If the input is [w1,w2,...,w l-1 ], the model predicts and generates w′ l In the above steps, w′1, w′2, w′3, …, w′ l Represents the generated tokens and they all come from the vocabulary W. The conditional autoregressive generation method can be expressed as the product of the probability distribution of iteratively generating new tokens using the real token as a condition:
[0030]
[0031] p(w′ k |w1,w2,…,w k-1 ) means to convert t token Input to the pre-trained generative large language model and iteratively train it to generate w′ using conditional autoregressive generation k The probability distribution of t token The product of the probability distribution p(t token ).
[0032] S2.4.2 Set w1,w2,...,w l Use the one-hot encoding method to represent them as high-dimensional vectors p(w1), p(w2),…, p(w l ), calculate p(w1),p(w2),…,p(w l ) to obtain p(t token )’s real token high-dimensional vector product p(real), calculate p(t token ) and p(real), and updates the weight matrix of the model using the gradient descent algorithm based on the cross entropy loss.
[0033] S2.5 continuously adjusts hyperparameters such as learning rate β, training round epoch, training batch size batch_size, etc. Through the iterative training of step S2.4, we finally obtain the pre-trained generative large language model M for App data synthesis after fine-tuning using the text dataset T.
[0034] S3 Cellular User App Usage Data Synthesis
[0035] S3.1 Use the fine-tuned model M to train the tabular dataset D by setting input prompts real The feature joint distribution sampling generates text sequence data, and converts the text sequence data into table data to obtain a synthetic cellular user App table data set. The feature joint distribution refers to D real The number distribution of user attribute feature rows and App usage traffic feature rows in is represented by p(v1,…,v n ) indicates that v i represents the name of the ith feature. The input prompt includes: (1) feature name prompt, the input format is "[Feature]is", using model M to select from p(v1,…,v n ) to sample and generate text sequence data. (2) Feature key value prompt, the input format is ″[Feature i ]is[Values i ]", the subscript i represents the number of n ) Randomly select v i , model M samples from the joint distribution of the remaining features according to the above prompts and generates text sequence data.
[0036] S3.2 preprocesses the input prompt and encodes the input prompt into a conditional token sequence [w1,w2,…,w k] is input to the model M, k represents the number of tokens after the input prompt is encoded, and its output is the logits distribution z of ω, where ω represents the next token to be generated. Specifically, a temperature parameter T>0 is used to adjust the randomness of ω. By normalizing the logits distribution z, the probability distribution of ω is expressed as:
[0037]
[0038] Among them, z i represents the logits distribution z that predicts ω as the logits of each token in the vocabulary W, is the sum of the exponential logits from the logits distribution z that predicts ω as each token in the vocabulary W. It acts as a normalization factor to ensure that the sum of the probabilities of all tokens is 1.
[0039] S3.3 Use the sample() function to sample the probability distribution p(ω|w1,w2,…,w k ) is randomly sampled to select a token from the vocabulary W as the final generation result to obtain the next token ω to be generated.
[0040] The sample() function is implemented as follows:
[0041] S3.3.1 For a given probability distribution p(ω|w1,w2,…,w k ), calculate the cumulative probability of each token by taking the probability distribution p(ω|w1,w2,…,w k ) are arranged in descending order and then added up sequentially to obtain an increasing cumulative probability distribution.
[0042] S3.3.2 Generate a random number r between 0 and 1.
[0043] S3.3.3 Find the first token corresponding to the probability value greater than the random number r from the cumulative probability distribution, and select the token as the final generated result.
[0044] S3.4 takes ω as the k+1th token w k+1 Add to the conditional token sequence [w1,w2,…,w k ], repeat steps S3.2 to S3.3 to continuously generate new tokens until the conditional token sequence [w1,w2,…,w k] Sample the k+1th to lth tokens, where l represents the number of tokens required to encode the input text sequence t in step S2.2. Finally, all tokens are concatenated together to form the input prompt and the token sequence generated according to the input prompt, and the synthesized cellular user App usage text sequence data is obtained.
[0045] S3.5 performs regular expression decoding on the text sequence data to extract the user attribute feature column names and App traffic usage feature column names and their corresponding values. Based on the decoded features and values, the corresponding tabular data is constructed to obtain a synthetic cellular user App usage tabular data set.
[0046] The present invention has the following beneficial effects:
[0047] (1) The present invention provides a method and system for synthesizing cellular user App usage data based on a large language model. Compared with other data synthesis methods, this method allows users to probabilistically control the data generation process through a variety of input prompt settings. Users can flexibly define conditions according to their own needs to achieve controllable synthesis of cellular user App usage data.
[0048] (2) The present invention fully utilizes the advantages of large models in text understanding by textually encoding cellular user App usage data, and generates more context-aware cellular user App usage data based on training data with contextual information and semantic associations. The quality of the synthesized data and its usability in downstream applications are higher. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0050] Figure 1 It is the overall framework diagram of the method proposed in the present invention;
[0051] Figure 2 This is an illustrative example diagram of the proposed method in the text encoding and data synthesis process:
[0052] (a) is an illustrative example diagram of the proposed text encoding method.
[0053] (b) is an illustrative example graph of the synthesis of cellular user App usage data;
[0054] Figure 3 This is a performance comparison chart for evaluating synthetic data on the overall distribution:
[0055] (a) is the JSD value histogram of user attribute features
[0056] (b) is the EMD value bar graph of the user's App traffic characteristics;
[0057] Figure 4 This is a performance comparison chart for evaluating synthetic data on user classification tasks:
[0058] (a) is the accuracy bar graph of the user classification task
[0059] (b) is the AUC value histogram of the user classification task
[0060] (c) is the F1 score histogram of the user classification task;
[0061] Figure 5 It is the Jaccard similarity histogram of the App positioning task. DETAILED DESCRIPTION
[0062] The following is a clear and complete description of the technical solutions in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0064] Glossary:
[0065] LLM: LLM stands for "Large Language Model", which refers to a large language model that usually contains billions of parameters and can learn and generate more complex and accurate texts.
[0066] Fine-tuning: Fine-tuning means further training the LLM using a task-specific dataset based on pre-training. Through fine-tuning, the LLM can adapt to the requirements of a specific task and improve its performance on that task.
[0067] Pre-training: Pre-training refers to the process of initial training of LLM on large-scale text data. Pre-training usually uses unsupervised learning methods such as autoencoders or masked language models to enable LLM to learn the general characteristics and statistical laws of language.
[0068] In order to solve the problem of scarcity of publicly available cellular user App usage data and the complex correlation between user attributes and App traffic encountered when synthesizing real diverse cellular user App usage data, this paper proposes a cellular user App usage data synthesis method based on a large language model, which is named LLMApp. Figure 1 As shown in the figure, LLMApp consists of three main parts: (1) Text encoding method. Text encoding is performed on real tabular data, and the tabular data used by the App is converted into a text sequence representation; (2) Pre-training and fine-tuning. The pre-trained generative large language model is fine-tuned using a text dataset; (3) App data sampling and synthesis. The fine-tuned pre-trained generative large language model is used to sample and generate text sequence data, and the text sequence data is converted into tabular data to obtain a synthesized tabular dataset. Specifically, it is divided into the following steps:
[0069] S1 Cellular User App uses data text encoding
[0070] The standard pre-trained generative large language model usually takes a word sequence as input. The present invention proposes a text encoding method specific to the cellular user App usage dataset, which converts each row of data samples in the App table data into a text sequence representation, as follows:
[0071] S1.1 Given a table data set D real , which includes p user attribute feature columns and q App usage traffic feature columns. In a specific embodiment of the present invention, it includes 3 user attribute feature columns: user ID column "uid", user age column "age", user gender column "gender" and 4 App usage traffic feature columns: App usage name column "app_label3", App consumption traffic column "app_consume_traffic", App usage days column "app_use_days", App usage times column "app_use_times".
[0072] Then, for D real According to the user ID column "uid" in the user attribute feature column, duplicate rows are removed to ensure that real Each user in D contains only one unique user attribute feature row. For example, the user with user ID 922***305 in D realThere are 3 repeated user attribute feature rows "922***305, 19, 11", and only 1 is left after deduplication. At the same time, the App usage traffic feature rows of the same user are aggregated to obtain the App usage traffic feature set. For example, the user with user ID 922***305 mentioned above contains 3 App usage traffic feature rows: "Pinduoduo, 67.02, 30, 2148", "Mobile Taobao, 30.53, 20, 569" and "iQiyi, 93.45, 28, 1356". After aggregation, 4 App usage traffic feature sets are obtained, namely "Pinduoduo, Mobile Taobao, iQiyi", "67.02, 30.53, 93.45" and "30, 20, 28", "2148, 569, 1356". The App usage traffic feature sets of all users are used as new columns to replace the previous user App usage traffic feature columns to obtain a new table data set D′ real There are n rows of data samples, where the user attribute feature column is defined as App usage traffic characteristics are listed as
[0073] S1.2 For the tabular data set D′ real Each row of data samples in is converted into a text sequence representation. The specific implementation process is as follows:
[0074] S1.2.1 Order Represents the value of the jth user attribute feature of the i-th row data sample, where i∈{1,2,…,n},j∈{1,2,…,p}, and the user attribute text encoding feature clause It is expressed as:
[0075]
[0076] S1.2.2 Order Represents the set of features of the kth user App usage flow of the i-th row data sample, where k∈{1,2,…,q}, and the user App usage flow text encoding feature clause It is expressed as:
[0077]
[0078] Contains user attribute features and their corresponding values. Contains App usage traffic characteristics and their corresponding sets, and D′ real Each row of data samples is converted into a text sequence representation t i :
[0079]
[0080] Figure 2(a) is an illustrative example of the proposed text encoding method, in which the App usage data table is converted into the sequence of "uid is 922***305, age is 19, gender is 11, app_label3 is Pinduoduo, Taobao Mobile, iQiyi, app_consume_traffic is 67.02, 30.53, 93.45, app_use_days is 30, 20, 28, app_use_times is 2148, 569, 1356". Although word order is crucial in standard natural language sequences, in the present invention, there is no preferred order between the various features.
[0081] S1.3 Through step S1.2, the data set D′ real Each row of data samples in is converted into a text sequence representation t i , and in and The marker symbol "," is added to the text sequence representation. i In the dataset D′ real Each row of data samples in the text sequence represents t i The position in.
[0082] In addition, in order to ensure the independence of feature order, a random feature order permutation function Z(·) is designed to randomly arrange the text representation t i ; The steps are as follows:
[0083] S1.3.1 Representation of input text sequence Create a random permutation list perm of length p+q containing all random numbers from 1 to p+q.
[0084] S1.3.2 Perm a Defined as the first p elements of perm, perm b Defined as the last q elements of perm.
[0085] S1.3.3 Function Z(·) represents the input text sequence t i Perform random feature order permutation, expressed as Z(t i ):
[0086]
[0087] Then, we get the text sequence representation t i The text dataset T after random feature order permutation is i )} i=1,2,…,n .
[0088] S2 Pre-trained Generative Large Language Model Fine-tuning
[0089] Pre-trained generative large language models are trained on a large amount of unlabeled data and have strong feature extraction and representation capabilities. However, these models cannot be directly applied to specific downstream tasks. Through fine-tuning, the model can be better adapted to the data distribution and characteristics of a specific task, thereby improving the performance of the model's synthetic data.
[0090] The pre-trained generative large language model is fine-tuned using the text dataset T to obtain a large model suitable for App data synthesis. The embodiment of the present invention uses the GPT2 model as the pre-trained generative large language model.
[0091] Before fine-tuning S2.1, obtain the vocabulary W of the pre-trained model. The vocabulary W is given by the developer of the model during the pre-training process and is fixed in the model parameters. W contains all the tokens required by the GPT2 model in the pre-training data, expressed as follows:
[0092] W = [w1, w2, ..., w N ]
[0093] Among them, w i ∈W represents the i-th token in the vocabulary W. A token is the smallest processing unit in a pre-trained generative language model, including a word, a subword, a character, or other smaller units. The present invention uses word-level tokens, where one token corresponds to one word. For example, in the sentence "I love cats", each word "I", "love", and "cats" can be used as an independent token. N represents the number of tokens in the vocabulary W.
[0094] S2.2 To fine-tune the GPT2 model using the text dataset T, each line of input text sequence t is encoded into a token sequence t according to the following formula token :
[0095] t token =TOKENIZE(t)=[w1,w2,...,w l ], t∈T
[0096] Wherein, TOKENIZE(·) represents the process of tokenizing the input text sequence t. Tokenization is the process of dividing the input text sequence into individual tokens. Common tokenization methods include: space-based division, character-based division, and subword-based division. The present invention adopts space-based division, that is, the sentence is divided by space, and each word is a separate token. For example, the sentence "I love cats" can be tokenized as [I, love, cats]. l represents the encoded input text sequence t token The number of tokens required for t token Each token in: w1,w2,...,w l ∈W. Encode the text dataset T by row, and get token The token sequence data set T token .
[0097] S2.3 sets the hyperparameters required for model fine-tuning, including learning rate β, training round epoch, and training batch size batch_size, for example, β = 5e-4, epoch = 25, batch_size = 16.
[0098] S2.4 Dataset T token According to the training batch size batch_size, it is input into the GPT2 model for fine-tuning training. For the dataset T token Each batch_size t token , follow the steps below to fine-tune the model:
[0099] S2.4.1 In order to pass the input t token To further fine-tune the GPT2 model to generate new token sequences, the conditional autoregressive generation method is used for iterative training, including the following steps: 1) If there is no input, the model randomly generates w′1; 2) If the input is [w1], the model predicts and generates w′2; 3) If the input is [w1,w2], the model predicts and generates w′3, and so on. l-1 ], the model predicts and generates w′ l .
[0100] In the above steps, w′1,w′2,w′3,…,w′ l Represents the generated tokens and they all come from the vocabulary W. The conditional autoregressive generation method can be expressed as the product of the probability distribution of iteratively generating new tokens using the real token as a condition:
[0101]
[0102] Among them, p(w′1) represents the probability distribution of the model randomly generating w′1 under the condition of no input, p(w′2|w1) represents the probability distribution of the model predicting and generating w′2 under the condition of input [w1], p(w′3|w1,w2) represents the probability distribution of the model predicting and generating w′3 under the condition of input [w1,w2], and p(w′ l |w1,w2,…,w l-1 ) means that in the input [w1,w2,...,w l-1 ] under the condition that the model predicts and generates w′ l The probability distribution of t token The product of the probability distribution p(t token ).
[0103] S2.4.2 Set w1,w2,...,w l Use the one-hot encoding method to represent them as high-dimensional vectors p(w1), p(w2),…, p(w l ), calculate p(w1),p(w2),…,p(w l ) to obtain p(t token )’s real token high-dimensional vector product p(real):
[0104]
[0105] Use the following formula to calculate p(t token ) and p(real) cross entropy loss L:
[0106]
[0107] Among them, p(real) i and p(t token ) i Respectively represent p(t token ) and the probability value of p(real) at the i-th position.
[0108] According to the cross entropy loss L, the gradient descent algorithm is used to update the weight matrix θ of the model neural network layer. The update rule of the gradient descent algorithm is expressed as:
[0109]
[0110] Among them, θ e is the model weight matrix θ after the e-th training round iteration, β is the learning rate, represents the gradient of the cross entropy loss L with respect to the model weight matrix θ.
[0111] After repeated iterative training for epochs, the cross entropy loss L gradually decreases, and the performance of the model on synthetic data gradually improves.
[0112] S2.5 continuously adjusts hyperparameters such as learning rate β, training round epoch, training batch size batch_size, and through iterative training in step S2.4, finally obtains the GPT2 model M for App data synthesis after fine-tuning using the text dataset T.
[0113] S3 Cellular User App Usage Data Synthesis
[0114] Use the fine-tuned model M to sample and generate text sequence data, and convert the text sequence data into tabular data to obtain a synthetic cellular user App usage dataset. Figure 2 (b) illustrates the idea of using model M to sample and generate text sequence data, starting from the input prompt predefined by the given model, and gradually generating each token of the text sequence data until the required length is reached.
[0115] S3.1 Through step S2, we obtain the model M fine-tuned using the text dataset T. We use the model M to train the table dataset D by setting input prompts. real The feature joint distribution sampling generates text sequence data. The feature joint distribution refers to D real The number distribution of user attribute feature rows and App usage traffic feature rows in is represented by p(v1,…,v n ) indicates that v i Represents the name of the i-th feature.
[0116] Preferably, the input prompt includes: (1) a feature name prompt, the input format is "[Feature]is", using the model M to select the feature name from p(v1,…,v n ) to sample and generate text sequence data. (2) Feature key value prompt, the input format is "[Feature i ]is[Values i ]", the subscript i represents the number of n ) Randomly select v i , model M samples from the joint distribution of the remaining features according to the above prompts and generates text sequence data.
[0117] S3.2 preprocesses the input prompt, which is encoded into a conditional token sequence [w1,w2,…,w k ] is input to the model M, k represents the number of tokens after the input prompt is encoded, and its output is the logits distribution z of ω, ω represents the next token to be generated, and z is expressed as follows:
[0118] z=M([w1,w2,…,w k ])
[0119] Specifically, a temperature parameter T>0 is used to adjust the randomness of generating ω. By normalizing the logits distribution z, the probability distribution of ω is expressed as:
[0120]
[0121] Among them, z i represents the logits distribution z that predicts ω as the logits of each token in the vocabulary W, is the sum of the exponential logits from the logits distribution z that predicts ω as each token in the vocabulary W. It acts as a normalization factor to ensure that the sum of the probabilities of all tokens is 1.
[0122] The above formula scales and normalizes the logits of each token in the logits distribution z by temperature T. A higher temperature will make the logits distribution more uniform, increase the probability of sampling tokens with low probability, and lead to more diverse outputs. A lower temperature will concentrate the probability on the most likely tokens, making the sampling process more deterministic and leading to more predictable outputs.
[0123] S3.3 Use the sample() function to sample the probability distribution p(ω|w1,w2,…,w k ) is randomly sampled to select a token as the final generated result and obtain the next token ω to be generated:
[0124] ω=sample(p(ω|w1,w2,…,w k ))
[0125] The sample() function is implemented as follows:
[0126] S3.3.1 For a given probability distribution p(ω|w1,w2,…,w k ), calculate the cumulative probability of each token by taking the probability distribution p(ω|w1,w2,…,w k ) are arranged in descending order and then added up sequentially to obtain an increasing cumulative probability distribution.
[0127] S3.3.2 Generate a random number r between 0 and 1.
[0128] S3.3.3 Find the first token corresponding to the probability value greater than the random number r from the cumulative probability distribution, and select the token as the final generated result.
[0129] S3.4 takes ω as the k+1th token w k+1 Add to the conditional token sequence [w1,w2,…,w k ] as a new conditional token sequence [w1,w2,…,w k ,w k+1 ], repeat steps S3.2 to S3.3 to continuously generate new tokens until the conditional token sequence [w1,w2,…,w k ] Sample the k+1th to lth tokens, and record the sampled token sequence as [w k+1 ,w k+2 ,…,w l ], where l represents the number of tokens required to encode the input text sequence t in step S2.2. k ,w k+1 ] and [w k+1 ,w k+2 ,…,w l ] are spliced together to form the input prompt and the token sequence generated according to the input prompt, and the synthetic text sequence data is obtained.
[0130] exist Figure 2 In the illustrative embodiment of (b), model M is used to sample from the joint distribution of features to generate text sequence data such as "uid is 922***305, age is 19, gender is 11, app_label3 is Pinduoduo, Taobao Mobile, iQiyi, app_consume_traffic is 67.02, 30.53, 93.45, app_use_days is 30, 20, 28, app_use_times is 2148, 569, 1356" and "uid is 748***346, age is 28, gender is 10, app_label3is WeChat, Taobao Mobile, app_consume_traffic is 20.56, 50.63, app_use_days is 30, 13, app_use_times is 37462, 142".
[0131] S3.5 performs regular expression decoding on the text sequence data to obtain the user attribute feature column names and App traffic usage feature column names and their corresponding values. For example, for the line of text sequence data "uid is 922***305, age is 19, gender is 11, app_label3 is Pinduoduo, Taobao Mobile, iQiyi, app_consume_traffic is 67.02, 30.53, 93.45, app_use_days is 30, 20, 28, app_use_times is 2148, 569, 1356", first extract the user attribute feature column name and App traffic usage feature columns "uid", "age", "gender", "app_label3", "app_consume_traffic", "app_use_days", and "app_use_times", then extract the values corresponding to the App traffic usage feature column names and separate them into multiple values according to the separator ",", then extract the values corresponding to the user attribute column names and copy them according to the number of values corresponding to the App traffic usage feature column names, and obtain the extracted value: "922***305 1911 Pinduoduo 67.02 30.53 302148″、″922***305 19 11 Taobao mobile 30.53 20 569″、″922***30519 11 iQiyi 93.45 281356″.
[0132] According to the user attribute feature column names and App traffic usage feature column names and their corresponding values, the corresponding tabular data is constructed, and the cellular user App usage dataset synthesized using model M is obtained.
[0133] In order to better illustrate the embodiment of the present invention, the following experiments were conducted in this embodiment:
[0134] The embodiment of the present invention is based on a cellular user App usage dataset of a large network operator. In November 2020, 260,000 tabular data used by 6,102 users on 1,129 apps were run on a machine equipped with 8 NVIDIA Tian VGPUs, each GPU is equipped with 12GB of memory and 60GB of CPU memory. The experiment uses Python 3.8 and PyTorch 1.10 frameworks. In order to ensure training efficiency, the training batch size is set to 16, the training rounds are set to 25, and the initial learning rate is set to 5e-4. These hyperparameters are determined through multiple experiments to achieve the best performance of the model. The specific experimental process is:
[0135] 1. Performance indicators
[0136] The present invention uses different evaluation metrics to assess data fidelity, practicality, and privacy, as shown below.
[0137] 1) In terms of data fidelity evaluation, in order to evaluate the fidelity of synthetic data, the Earth Mover Distance (EMD) (also known as Wasserstein-1 distance) is used to evaluate the distribution of synthetic data and real data on user App usage flow, number of times and days, and Jensen-Shannon Divergence (JSD) is used for the distribution of synthetic data and real data on user ID (uid), age, gender and App name. Since the EMD of different data fields has different scales, the EMD of each field is normalized to [0.1, 0.9]. Specifically, given two probability distributions P and Q of set X, the EMD is calculated as:
[0138]
[0139] Where γ is a parameter representing the quality of the transfer from each element x in P to each element y in Q, and d(x,y) is the distance between elements x and y in the metric space. JSD is calculated as:
[0140]
[0141] Where x represents an element in the set X, P(x) and Q(x) are the probabilities of x in the distribution P and Q respectively.
[0142] 2) In terms of data utility evaluation, it is evaluated through two typical downstream applications, namely user classification based on personalized recommendations and App positioning based on advertising.
[0143] For user classification, three commonly used indicators are used: Accuracy, AUC and F1-Score, calculated as follows:
[0144]
[0145] Among them, TP (True Positive) refers to the true positive example, which refers to the number of samples correctly classified as positive examples by the model; FP (False Positive) refers to the false positive example, which refers to the number of samples incorrectly classified as positive examples by the model; TN (True Negative) refers to the true negative example, which refers to the number of samples correctly classified as negative examples by the model; FN (False Negative) refers to the false negative example, which refers to the number of samples incorrectly classified as negative examples by the model. TPR (True Positive Rate) refers to the true positive rate, and the calculation formula is It indicates the proportion of samples correctly classified as positive examples to all positive examples; FPR (False Positive Rate) indicates the false positive rate, which is calculated as It indicates the proportion of samples that are misclassified as positive examples to all negative examples.
[0146] Precision refers to the proportion of true positive examples predicted by the model, indicating the accuracy of the model's prediction. Recall refers to the proportion of actual positive examples that are correctly predicted as positive examples by the model, indicating the model's ability to identify positive examples.
[0147] For downstream applications of App positioning, the evaluation indicator used is Jaccard Similarity, which is an indicator used to measure the similarity between two sets. It is calculated as follows:
[0148]
[0149] Among them, P and Q are two sets, and the Jaccard similarity is between 0 and 1. The closer it is to 1, the higher the similarity between the two sets.
[0150] 3) In terms of privacy assessment, a distance-based privacy metric is used to evaluate privacy using DCR, which is calculated by measuring the Euclidean distance between any synthetically recorded data point and its closest real neighboring data point. For the coordinates of two points x and y (x1, x2, ... x n ) and (y1,y2,…y n ), the Euclidean distance is calculated as follows:
[0151]
[0152] 2. Comparative Experiment
[0153] The technical solution of the embodiment of the present invention and several state-of-the-art data synthesis methods are compared and evaluated in terms of data fidelity, data utility, and privacy performance, including GReaT, a method based on a large language model, TabDDPM, a method based on a diffusion model, STAN, a method based on an autoregressive network, and CTGAN and NetShare, methods based on a generative adversarial network. The results are as follows:
[0154] (1) First, the data fidelity of the proposed method LLMApp is compared with other baseline methods. Figure 3 (a) and Figure 3(b) shows the performance histogram of JSD and EMD values on the cellular network App user dataset. It can be observed that LLMApp outperforms several baseline methods in almost all indicators. For example, the EMD values of LLMApp, GReaT, TabDDPM, CTGAN, NetShare, and STAN on user App usage traffic are approximately 0.100, 0.321, 0.900, 0.271, 0.460, and 0.605, respectively, which means that compared with these baselines, LLMApp reduces EMD by approximately 68.85%, 88.66%, 63.84%, 78.26%, and 83.47%, respectively. In addition, LLMApp has the lowest JSD value.
[0155] (2) In addition to data fidelity, data utility is a key evaluation angle, that is, whether the performance of the generated data is better than that of the baseline model when used for certain downstream applications. The present invention evaluates data utility from two downstream applications, personalized recommendation and advertising.
[0156] 1) First, a user classification task is designed based on personalized recommendation. The relevant user attribute information (such as gender, age) and App usage data (such as App name, usage traffic and frequency) are input into the classification model to determine which category the user belongs to, so as to predict user preferences and provide users with personalized recommendation services.
[0157] The present invention uses four commonly used classification algorithms in machine learning: random forest, multi-layer perceptron, extreme gradient boosting, and lightweight gradient boosting to achieve the user's classification task. Figure 4 (a) Figure 4 (b) Figure 4 (c) are the bar graphs of accuracy, AUC value and F1 score of user classification task evaluation, respectively. The LLMApp method achieved the best results.
[0158] 2) The second downstream task is designed from the perspective of advertising. Specifically, for advertisers, if they want to place an ad on a certain app, the first factor they consider is the traffic consumption on the app, which is used as the basis for placing the ad.
[0159] Therefore, the cellular user app usage data generated by the data synthesis model is used by advertisers to locate the top n apps in terms of traffic consumption among all users. What needs to be evaluated is whether there is a correlation between the top n apps in terms of traffic consumption in the synthetic data and the top n apps in terms of traffic consumption in the real data. Figure 5 The figure shows the Jaccard similarity bar chart of the App positioning task evaluation. It can be seen that the data synthesized by the LLMApp method is closest to the real data.
[0160] (3) Finally, privacy evaluation is performed. The distance to nearest record (DCR) metric is used to evaluate the distance-based privacy protection capability of each model. This metric focuses on ensuring that the synthetic data maintains high quality while avoiding leaking the original records in the dataset. Specifically, for each synthetic sample, the minimum Euclidean distance to the nearest real record is calculated. Then, DCR takes the average of these distances for all generated samples. In general, the larger the DCR value, the worse the quality of the synthetic data, while the smaller the DCR value may mean a greater risk of leaking sensitive information from the training data. Models that achieve a balance between data fidelity (lower EMD) and data privacy (larger DCR) represent better models. Therefore, evaluating privacy protection is relative and depends on finding an appropriate balance between data fidelity and privacy. Table 1 shows the DCR of synthetic data using different methods. It can be seen that the DCR value of the LLMApp method is at a more appropriate size.
[0161] Table 1 DCR of synthetic data using different methods
[0162]
Claims
1. A method for synthesizing cellular user App usage data based on a large language model, characterized in that: The method is divided into the following steps: S1 Cellular User App uses data text encoding The standard pre-trained generative large language model usually takes a word sequence as input. The present invention proposes a text encoding method specific to the cellular user App usage dataset, which converts each row of data samples in the App table data into a text sequence representation, as follows: S1.1 Given an original table dataset D real , which contains p user attribute feature columns and q App usage traffic feature columns, each row represents the corresponding feature value; after outlier screening and data aggregation preprocessing operations, the data set D′ is obtained real , contains a total of n rows of samples, where the user attribute feature column is defined as App usage traffic characteristics are listed as S1.2 In order to carry out subsequent model training, feature conversion is first performed. Specifically, for the dataset D′ real For each row of data samples in , convert it into a text sequence representation. The steps are as follows: S1.2.1 Order Represents the value of the jth user attribute feature of the i-th row data sample, where i∈{1,2,…,n},j∈{1,2,…,p}, and the user attribute text encoding feature clause It is expressed as: S1.2.2 Order Represents the set of features of the kth user App usage flow of the i-th row data sample, where k∈{1,2,…,q}, and the user App usage flow text encoding feature clause It is expressed as: Contains user attribute features and their corresponding values. Contains App usage traffic characteristics and their corresponding sets, and D′ real Each row of data samples is converted into a text sequence representation t i : S1.3 Through step S1.2, the data set D′ real Each row of data samples in is converted into a text sequence representation t i After that, and The marker symbols ","," are added respectively, and the pseudo position information is introduced into the text sequence representation t i In the dataset D′ real The position of each row of data samples in the text sequence representation ti; In order to ensure the independence of feature order, a random feature order permutation function Z(·) is designed to randomly arrange the text representation t i , the steps are as follows: S1.3.1 Representation of input text sequence Create a random permutation list perm of length p+q, which contains all random numbers from 1 to p+q; S1.3.2 Perm a Defined as the first p elements of perm, perm b Defined as the last q elements of perm; S1.3.3 Function Z(·) represents the input text sequence t i Perform random feature order permutation, expressed as Z(t i ): Get the text sequence representation t i The text dataset T after random feature order permutation is i )} i=1,2,…,n ; S2 Pre-trained Generative Large Language Model Fine-tuning Use the text dataset T to fine-tune the pre-trained generative large language model to obtain a large model suitable for App data synthesis, as follows: S2.1 Before fine-tuning, obtain the vocabulary W of the pre-trained generative large language model. The vocabulary W is given by the developer of the model during the pre-training process and is fixed in the model parameters. W contains all the tokens required by the pre-trained generative large language model in the pre-training data. S2.2 Encode each line of input text sequence t∈T using a space-based tokenization method, that is, divide the sentence by space, and treat each word as an independent token to obtain a token sequence t token =[w1,w2,...,w l ], l represents the encoded input text sequence t token The number of tokens required; for t token Each token in: w1,w2,...,w l ∈W; Encode the text dataset T row by row, and get the token The token sequence data set T token; S2.3 sets the hyperparameters required for model fine-tuning, including learning rate β, training round epoch, and training batch size batch_size; S2.4 Dataset T token According to the training batch size batch_size, it is input into the pre-trained generative large language model for model fine-tuning training; the details are as follows: S2.4.1 For dataset T token Each batch_size t token =[w1,w2,...,w l ], iterative training is performed using conditional autoregressive generation, which can be expressed as the product of the probability distribution of iteratively generating new tokens using the real token as a condition: p(w′ k |w1,w2,…,w k-1 ) means to convert t token Input to the pre-trained generative large language model and iteratively train it to generate w′ using conditional autoregressive generation k The probability distribution of t token The product of the probability distribution p(t token ); S2.4.2 Set w1,w2,...,w l Use the one-hot encoding method to represent them as high-dimensional vectors p(w1), p(w2),…, p(w l ), calculate p(w1),p(w2),…,p(w l ) to obtain p(t token )’s real token high-dimensional vector product p(real), calculate p(t token ) and p(real), and update the weight matrix of the model using the gradient descent algorithm based on the cross entropy loss; S2.5 continuously adjusts the learning rate β, training round epoch, and training batch size batch_size hyperparameters, and through iterative training in step S2.4, finally obtains the pre-trained generative large language model M for App data synthesis after fine-tuning using the text dataset T; S3 Cellular User App Usage Data Synthesis S3.1 Use the fine-tuned model M to train the tabular dataset D by setting input prompts real The feature joint distribution sampling generates text sequence data, and the text sequence data is converted into table data to obtain a synthetic cellular user App usage data set; the feature joint distribution refers to D real The number distribution of user attribute feature rows and App usage traffic feature rows in is represented by p(v1,…,v n ) indicates that v i Represents the name of the i-th feature; S3.2 preprocesses the input prompt and encodes the input prompt into a conditional token sequence [w1,w2,…,w k ] is input to the model M, k represents the number of tokens after the input prompt is encoded, and its output is the logits distribution z of ω, ω represents the next token to be generated; specifically, a temperature parameter T>0 is used to adjust the randomness of ω, and by normalizing the logits distribution z, the probability distribution of ω is expressed as: Among them, z i represents the logits distribution z that predicts ω as the logits of each token in the vocabulary W, is the sum of the exponential logits from the logits distribution z that predicts ω as each token in the vocabulary W. It acts as a normalization factor to ensure that the sum of the probabilities of all tokens is 1. S3.3 Use the sample() function to sample the probability distribution p(ω|w1,w2,…,w k ) performs random sampling to select a token from the vocabulary W as the final generation result, and obtains the next token ω to be generated; The sample() function is implemented as follows: S3.3.1 For a given probability distribution p(ω|w1,w2,…,w k ), calculate the cumulative probability of each token by taking the probability distribution p(ω|w1,w2,…,w k ) are arranged in descending order and then added in sequence to obtain an increasing cumulative probability distribution; S3.3.2 generate a random number r between 0 and 1; S3.3.3 Find the first token corresponding to the probability value greater than the random number r from the cumulative probability distribution, and select the token as the final generated result; S3.4 takes ω as the k+1th token w k+1 Add to the conditional token sequence [w1,w2,…,w k ], repeat steps S3.2 to S3.3 to continuously generate new tokens until the conditional token sequence [w1,w2,…,w k ] Sample the k+1th to lth tokens, where l represents the number of tokens required to encode the input text sequence t in step S2.2; finally, concatenate all tokens together to form an input prompt and a token sequence generated according to the input prompt, and obtain the synthesized cellular user App usage text sequence data; S3.5 performs regular expression decoding on the text sequence data to extract the user attribute feature column names and App traffic usage feature column names and their corresponding values. Based on the decoded features and values, the corresponding tabular data is constructed to obtain a synthetic cellular user App usage dataset.
2. A method for synthesizing cellular user App usage data based on a large language model according to claim 1, characterized in that: In S2, the GPT2 model is used as a pre-trained generative large language model.
3. A method for synthesizing cellular user App usage data based on a large language model according to claim 1, characterized in that: In S2.3, the hyperparameters required for model fine-tuning are learning rate β=5e-4, training round epoch=25, and training batch size batch_size=16.
4. A method for synthesizing cellular user App usage data based on a large language model according to claim 1, characterized in that: In S2.4.1, for the dataset T token Each batch_size t token =[w1,w2,...,w l ], iterative training using conditional autoregressive generation, including the following steps: 1) If there is no input, the model randomly generates w′1; 2) If the input is [w1], the model predicts and generates w′2; 3) If the input is [w1,w2], the model predicts and generates w′3, and so on. If the input is [w1,w2,...,w l-1 ], the model predicts and generates w′ l .
5. A method for synthesizing cellular user App usage data based on a large language model according to claim 1, characterized in that: In S2.4.2, p(t token ) and p(real) cross entropy loss L: Among them, p(real) i and p(t token ) i Respectively represent p(t token ) and the probability value of p(real) at the i-th position.
6. A method for synthesizing cellular user App usage data based on a large language model according to claim 1, characterized in that: In S2.4.2, the update rule of the gradient descent algorithm is expressed as: Among them, θ e is the model weight matrix θ after the e-th training round iteration, β is the learning rate, represents the gradient of the cross entropy loss L with respect to the model weight matrix θ.
7. A method for synthesizing cellular user App usage data based on a large language model according to claim 1, characterized in that: In S3.1, the input prompts include: (1) feature name prompts, the input format is "[Feature]is", using model M to select from p(v1,…,v n ) to sample and generate text sequence data; (2) Feature key value prompt, the input format is "[Feature i ]is[Values i ]", the subscript i represents the number of n ) Randomly select v i , the model M samples from the joint distribution of the remaining features according to the above prompts and generates text sequence data.
Citation Information
Patent Citations
Large language model hot search generation method and device based on fine tuning
CN117494786A
Training data synthesis method and device for event extraction task
CN118114198A
Language-model-based data augmentation method for textual classification tasks with little data
US20210350076A1
Cited By
Multi-dimensional traffic data synthesis method and system based on large language model
CN118886397A
Multi-dimensional Traffic Data Synthesis Method and System Based on Large Language Model
CN118886397B
Synthetic data generation method for recommendation system and medium
CN122413348A
