Multi-dimensional traffic data synthesis method and system based on large language model
Through a multi-dimensional traffic data synthesis method based on a large language model, text feature coding and autoregressive model fine-tuning is used to solve the problem that effective time series data cannot be generated in the existing technology, and efficient and flexible cellular traffic data synthesis is achieved. High fidelity and privacy protection.
Patent Information
- Application Number
- CN202410930622.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-07-11
AI Technical Summary
Existing cellular traffic data synthesis methods cannot effectively generate heterogeneous data containing time series, and methods based on GAN and diffusion models lack flexibility and information loss problems in data preprocessing.
Using a multi-dimensional traffic data synthesis method based on large language models, encode historical data through text features, build a large autoregressive language model, fine-tune it, generate synthetic table data, and use text encoding to preserve context information and effectively model the time series. .
It realizes cellular network data synthesis with high fidelity, high practicality and privacy protection under any conditions, improving the efficiency and accuracy of data synthesis.
Smart Images

Figure CN118886397A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cellular data synthesis, and in particular to a method and system for synthesizing multi-dimensional traffic data based on a large language model. Background Art
[0002] Cellular traffic data synthesis is an important research direction in the field of mobile communications. Existing research methods mainly include methods based on generative adversarial networks, methods based on diffusion models, and methods based on deep language models. Research based on generative adversarial networks and diffusion models lacks flexibility and cannot support conditional generation with specific feature combinations. In addition, data preprocessing operations (such as one-hot encoding of categorical data) are susceptible to multidimensional data modeling, and they are also likely to lose some important information due to data transformation (i.e., data scaling and smoothing). With the rapid development of large language models, thanks to the rich and efficient understanding and expression capabilities of large language models, it provides us with an opportunity to flexibly generate synthetic cellular traffic data with arbitrary feature conditions.
[0003] Cellular traffic data usually contains information of different dimensions, including user attributes and application (app) preferences, traffic sequences, etc., which are collected from mobile phone users using mobile applications on their smartphones. Compared with existing GAN and diffusion model-based methods, large language models have two main advantages in traffic synthesis in a natural language processing manner. First, the tokenization process of large language models only processes text and does not require the data type of features to be specified in advance. The tokenization method based entirely on text can alleviate the problem of lossy preprocessing and solve the dimensionality explosion problem caused by using one-hot encoding for high-dimensional data. Second, both feature names and feature values are included in the text encoding, which retains more contextual information than usual data conversion and is conducive to modeling heterogeneous data with various features for learning models. However, existing language model-based table synthesis methods ignore the modeling of time series and cannot effectively generate cellular traffic sequences. Summary of the invention
[0004] The present invention provides a multi-dimensional traffic data synthesis method and system based on a large language model, which is used to solve the technical problem that the existing cellular data synthesis method cannot synthesize effective heterogeneous data containing time series under any conditions.
[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:
[0006] A multi-dimensional traffic data synthesis method based on a large language model, comprising:
[0007] Acquire historical table data, wherein the historical table data includes flow-related data of multiple different dimensions;
[0008] Performing text feature encoding on the flow-related data in the historical table data to obtain historical text encoding features;
[0009] A large autoregressive language model is constructed, and the large autoregressive language model is fine-tuned using the historical text encoding features;
[0010] Target table data is acquired, and the target table data is sampled using the autoregressive large language model, and synthetic table data is generated according to the sampling result.
[0011] Preferably, the traffic-related data includes user attribute data, program usage time series data, and program usage traffic series data;
[0012] Performing text feature encoding on the flow-related data in the historical table data includes:
[0013] Acquire user attribute data in the historical table data, and perform text encoding on the user attributes to obtain user attribute features;
[0014] Obtaining program usage time series data in the historical table data, and performing text encoding on the time series data to obtain time series features;
[0015] Obtaining program usage flow sequence data in the historical table data, and performing text encoding on the flow sequence data to obtain flow sequence features;
[0016] The user attribute features, time series features and flow series features are concatenated to obtain historical text coding features.
[0017] Preferably,
[0018] The user attribute feature encoding is expressed in the form of:
[0019] Where i represents the subscript of the user attribute, and j represents the subscript of the user attribute value; Represents the characteristic name of the i-th user attribute, represents the feature value of the i-th user attribute; the superscript u indicates that both the feature name and the feature value are derived from the user attribute data;
[0020] and / or
[0021] The template for the time series feature encoding is: “start tag-time-end tag:
[0022] The start time sequence encoding is expressed as The end time series encoding is expressed as Where i represents the time feature subscript, j represents the subscript of the time feature value; TIME_START and / TIME_START are respectively used to mark the prefix and suffix of the sequence start time information, TIME_END and / TIME_END are respectively used to mark the prefix and suffix of the sequence end time information. Represents the feature name of the i-th time feature, It represents the characteristic value of the i-th time feature, and the superscript d indicates that both the characteristic name and the characteristic value are from time series data;
[0023] and / or
[0024] The expression form of the traffic sequence feature coding is:
[0025] Where i represents the subscript of the traffic sequence feature, j represents the subscript of the traffic sequence feature value, and f i r is the feature name of the i-th traffic sequence feature, is the eigenvalue of the ith traffic sequence feature, and the superscript r indicates that both the feature name and the eigenvalue are derived from the traffic sequence data;
[0026] and / or
[0027] The historical text encoding feature is t i,j = {u i,j ,d i,j,start ,d i,j,end ,r i,j}, t i,j Represents the jth feature text encoding in the i-th continuous time period.
[0028] Preferably, fine-tuning the autoregressive large language model using the historical text encoding features comprises:
[0029] Based on the historical table data, multiple historical text encoding features of continuous time periods of the same user ID are constructed to construct time series text data t i =(t i,1 ,“,”,t i,2 ,…,“,”,t i,m ), where t i,1 Encode the first feature text in the i-th continuous time period; t i,2 Encode the second feature text in the i-th continuous time period; t i,m is the mth feature text code in the i-th continuous time period; m is the number of historical text coding features;
[0030] Use the random permutation function P to t i Rearrange to get sorted time series text data where [k1,k2,…,k m ]=P([1,2,…,m]), where t i,1 Encode the first feature text in the i-th continuous time period; t i,2 Encode the second feature text in the i-th continuous time period; t i,m Encode the mth feature text in the i-th continuous time period;
[0031] For the sorted time series text data t i ′ is decomposed into single words, subwords and characters, and the token sequence [w1,w2,…,w j ], w1 is the first mark number, w2 is the second mark number, w j is the jth marker number;
[0032] Decompose the probability of learning the sequence of tokens through autoregression and adjust the parameters of the autoregressive large language model;
[0033] Preferably, the probability of learning the tag sequence is decomposed by autoregression, and the parameters of the autoregressive large language model are adjusted, including:
[0034] Constructing a loss function and an optimizer to train the autoregressive large language model;
[0035] The loss function is calculated as follows:
[0036] Among them, w k The current tag to be generated in the original tag sequence, p(w k ) is the autoregressive training, the current generated tag is the current tag to be generated w k The probability of
[0037] The process of updating the weight parameters of the optimizer is as follows:
[0038] Calculate the first-order moment estimate m of the k-th gradient k :
[0039] m k =β1m k-1 +(1-β1)g k ; where k is the number of iterations, g k represents the kth parameter gradient, β1 is the first-order weight attenuation coefficient, m k-1 is the first-order moment estimate of the k-1th gradient;
[0040] The first-order moment estimate m based on the k-th gradient k Compute the second moment estimate v of the gradient k :
[0041] β2 is the second-order weight attenuation coefficient, v k-1 is the second-order moment estimate of the k-1th gradient;
[0042] Correct the bias in the first moment estimate:
[0043]
[0044] in, is the corrected first-order moment estimate;
[0045] Correcting bias in second moment estimates in, is the corrected second-order moment estimate;
[0046] Use the following formula to update the parameters:
[0047]
[0048] Among them, θ k represents the k-th model parameter, θ k-1 represents the k-1th model parameter, α is the learning rate, λ is the L2 regularization coefficient,
[0049] ∈ is a small constant added for numerical stability.
[0050] Preferably, the probability of learning the sequence of labeled numbers is decomposed by autoregression, which is achieved by the following formula:
[0051]
[0052] Among them, t represents the time series text sentence of this training; p(t) represents the probability that the text generated by autoregression is exactly the original text; p(t) is the goal of training, that is, maximizing the probability.
[0053] Preferably, sampling the target table data using the autoregressive large language model and generating synthetic table data according to the sampling result includes:
[0054] The input prompt is preprocessed and decomposed into token numbers [w1,w2,…,w k-1 ], the autoregressive large language model predicts the next token number w based on the observed previous k-1 token numbers k , until the end tag is predicted;
[0055] Convert the tokens predicted by the large language model into text, and use predefined rules to convert the generated text data into tabular data, and then assign the corresponding data type to the synthetic data based on the information of the feature column of the original data;
[0056] Perform type and range checks on the generated data. If the generated feature data cannot be converted into the data type corresponding to the original data or the generated feature data exceeds the range specified by the original data, this part of the feature data will be discarded, and the successfully synthesized data will be used to reconstruct the feature key-value pair prompts and regenerate the data until the type and range of all synthesized data are correct.
[0057] Preferably, the input prompt includes: a feature name prompt, the input format of the feature name prompt is "[Feature]is", and the autoregressive large language model extracts the feature name from the joint distribution p(v1,…,v n ) and generate samples, where v i Represents the random variable corresponding to feature i, where i∈{1,…,n}.
[0058] Preferably, the input prompt includes: a feature key value prompt, and the input format of the feature key value prompt is "[Feature i ]is[Values i ]", where i∈{1,…,n}, and the autoregressive large language model generates data by sampling from the distribution of the remaining features according to this hint.
[0059] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.
[0060] The present invention has the following beneficial effects:
[0061] 1. In the present invention, a text encoder is constructed to convert the original table data into text data according to the type of feature, and the text data describes the features and feature values of the original table data in the form of natural language; an autoregressive large language model is constructed, and the autoregressive large language model is fine-tuned using the table data after text encoding, so that it can fully learn and understand the data distribution of each feature, the dependency between data of different features, and the time dependency between historical traffic data; a synthetic data sampler is constructed, and the fine-tuned large language model is used to sample in the original table data, and text with a distribution similar to the original table data and good data privacy is generated according to different prompt inputs, and the generated text data is converted into synthetic table data. The present invention can use the large language model to flexibly and effectively synthesize mobile cellular network data with high fidelity, high practicality and effective privacy protection under any conditions, and adopts a regeneration strategy when synthesizing data, which greatly improves the efficiency of data synthesis.
[0062] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0064] Figure 1 It is a framework diagram of a cellular traffic data synthesis model (LLMCell) in a preferred embodiment of the present invention;
[0065] Figure 2 It is a CDF comparison diagram of the EMD score of the uplink traffic synthetic data in the preferred embodiment of the present invention;
[0066] Figure 3 It is a CDF comparison diagram of the EMD score of the synthetic data of the downlink traffic in the preferred embodiment of the present invention;
[0067] Figure 4 This is a CDF comparison chart of the JSD scores of the synthetic data of the user application preference labels in the preferred embodiment of the present invention;
[0068] Figure 5 This is a MAE performance comparison chart of using LSTM to predict traffic flow on overall data in a preferred embodiment of the present invention;
[0069] Figure 6 This is a RMSE performance comparison chart of using LSTM to predict traffic flow on overall data in a preferred embodiment of the present invention;
[0070] Figure 7 This is a MAE performance comparison chart of using ConvLSTM to predict traffic flow on overall data in a preferred embodiment of the present invention;
[0071] Figure 8 This is a RMSE performance comparison chart of using ConvLSTM to predict traffic flow on overall data in a preferred embodiment of the present invention;
[0072] Fig. 9 This is a MAE performance comparison chart of using LSTM to predict traffic for data with entertainment application preference in a preferred embodiment of the present invention;
[0073] Fig.10 This is a MAE performance comparison chart of using LSTM to predict traffic for data with application preference as social media in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0074] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.
[0075] Embodiment 1:
[0076] This embodiment discloses a method for synthesizing multi-dimensional traffic data based on a large language model, including the following steps:
[0077] Construct a text encoder to convert the original table data into text data according to the type of features. The text data describes the features and feature values of the original table data in the form of natural language.
[0078] Build an autoregressive large language model and use the text-encoded tabular data to fine-tune the autoregressive large language model so that it can fully learn and understand the data distribution of each feature, the dependencies between data of different features, and the time dependencies between historical traffic data.
[0079] Build a synthetic data sampler, use the fine-tuned large language model to sample the original tabular data, generate text with a distribution close to the original tabular data and good data privacy based on different prompt inputs, and convert the generated text data into synthetic tabular data.
[0080] In addition, in this embodiment, a computer system is also disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the steps of the above method are implemented when the processor executes the computer program.
[0081] The present invention can utilize a large language model to flexibly and effectively synthesize mobile cellular network data with high fidelity, high practicality and effective privacy protection under any conditions, and adopts a regeneration strategy when synthesizing data, thereby greatly improving the efficiency of data synthesis.
[0082] Embodiment 2:
[0083] Embodiment 2 is a preferred embodiment of Embodiment 1, and its difference from Embodiment 1 is that the specific steps of the flexible and effective cellular traffic data synthesis method based on a large language model are expanded:
[0084] In this embodiment, a flexible and effective cellular traffic data synthesis method based on a large language model is disclosed. Through the text encoding mode and the autoregressive fine-tuning mechanism of the large language model, the time dependency of mobile cellular network data and the dynamic relationship between features are effectively captured to improve the accuracy of cellular traffic data synthesis under any given conditions.
[0085] In this embodiment, the method is applied to the synthesis of mobile cellular traffic data of a large operator, taking the synthesis of mobile cellular traffic data as an example. Specifically, the following steps are included:
[0086] Step 1: Mobile cellular traffic data preprocessing
[0087] This example data set contains the mobile cellular network traffic data records of a large operator. By cleaning and preprocessing the data set, the mobile cellular traffic usage data of 9,900 users are obtained. The statistical time interval of the traffic data is hours, and the traffic data of each user includes the cellular traffic data used within one month from December 1, 2020 to December 30, 2020. Each user data includes: user id, user age, user gender, user preferred application type, cellular traffic data generation time, and the size of the uplink and downlink traffic generated by the user in the past hour. The present invention uses the entire data set for autoregressive training of a large language model, and divides each user's 24 consecutive cellular traffic data into one sample. The entire training set contains a total of 306,900 samples.
[0088] Step 2: Build a text encoder and convert raw tabular data into text representation
[0089] Since the input form of large language models is text, it is necessary to encode the tabular data into a text representation with semantic information. Figure 1 As shown in the “Textual Encoder” in the figure, the text encoder constructed by the present invention has three modules in total: a user attribute label encoding module, a date and time encoding module, and a traffic sequence encoding module, which encode different parts of the cellular traffic data respectively.
[0090] The user attribute label encoding module encodes the user attribute information in each sample according to the formula In a sample, although there are 24 data, they all belong to the same user, so each user attribute has only one value. For example, if the user age corresponding to a sample is 22, it will be encoded as the text "age is 22,".
[0091] The date and time encoding module converts the start sampling time of each sample, that is, the time of the first data, into the formula Encode; the sampling end time of each sample, that is, the time of the last data, is calculated according to the formula For example, if the start sampling time of a cellular traffic data sample is 0:00 on December 2, 2020, and the end sampling time is 23:00 on December 2, 2020, the start time will be encoded into the text "TIME_START date is 20201202, houris 0 / TIME_START", and the end time will be encoded into the text "TIME_END date is 20201202, gour is 23 / TIME_END".
[0092] The traffic sequence encoding module converts the traffic sequence of each sample into In a sample, there are 24 data, each of which has a traffic value, and the traffic values of the 24 data constitute a traffic sequence with a length of 24. For example, the uplink traffic values of the 24 data in a sample are 1.52, 2.66, ..., 55.219, respectively. The traffic sequence feature of the sample is finally encoded into the text "Uplink traffic series is (1.52, 2.66, ..., 55.219)".
[0093] After the user attributes, date and time, and traffic sequences are encoded separately, they are separated by commas to form the final time series text data.
[0094] Step 3: Fine-tune the large language model based on the text-encoded tabular data
[0095] The model base used in this implementation example is GPT2 after knowledge distillation, namely DistilGPT-2, which has a certain semantic understanding ability and is suitable for fine-tuning using specific data to adapt to downstream tasks.
[0096] After the data text encoding is completed, each sample will constitute a time series text data t i =(t i,1 ,“,”,t i,2 ,…,“,”,t i,m ), where t i,j = {u i,j ,d i,j,start ,d i,j,end ,r i,j}. However, in the text encoding transfer, the features of the table data are arranged in a fixed order, and the order of the features before and after is highly coupled, which will affect the sampling diversity of the fine-tuned model when synthesizing data and the arbitrariness of the data synthesis conditions. In order to ensure the relative independence of the features, the present invention uses a random permutation function P to t i Rearrange to get where [k1,k2,…,k m]=P([1,2,…,m]). The rearranged text data provides the benefit of synthesizing data under arbitrary conditions during subsequent sampling.
[0097] The present invention uses an autoregressive approach to fine-tune a large language model. In the fine-tuning process, firstly, a specific tagging scheme is used to pre-process the sorted text sentence t into single words, subwords and characters using a text corpus, so that the sentence t is decomposed into a series of tag numbers [w1, w2, ..., w j ], then the language model decomposes and calculates the probability of the tag sequence, and adjusts the model parameters by the error between the predicted sequence and the true sequence. The estimated value of the entire sequence is obtained by predicting the conditional probability of the next tag under the premise of the calculated tags, as shown in formula (1):
[0098]
[0099] Step 4: Generate tabular data based on the fine-tuned large language model
[0100] After the fine-tuning step, a fine-tuned large language model is obtained. The model learns the distribution of the same feature of the original table data at different times and the correlation between different features. It can sample from the original data to generate synthetic data with a similar distribution to the original data.
[0101] During the data generation process, for each sample to be generated, a text prompt of the start time and end time of the sample is given to the large language model, such as "TIME_START date is 20201202, hour is 0 / TIMF_START, TIME_END date is 20201202, hour is 23 / TIME_END". The large language model converts the text prompt into a token and uses the observed tokens [w1, w2, …, w k-1 ]Predict the next token w k The probability of , predict the remaining feature information, and finally synthesize a sample data.
[0102] After obtaining the sample data, the corresponding features and feature values are extracted from the text data using a specific regular expression and converted into the same type as the original table data. Due to the presence of random noise in the data generation process, the generated data may be missing or exceed the range specified by the original data. Therefore, after the data conversion, the abnormal data is discarded, and the correctly generated synthetic data is used to reconstruct the feature key-value pair prompts. The large model uses the regeneration strategy to generate missing data instead of all data, thereby improving the efficiency of data synthesis. After all feature values are correctly synthesized, the data is saved as a CSV file to obtain the final table data.
[0103] Step 5: Model performance evaluation
[0104] In order to verify the effectiveness of the model of the present invention, synthetic data is used to conduct fidelity, practicality, and privacy evaluation experiments. The data synthesis method is shown in steps 1 to 4.
[0105] This step compares the data synthesis model LLMCell in this implementation example with the following state-of-the-art data synthesis methods, including STAN, DoppelGANger, TabDDPM, and GReaT. Among them, STAN is a network traffic data synthesis tool using an autoregressive neural model. DoppelGANger is a multidimensional time series generation model based on GAN. TabDDPM is a model that generates a synthetic tabular data set using a denoising diffusion probability model. GReaT is a method for sampling synthetic tabular data using an autoregressive generative model.
[0106] Step 5.1 Fidelity evaluation experiment
[0107] In the data synthesis task, the distribution of the synthesized data should be similar to that of the original data, and the consistency between the distribution of the synthesized data and the original data is called fidelity. In this implementation example, the fidelity of the synthesized uplink and downlink traffic sequences is evaluated using EarthMover Distance (EMD), and classification features such as user application preference labels are evaluated using Jensen-Shannon divergence (JSD).
[0108] EMD is used to calculate the similarity between the continuous type data in the real data and the synthetic data. P and Q represent the two probability distributions of a given set X respectively. The calculation method of EMD is shown in formula (2), where d(x, y) represents the distance between the elements x and y in the two probability distributions, and γ(x, y) represents the mass transferred from each element x in P to each element y in Q under the premise of using the optimal transportation plan γ. The smaller the EMD value, the higher the similarity between the continuous type data in the real data and the synthetic data, and the higher the data fidelity.
[0109]
[0110] JSD is used to calculate the similarity between real data and the classified data in synthetic data. P and Q represent the two probability distributions of a given set X, respectively. The calculation method of JSD is shown in formula (3), where P(x) and Q(x) are the probabilities of x in distribution P and Q, respectively. The smaller the JSD value, the higher the similarity between the classified data in real data and synthetic data, and the higher the data fidelity.
[0111]
[0112] The present invention uses baseline data synthesis methods such as STAN, DoppelGANger, TabDDPM and GReaT to compare and evaluate the fidelity with the LLMCell model. The synthetic data of 9900 users are divided into 44 groups, each containing 225 users. The EMD average of the uplink and downlink traffic of each group of data of different synthesis methods is calculated, and the JSD average of user application preferences is calculated. The results are normalized to the range of [0.1, 0.9], and the CDF graph is drawn. The CDF graph of the uplink traffic EMD score is shown in the figure below. Figure 2 As shown in the figure, the CDF graph of the downstream traffic EMD score is as follows Figure 3 As shown in the figure, the CDF diagram of the JSD score of user application preference is as follows Figure 4 As shown, it can be concluded that: 1) The LLMCell model proposed in the present invention significantly outperforms other methods in terms of fidelity in both upstream and downstream traffic. Considering the 90% percentile, for upstream traffic, the EMD scores of GreaT, STAN, DGAN, and TabDDPM are approximately equal to 0.15, 0.13, 0.62, and 0.63, respectively, while the EMD score of LLMCell is 0.10, which is an increase of 30.12%, 20.75%, 83.54%, and 83.85%, respectively; similarly, for downstream traffic, the EMD scores of GreaT, STAN, DGAN, and TabDDPM are 0.18, 0.13, 0.55, and 0.58, respectively. The score of the LLMCell scheme is only 0.10. It is proved that the customized design of LLMCell, especially the text encoding of traffic sequences, fully considers the time series pattern while other baselines do not carefully consider this factor. 2) LLMCell outperforms STAN, DGAN, and TabDDPM in terms of fidelity of user application preferences. In particular, at the 80% percentile, the JSD values of STAN, DGAN, and TabDDMP reach about 0.33, 0.77, and 0.77, respectively, while the JSD values of GreaT and the LLMCell model proposed in this invention are only 0.03 and 0.07. This proves that the LLM-based method has a strong reasoning ability based on text representation and can effectively model categorical data.
[0113] Step 5.2 Practicality evaluation experiment
[0114] The present invention evaluates the practicality of the proposed LLMCell model through two common downstream applications, namely, upstream traffic prediction and downstream traffic prediction. The present invention uses mean absolute error (MAE) and root mean square error (RMSE) as evaluation indicators for upstream and downstream traffic prediction. These two indicators are widely used in the evaluation of regression prediction tasks, and the smaller the indicator value, the better. i and They represent the true value and predicted value of the flow at the i-th time step, and n is the total number of predictions. The calculation method is as shown in formulas (4) and (5):
[0115]
[0116]
[0117] The present invention adopts two commonly used time series prediction models, including long short-term memory network and convolutional long short-term memory network.
[0118] Long Short-Term Memory (LSTM): A variant of Recurrent Neural Network (RNN) that is widely used as a deep learning model for time series forecasting.
[0119] Convolutional Long Short-Term Memory (ConvLSTM): A time series forecasting model that combines a convolutional neural network (CNN) and a long short-term memory network (LSTM).
[0120] In order to test the generalization of the model trained on synthetic data, the present invention sorts the real data and synthetic data based on timestamp, and then divides them into 70% training set and 30% test set, respectively, and compares the indicators in two cases: 1) training in real data and testing in real data; 2) training in synthetic data and testing in real data.
[0121] (1) Overall performance comparison
[0122] To evaluate the overall performance of the LLMCell model, this implementation example compares downstream applications on all synthetic data. Figure 5 and Figure 6 Represents the average performance of MAE indicator and LSTM indicator with error bars evaluated using LSTM predictor, Figure 7 and Figure 8Denote the MAE metric with error bars and the average performance of the LSTM metric evaluated using the ConvLSTM predictor, respectively. It can be observed that among the LSTM and ConvLSTM predictions, the evaluation results of the LLMCell model have the best performance and are closest to the real data compared with other baselines. Specifically, when using the LSTM model for uplink traffic prediction, the MAE values of the synthetic data of GreaT, TabDDPM, DGAN, and STAN are 68.25, 71.91, 671.63, and 4261.39, respectively, and the corresponding RMSE values are 135.76, 150.81, 683.49, and 4264.13, respectively. In contrast, the MAE and RMSE values of LLMCell are only 59.44 and 122.78, respectively. In addition, when observing the error bars, LLMCell shows lower deviation among all baselines, indicating the robustness of its synthetic data, further demonstrating that LLMCell can effectively capture the temporal pattern of cellular traffic data.
[0123] (2) Performance comparison of users with different application preferences
[0124] To evaluate the robustness of the data utility, we further examine the downstream task performance of user data with different application preference labels. Fig. 9 and 10 The MAE performance of two representative applications (entertainment and social media) represented by application preference is evaluated using the LSTM predictor. It can be observed that LLMCell obtains the MAE score closest to the real data and outperforms other baselines, indicating that LLMCell still performs best under different application preference labels, further proving the robustness of the LLMCell model for synthetic data.
[0125] Step 5.3 Privacy evaluation experiment
[0126] In order to ensure the synthesis of high-quality cellular traffic data while preventing the leakage of user privacy, the present invention uses the closest distance record (DCR) indicator to evaluate the distance-based privacy of each model. x and y represent real data and synthetic data respectively, and the calculation method of DCR is shown in formula (6). A larger DCR usually indicates lower data quality, while a smaller DCR may mean a potential risk of leaking sensitive information. Therefore, privacy assessment is relative, and the synthetic data model should maintain a relatively balanced position in DCR.
[0127]
[0128] Table 1 shows the DCR indicators calculated from the uplink and downlink traffic data synthesized by different models and the real data. The results show that the synthetic data generated by TabDDPM has the shortest distance from the real data, which means that its synthetic data essentially imitates some real data points and has the highest possibility of privacy violation; while the synthetic data generated by GreaT presents the largest distance, which indicates that it generates new data rather than a copy close to the real data; by considering the comprehensive performance, the DCR achieved by LLMCell can achieve a balance between data fidelity, practicality and privacy compared with the baseline.
[0129] Table 1. Privacy evaluation experimental results
[0130]
[0131]
[0132] In summary, the flexible and effective cellular traffic data synthesis method based on a large language model disclosed in the present invention effectively captures the time dependency of mobile cellular network data and the dynamic relationship between features through a text encoding mode and an autoregressive fine-tuning mechanism of a large language model, so as to improve the accuracy of cellular traffic data synthesis under any given conditions.
[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A multi-dimensional traffic data synthesis method based on a large language model, It is characterized in that include: Acquire historical table data, wherein the historical table data includes flow-related data of multiple different dimensions; Performing text feature encoding on the flow-related data in the historical table data to obtain historical text encoding features; A large autoregressive language model is constructed, and the large autoregressive language model is fine-tuned using the historical text encoding features; Target table data is acquired, and the target table data is sampled using the autoregressive large language model, and synthetic table data is generated according to the sampling result.
2. According to the multi-dimensional traffic data synthesis method based on the large language model according to claim 1, It is characterized in that The traffic-related data includes user attribute data, program usage time series data, and program usage traffic series data; Performing text feature encoding on the flow-related data in the historical table data includes: Acquire user attribute data in the historical table data, and perform text encoding on the user attributes to obtain user attribute features; Obtaining program usage time series data in the historical table data, and performing text encoding on the time series data to obtain time series features; Obtaining program usage flow sequence data in the historical table data, and performing text encoding on the flow sequence data to obtain flow sequence features; The user attribute features, time series features and flow series features are concatenated to obtain historical text coding features.
3. According to claim 2, the multi-dimensional traffic data synthesis method based on the large language model, It is characterized in that The user attribute feature encoding is expressed in the form of: Where i represents the subscript of the user attribute, j represents the subscript of the user attribute value; f i u Represents the characteristic name of the i-th user attribute, represents the feature value of the i-th user attribute; the superscript u indicates that both the feature name and the feature value are derived from the user attribute data; and / or The template for the time series feature encoding is: "start tag-time-end tag: The start time sequence encoding is expressed as The end time series encoding is expressed as Where i represents the time feature subscript, j represents the subscript of the time feature value; TIME_START and / TIME_START are respectively used to mark the prefix and suffix of the sequence start time information, TIME_END and / TIME_SED are respectively used to mark the prefix and suffix of the sequence end time information, i d Represents the feature name of the i-th time feature, It represents the characteristic value of the i-th time feature, and the superscript d indicates that both the characteristic name and the characteristic value are from time series data; and / or The expression form of the traffic sequence feature coding is: Where i represents the subscript of the traffic sequence feature, and j represents the subscript of the traffic sequence feature value. is the feature name of the i-th traffic sequence feature, is the eigenvalue of the ith traffic sequence feature, and the superscript r indicates that both the feature name and the eigenvalue are derived from the traffic sequence data; and / or The historical text encoding feature is t i,j = {u i,j ,d i,j,start ,d i,j,end ,r i,j }, t i,j Represents the jth feature text encoding in the i-th continuous time period.
4. The multi-dimensional traffic data synthesis method based on a large language model according to any one of claims 1 to 3, It is characterized in that Fine-tuning the autoregressive large language model using the historical text encoding features includes: Based on the historical table data, multiple historical text encoding features of continuous time periods of the same user ID are constructed to construct time series text data t i =(t i,1 ,",",t i,2 ,…,",”,t i,m ), where t i,1 Encode the first feature text in the i-th continuous time period; t i,2 Encode the second feature text in the i-th continuous time period; t i,m is the mth feature text code in the i-th continuous time period; m is the number of historical text coding features; Use the random permutation function t i Rearrange to get sorted time series text data in Among them, t i,1 Encode the first feature text in the i-th continuous time period; t i,2 Encode the second feature text in the i-th continuous time period; t i,m Encode the mth feature text in the i-th continuous time period; For the sorted time series text data t′ i Decompose into single words, subwords and characters, and get the token sequence [w 1 ,w 2 ,…,w j ],w 1 is the first marker number, w 2 is the second marker number, w j is the jth marker number; The probability of learning the sequence of tokens is decomposed through autoregression, and the parameters of the autoregressive large language model are adjusted.
5. According to claim 4, the multi-dimensional traffic data synthesis method based on the large language model, It is characterized in that Decompose the probability of learning the sequence of tokens through autoregression and adjust the parameters of the autoregressive large language model, including: Constructing a loss function and an optimizer to train the autoregressive large language model; The loss function is calculated as follows: Among them, w k The current tag to be generated in the original tag sequence, p(w k ) is the autoregressive training, the current generated tag is the current tag to be generated w k The probability of The process of the optimizer updating weight parameters is as follows: Calculate the first-order moment estimate m of the k-th gradient k : m k =β 1 m k-1 +(1-β 1 ) k ; where k is the number of iterations, g k represents the k-th parameter gradient, β 1 is the first-order weight attenuation coefficient, m k-1 is the first-order moment estimate of the k-1th gradient; The first-order moment estimate m based on the k-th gradient k Compute the second moment estimate v of the gradient k : β 2 is the second-order weight attenuation coefficient, v k-1 is the second-order moment estimate of the k-1th gradient; Correct the bias in the first moment estimate: in, is the corrected first-order moment estimate; Correcting bias in second moment estimates in, is the corrected second-order moment estimate; The parameters are updated using the following formula: Among them, θ k represents the k-th model parameter, θ k-1 represents the k-1th model parameter, α is the learning rate, λ is the L2 regularization coefficient, and ∈ is a small constant added for numerical stability.
6. The multi-dimensional traffic data synthesis method based on a large language model according to claim 4, It is characterized in that The probability of learning the sequence of labeled numbers is decomposed by autoregression, which is achieved by the following formula: Among them, t represents the time series text sentence of this training; p(t) represents the probability that the text generated by autoregression is exactly the original text; p(t) is the goal of training, that is, maximizing the probability.
7. The multi-dimensional traffic data synthesis method based on a large language model according to claim 6, It is characterized in that The target table data is sampled using the autoregressive large language model, and synthetic table data is generated according to the sampling result, including: The input prompt is preprocessed and decomposed into token numbers [w 1 ,w 2 ,…,w k-1 ], the autoregressive large language model predicts the next token number w based on the observed previous k-1 token numbers k , until the end tag is predicted; Convert the tokens predicted by the large language model into text, and use predefined rules to convert the generated text data into tabular data, and then assign the corresponding data type to the synthetic data based on the information of the feature column of the original data; Perform type and range checks on the generated data. If the generated feature data cannot be converted into the data type corresponding to the original data or the generated feature data exceeds the range specified by the original data, this part of the feature data will be discarded, and the successfully synthesized data will be used to reconstruct the feature key-value pair prompts and regenerate the data until the type and range of all synthesized data are correct.
8. The multi-dimensional traffic data synthesis method based on a large language model according to claim 7, It is characterized in that The input prompt includes: a feature name prompt, the input format of the feature name prompt is "[Feature]is", and the autoregressive large language model is derived from the joint distribution p(v 1 ,…,v n ) and generate samples, where v i Represents the random variable corresponding to feature i, where i∈{1,…,n}.
9. The multi-dimensional traffic data synthesis method based on a large language model according to claim 7, It is characterized in that The input prompt includes: feature key value prompt, the input format of the feature key value prompt is "[Feature i ]is[Values i ]", where i∈{1,…,n}, and the autoregressive large language model generates data by sampling from the distribution of the remaining features according to this hint.
10. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
Citation Information
Patent Citations
Language model pre-training method and system for table pattern analysis and sequence masks
CN112559556A
Table data question and answer method and device, equipment and storage medium
CN117390169A
Table question and answer method based on large language model and natural language processing
CN117743526A
Audit report generation method and device, computer equipment and storage medium
CN117993369A
Multi-modal information marking method and device, equipment, storage medium and product
CN118038125A
Cited By
Traffic flow prediction method and device, electronic equipment and storage medium
CN119992820A
