Flow data generation method using hybrid generative adversarial network

Abstract

The invention provides a method for generating flow data by using a hybrid generative adversarial network, which comprises the following steps of: aggregating and structuring repeated transactions in original transaction flow data, generating an original transaction network graph, and training a graph generation module by using the original transaction network graph, therefore, the graph generation module can learn topological structure features in a real transaction network structure and generate a transaction network structure close to reality. Furthermore, a transaction network graph and transaction flow data are generated through a graph generation module and a table data generation module respectively and then are integrated to obtain optimized data, and through organic combination of the two modules, high-quality synthetic data which not only retains data distribution characteristics but also has a real transaction structure can be generated. The defect that an existing deep learning model is difficult to capture the structural characteristics of the transaction flow data is overcome, and the synthesis quality of the transaction flow data is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of table data generation, and in particular to a method for generating flow data using a hybrid generative adversarial network. Background Art

[0002] Against the backdrop of the rapid development of big data and artificial intelligence, the financial industry is undergoing profound changes. As a core resource, financial data involves sensitive content such as user personal information, account records, and transaction flows, and is widely used in customer management, risk control, marketing, anti-fraud and other scenarios. Its high sensitivity and high value make it face the dual challenges of privacy protection and systemic risks. Once leaked or destroyed, it may have a serious impact on users, institutions and even the stability of the financial market. Therefore, protecting the security of financial data has become an industry consensus and research focus.

[0003] At the same time, financial institutions have extremely high requirements for the security, integrity, and confidentiality of massive customer and transaction data. These data support key businesses such as risk assessment, credit assessment, and market analysis. However, as privacy protection regulations become increasingly stringent, data acquisition and use become increasingly difficult, leading to the intensification of data silos and information asymmetry, which limits data analysis capabilities and competitiveness. How to efficiently use data while ensuring compliance and privacy protection is a core issue that the financial industry needs to solve urgently.

[0004] Synthetic data is a form of data that does not directly map to real personal information. It can be used, shared and stored freely, effectively avoiding the risk of privacy leakage. Compared with traditional methods, synthetic data technology based on deep learning can better retain the statistical and structural characteristics of data and better meet the needs of financial technology products and model training. Its applications cover customer analysis, credit decision-making, risk management, anti-fraud and other scenarios, significantly improving data utilization efficiency while reducing legal and ethical risks.

[0005] Currently, tabular data generation models are usually divided into two approaches:

[0006] One of them is the likelihood-based generative method. This method uses the log-likelihood or a suitable alternative as the training target, and must use a specific architecture to build a normalized probability model (autoregressive model, flow models) or use an alternative loss (VAE). For a given dataset {x1, x2, …, x n} obeys a certain distribution p(x), and the goal is to train a generator to approximate the distribution. Finally, by sampling from the approximate distribution, the generated data X' is obtained. Variational auto encoder (VAE), flow-based models and diffusion models are likelihood-based methods. VAE consists of two similar networks, an encoder and a decoder. The encoder receives the input and converts it into a smaller dimensional representation, which the decoder can use to convert it into the original input. The latent space they convert the input to and the space where their encoding vectors are located may not be continuous. To solve this problem, the variational autoencoder has the characteristics of a continuous latent space, which makes random sampling and interpolation operations more convenient. The flow-based model directly calculates the probability distribution, which is composed of a series of reversible functions f i , and each f i The diffusion model is a model inspired by nonequilibrium thermodynamics, which defines a Markov chain that gradually adds random noise to the data and then learns the inverse diffusion process to construct the required data samples from the noise. Variant models such as TVAE and DDPM are generated for tabular data.

[0007] The other is the adversarial generation method, among which the Generative Adversarial Networks (GAN) is the most representative model. GAN is an unsupervised model that takes random noise as input and generates output. The generator and discriminator are trained in parallel through mutual confrontation and mutual learning. Generative adversarial networks also have many variant models for tabular data, such as MedGAN, TableGAN, CTGAN, CTAB-GAN, CTAB-GAN+, etc.

[0008] However, when generating transaction flow data, existing tabular data generation technologies can usually only restore the distribution of features, while ignoring the structural information in the transaction flow data set. Specifically, users often generate a large number of repeated transactions with the same merchant, and existing tabular data generation models based on generative adversarial networks (GANs) usually cannot effectively capture such repeated features and can only restore the numerical distribution features, resulting in substantial differences between the generated transaction flow data and the actual data, which in turn affects the accuracy of related evaluation and analysis based on the generated data. Summary of the invention

[0009] The present invention is made to solve the above problems, and aims to provide a transaction flow data generation method that can generate high-quality synthetic data that retains data distribution characteristics and has a real transaction structure. The present invention adopts the following technical solutions:

[0010] The present invention provides a method for generating transaction flow data using a hybrid generative adversarial network. The method has the following technical features: a hybrid generative adversarial network model is adopted, and the model includes a preprocessing module, a graph generation module, a table data generation module and an integration module. The method includes the following steps: step S1, the preprocessing module aggregates and structures repeated transactions in original transaction flow data to generate an original transaction network graph; step S2, the graph generation module is trained using a data set composed of the original transaction network graph, and a transaction network graph is generated by the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data; step S3, the integration module integrates the transaction network graph and the transaction flow data to obtain optimized transaction flow data.

[0011] The method for generating transaction flow data using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein, in step S1, repeated transactions in the original transaction flow data are aggregated, and the number of transactions of the repeated transactions is counted to obtain aggregated data, and the structural information in the aggregated data is used as nodes, transaction relationships as edges, and the number of transactions as weight information of the edges to construct the original transaction network graph.

[0012] The pipeline data generation method using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein the repeated transactions are multiple transactions between the same group of user identification numbers and merchant identification numbers, the structural information includes the user identification number and the merchant identification number, and the transaction relationship includes at least the transaction amount, transaction time, and transaction type.

[0013] The pipeline data generation method using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein the graph generation module includes: a sampling unit, which samples the original transaction network graph to obtain a true random walk sequence; and a generator and a discriminator, which alternately learn based on the true random walk sequence and the generation result of the generator, so that the generator captures the topological structure characteristics in the original transaction network graph.

[0014] The pipeline data generation method using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein, in each training round, random noise is input into the generator, a walk sequence is generated using the generator, a loss function for generating the walk sequence is calculated, and the parameters of the generator are updated based on the corresponding loss; in each training round, the sampling unit extracts the real random walk sequence, the real random walk sequence and the walk sequence generated by the generator are input into the discriminator, the loss function of the discriminator is calculated, and the parameters of the discriminator are updated based on the corresponding loss.

[0015] The pipeline data generation method using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein the sampling unit uses a weighted random walk algorithm to sample the original transaction network graph and sets the walk length to obtain the real random walk sequence.

[0016] The method for generating flow data using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein, in step S3, the integration module matches the weight information of each edge in the generated transaction network graph with the transaction number information in the generated transaction flow data, thereby combining the generation results of the graph generation module with the table data generation module to obtain combined data.

[0017] The method for generating transaction flow data using a hybrid generative adversarial network provided by the present invention may also have such a technical feature, wherein, in step S2, the table data generation module generates the transaction flow data based on the input aggregated data, and in step S3, the integration module also splits the aggregated edges in the combined data to obtain the optimized transaction flow data.

[0018] Functions and Effects of the Invention

[0019] According to the transaction flow data generation method using a hybrid generative adversarial network provided by the present invention, repeated transactions in the original transaction flow data are aggregated and structured to generate an original transaction network graph, which is used for training the graph generation module. Therefore, the graph generation module can learn the topological structure characteristics in the real transaction network structure and generate a transaction network structure close to the real one; further, since the transaction network graph and transaction flow data are generated by the graph generation module and the table data generation module respectively, and then integrated to obtain optimized data, through the organic combination of the two modules, high-quality synthetic data that retains the data distribution characteristics and has a real transaction structure can be generated, which makes up for the shortcoming that the existing deep learning model is difficult to capture the structural characteristics of the transaction flow data, and further improves the synthesis quality of the transaction flow data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the key steps of the pipeline data generation method using a hybrid generative adversarial network in an embodiment of the present invention;

[0021] Figure 2 is a flow chart of a method for generating pipeline data using a hybrid generative adversarial network in an embodiment of the present invention;

[0022] Figure 3 is an example diagram of aggregating and counting repeated transactions in an embodiment of the present invention;

[0023] Figure 4 It is a comparison diagram of the original transaction network diagram in the embodiment of the present invention and the transaction network diagram of the transaction flow data generated by three models. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of the method for generating flow data using a hybrid generative adversarial network of the present invention in combination with embodiments and drawings.

[0025] <Example>

[0026] Figure 1 is a schematic diagram of the key steps in the pipeline data generation method using a hybrid generative adversarial network in this embodiment, Figure 2 It is a flowchart of the pipeline data generation method using a hybrid generative adversarial network in this embodiment.

[0027] like Figure 1 and Figure 2 As shown in Figure 1, the hybrid generative adversarial network model includes a preprocessing module, a graph generation module, a table data generation module, and an integration module. The pipeline data generation method using the hybrid generative adversarial network includes the following three key steps:

[0028] Step S1, the preprocessing module aggregates and structures the repeated transactions in the original transaction flow data set to generate an original transaction network graph.

[0029] Step S2, using the data set consisting of the original transaction network graph to train the graph generation module, and generate the transaction network graph through the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data set.

[0030] Step S3, the integration module integrates the transaction network diagram and the transaction flow data to obtain optimized transaction flow data.

[0031] The above steps will be described in detail below.

[0032] Step S1, the preprocessing module aggregates and structures the repeated transactions in the original transaction flow data set to generate an original transaction network graph.

[0033] In order to better process and analyze data, the original transaction flow data set is divided into two parts: structural information part and transaction information part. Structural information mainly includes user ID (unique identification number) and merchant ID. Transaction information covers transaction amount, transaction time, transaction type and other contents.

[0034] Since there are a large number of repeated transactions between the same node pairs (node ​​pairs of user ID and merchant ID) in the transaction flow data, in order to enable the hybrid generative adversarial network model in the subsequent steps to better generate data, in step S1, the preprocessing module aggregates the repeated transactions in the original transaction flow data and counts the number of repeated transactions between the user ID and the merchant ID to obtain aggregated data. Then, the user ID and merchant ID node pairs in the aggregated data are used as node sets, and other transaction information is used as edge sets to obtain the original transaction network graph composed of the structural information in the original transaction flow data, and the transaction network data set can be formed from the original transaction network graph.

[0035] Figure 3 This is an example diagram of aggregating and counting repeated transactions in this embodiment.

[0036] like Figure 3 As shown in the table in the upper middle, taking the original transaction data as an example, there are repeated transactions between the user with user ID (uid) 31xxxx46 and the merchant with merchant ID (mer_no) 808xxxxxxxxx000. The repeated transactions are aggregated according to the uid-cid pair, and the fields and data such as uid, mer_no, card ID (cid), card type (card-type), merchant type (mer_type), transaction type (trans_type), transaction date (pt_dt) are retained after aggregation, and the transaction weight (weight) and transaction times (times) fields are added, where the transaction weight can be calculated based on the amount of each transaction and the number of repeated transactions, thereby generating aggregated data.

[0037] Then, uid and mer_no in the aggregated data are used as nodes to form a node set, other transaction information (transaction records) in the aggregated data are used as aggregated edges to form an edge set, and the corresponding number of transactions is used as the weight information of each aggregated edge to construct the original transaction network graph, thereby obtaining a transaction network data set.

[0038] Step S2, using the data set consisting of the original transaction network graph to train the graph generation module, and generate the transaction network graph through the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data set.

[0039] Among them, the table data generation module can adopt the existing table data generation model. Since the input is an aggregated data set, the transaction flow data generated by it is also aggregated data of a similar structure, which includes the above-mentioned transaction repetition times.

[0040] The graph generation module is specialized in processing complex transaction network structures. It includes a sampling unit, a generator and a discriminator. The generator and the discriminator constitute a generative adversarial network. The sampling unit uses a weighted random walk algorithm to sample the original transaction network graph composed of structural information to obtain a real random walk sequence. Subsequently, based on the real random walk sequence and the generation results of the generator, the generator and the discriminator are alternately learned, and the parameters of the generator and the discriminator are updated respectively, so that the generator can capture the topological structure characteristics in the real transaction network graph. Finally, the generator with updated parameters is used to finally generate the transaction network graph. In the generated transaction network graph, the nodes are the user IDs and merchant IDs for the transaction, the edges are their corresponding transaction records, and the edges have weight information, which is the number of transactions mentioned above.

[0041] Table 1 below shows the pseudo code of the graph generation module.

[0042] Table 1 Pseudo code of the graph generation module

[0043]

[0044]

[0045] Step S3, the integration module integrates the generated transaction network diagram and transaction flow data to obtain optimized transaction flow data.

[0046] Among them, the core idea of ​​the integration is to combine the transaction network structure diagram generated by the graph generation module with the table feature data output by the table data generation module.

[0047] Specifically, the weight information of each edge in the transaction network graph is matched with the transaction number information previously added in the generated transaction flow data, so as to combine the generation results of the graph generation module and the table data generation module to obtain the combined data. Since the preprocessing module aggregates the repeated multiple transactions in step S1, it is also necessary to split the aggregated edges in the combined data to obtain the final structure-optimized transaction flow data.

[0048] This integration method ensures that the generated transaction flow data is not only very similar to the real data in structure, but also as close as possible to the actual transaction situation in terms of structural feature distribution. Compared with the existing table data generation model, the model and method of this embodiment can better maintain the diversity and high repeatability of the generated transaction flow data in topological structure.

[0049] The inventors compared the model and method of this embodiment with two models and methods in the prior art.

[0050] Figure 4 It is a comparison diagram of the original transaction network diagram in this embodiment and the transaction network diagrams of the transaction flow data generated by the three models.

[0051] like Figure 4 As shown in the figure, the two models of the prior art are CTGAN and TVAE. The transaction flow data are generated based on the same original transaction flow data by using the two existing models and methods and the TransGAN model and method of this embodiment. Then, the original transaction flow data, the data generated by the two existing models, and the data generated by the TransGAN model of this embodiment are structurally analyzed and visualized, thereby forming Figure 4 .

[0052] from Figure 4 It can be seen intuitively that the data generated by CTGAN and TVAE show more one-to-one transaction relationships, and their distribution characteristics (structural characteristics) are quite different from those of the original data. In addition, there are some overly simplified transaction patterns in the data they generate, such as random transactions between a user and a large number of merchants. These characteristics rarely appear in real transaction flow data.

[0053] In comparison, the distribution characteristics of the data generated by TransGAN in this embodiment are obviously more consistent with the distribution characteristics of the original data. There are many cases in the data where multiple users have transactions with the same merchant. This high-density connection pattern shows that TransGAN can effectively learn the diversity and complexity of user-merchant transactions in real transaction flow data, avoiding the above-mentioned over-simplification problem of CTGAN and TVAE, so that the generated data has a transaction network structure that is closer to the real one.

[0054] In addition, it is worth noting that the generation of numerical features of the hybrid generative adversarial network model of this embodiment is mainly based on the existing tabular data generation model. In the process of restoring structural information, the model of this embodiment affects the generation quality of numerical features to a certain extent, resulting in the similarity of the data distribution of its generated data may be slightly lower than the corresponding results of existing similar models and methods. However, after multiple experimental comparisons by the inventor, the difference in the similarity of data distribution is very small; and because structural features often contain important behavioral information and relationship information, which are usually more critical than numerical features, the overall effect of the model and method of this embodiment will be significantly better than similar models and methods in the prior art.

[0055] Functions and Effects of the Embodiments

[0056] According to the transaction flow data generation method using a hybrid generative adversarial network provided in this embodiment, repeated transactions in the original transaction flow data are aggregated and structured to generate an original transaction network graph, which is used for training the graph generation module. Therefore, the graph generation module can learn the topological structure characteristics in the real transaction network structure and generate a transaction network structure close to the real one; further, since the transaction network graph and transaction flow data are generated by the graph generation module and the table data generation module respectively, and then integrated to obtain optimized data, through the organic combination of the two modules, high-quality synthetic data that retains the data distribution characteristics and has a real transaction structure can be generated, which makes up for the shortcoming that the existing deep learning model is difficult to capture the structural characteristics of the transaction flow data, and further improves the synthesis quality of the transaction flow data.

[0057] In the embodiment, the original transaction flow data is divided into structural information (user ID and merchant ID) and other transaction information, and the original transaction network graph is constructed with the user ID and merchant ID as nodes and their transaction relationships as edges. The parameters of the generator are updated using the original transaction network graph as training data, so that the transaction network graph generated by the generator can well restore the structural information, reflect the diversity and complexity of user-merchant transactions in real data, and avoid the problem of over-simplification of transactions.

[0058] Furthermore, a random walk algorithm is used in the training of the generator. By analyzing the random walk paths between nodes, the structural characteristics of the network and the importance of nodes can be prompted, so that the generator parameters can be updated simply and effectively, thereby improving the generator training efficiency.

[0059] Furthermore, the integration module combines the generated results by matching the weight information of the edges in the graph structure with the number of transactions in the table data. The processing is relatively simple and the integration efficiency is high. Finally, the final transaction flow table data can be obtained by splitting the edges of each aggregation.

[0060] The above embodiments are only used to illustrate the specific implementation of the present invention, and the present invention is not limited to the description scope of the above embodiments. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for generating pipeline data using a hybrid generative adversarial network, characterized in that: A hybrid generative adversarial network model is used, which includes a preprocessing module, a graph generation module, a table data generation module and an integration module. The method includes the following steps: Step S1, the preprocessing module aggregates and structures repeated transactions in the original transaction flow data to generate an original transaction network diagram; Step S2, using the data set composed of the original transaction network graph to train the graph generation module, and generating a transaction network graph through the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data; Step S3, the integration module integrates the transaction network diagram and the transaction flow data to obtain optimized transaction flow data.

2. The method for generating pipeline data using a hybrid generative adversarial network according to claim 1, characterized in that: in, In step S1, the repeated transactions in the original transaction flow data are aggregated, and the transaction times of the repeated transactions are counted to obtain aggregated data. The original transaction network graph is constructed by using the structural information in the aggregated data as nodes, the transaction relationships as edges, and the transaction times as weight information of the edges.

3. The method for generating pipeline data using a hybrid generative adversarial network according to claim 2, characterized in that: in, The repeated transactions are multiple transactions between the same set of user identification numbers and merchant identification numbers. The structural information includes the user identification number and the merchant identification number, The transaction relationship includes at least transaction amount, transaction time, and transaction type.

4. The method for generating pipeline data using a hybrid generative adversarial network according to claim 2, Features: Wherein, the graph generation module includes: A sampling unit, sampling the original transaction network graph to obtain a real random walk sequence; and The generator and the discriminator are alternately learned based on the real random walk sequence and the generation result of the generator, so that the generator captures the topological structure characteristics in the original transaction network graph.

5. The method for generating pipeline data using a hybrid generative adversarial network according to claim 4, characterized in that: in, In each training round, random noise is input to the generator, a walk sequence is generated using the generator, a loss function for generating the walk sequence is calculated, and the parameters of the generator are updated based on the corresponding loss. In each of the training rounds, the sampling unit extracts the true random walk sequence, inputs the true random walk sequence and the walk sequence generated by the generator into the discriminator, calculates the loss function of the discriminator, and updates the parameters of the discriminator based on the corresponding loss.

6. The method for generating pipeline data using a hybrid generative adversarial network according to claim 4, characterized in that: in, The sampling unit samples the original transaction network graph using a weighted random walk algorithm and sets a walk length, thereby obtaining the true random walk sequence.

7. The method for generating pipeline data using a hybrid generative adversarial network according to claim 2, characterized in that: in, In step S3, the integration module matches the weight information of each edge in the generated transaction network graph with the transaction number information in the generated transaction flow data, thereby combining the generation results of the graph generation module and the table data generation module to obtain combined data.

8. The method for generating pipeline data using a hybrid generative adversarial network according to claim 7, characterized in that: in, In step S2, the table data generation module generates the transaction flow data based on the input aggregate data. In step S3, the integration module further splits the aggregated edges in the combined data to obtain the optimized transaction flow data.

Images