Stream data generation method using hybrid generative adversarial network

Abstract

The application provides a kind of stream data generation method using hybrid generative adversarial network, the repeated transaction in original transaction stream data is aggregated and structured, generates original transaction network graph, which is used for the training of graph generation module, so that the graph generation module can learn the topological structure characteristics in the real transaction network structure, and generate a transaction network structure close to reality;Further, since the transaction network graph and the transaction stream data are generated by the graph generation module and the table data generation module respectively, and then integrated to obtain the optimized data, through the organic combination of the two modules, high-quality synthetic data that retains data distribution characteristics and has real transaction structure can be generated, which makes up for the short board that the existing deep learning model is difficult to capture the structural characteristics of transaction stream data, and further improves the synthesis quality of transaction stream data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of table data generation, and particularly relates to a flow data generation method using a hybrid generative adversarial network. BACKGROUND

[0002] Under the background of rapid development of big data and artificial intelligence, the financial industry is undergoing profound changes. Financial data, as a core resource, involves sensitive content such as user personal information, account records, and transaction flow, and is widely used in customer management, risk control, market marketing, anti-fraud, and other scenarios. Its high sensitivity and high value make it face the dual challenges of privacy protection and systemic risk. Once leaked or destroyed, it may have a serious impact on users, institutions, and even the stability of the financial market. Therefore, protecting the security of financial data has become an industry consensus and research focus.

[0003] At the same time, financial institutions have very high requirements for the security, integrity, and confidentiality of massive customer and transaction data. These data support risk assessment, credit assessment, market analysis, and other key businesses. However, with the increasing strictness of privacy protection regulations, data acquisition and use are becoming more difficult, leading to the intensification of data silos and information asymmetry, limiting data analysis capabilities and competitiveness. How to efficiently use data while ensuring compliance and privacy protection is a core problem that the financial industry needs to solve.

[0004] Synthetic data is a form of data that does not directly map to real personal information, and can be freely used, shared, and stored, effectively avoiding privacy leakage risks. Compared with traditional methods, synthetic data technology based on deep learning can better preserve the statistical and structural characteristics of data and better meet the needs of financial technology products and model training. Its applications cover customer analysis, credit decision-making, risk management, anti-fraud, and other scenarios, significantly improving data utilization efficiency while reducing legal and ethical risks.

[0005] Currently, table data generation models are generally divided into two approaches:

[0006] One is a likelihood-based generation method. This method uses log-likelihood or a suitable substitute as the training target, and must use a specific architecture to build a normalized probability model (autoregressive model, flow model) or use a substitute loss (VAE). For a given data set {x1, x2, …, x n}Under a distribution p(x), the goal is to train a generator to approximate the distribution. Finally, by sampling from the approximate distribution, generated data X' is obtained. Variational auto-encoder (VAE), flow-based model and diffusion model are likelihood-based methods. VAE includes two similar networks, an encoder and a decoder. The encoder receives the input and converts it into a smaller dimensional representation, and the decoder can use the representation to convert it into the original input. The latent space they convert the input to and the space where their encoding vectors are located may not be continuous. To solve this problem, variational auto-encoder has a continuous latent space feature, making random sampling and interpolation operations more convenient. Flow-based model directly calculates the probability distribution, which is composed of a series of reversible functions f i , the inverse of each f i and the Jacobian determinant. Diffusion model is a model inspired by non-equilibrium thermodynamics, which defines a Markov chain that gradually adds random noise to the data, and then learns the inverse diffusion process to construct the required data samples from the noise. Variants such as TVAE, DDPM have been generated for tabular data.

[0007] Another is the adversarial-based generation method, of which the most representative model is generative adversarial networks (GAN). GAN is an unsupervised model that takes random noise as input and generates output, and is trained in parallel through the mutual opposition and mutual learning of the generator and the discriminator. Many variants of GAN have also appeared for tabular data, such as MedGAN, TableGAN, CTGAN, CTAB-GAN, CTAB-GAN+, etc.

[0008] However, existing tabular data generation techniques can usually only restore the distribution of features when generating transaction flow data, while ignoring the structural information in the transaction flow data set. Specifically, users often have a large number of repeated transactions with the same merchant, and existing generative adversarial network (GAN) based tabular data generation models usually cannot effectively capture such repeated features, and can only restore the distribution characteristics of numerical values, resulting in substantial differences between the generated transaction flow data and the actual data, and thus affecting the accuracy of related evaluation and analysis based on generated data. SUMMARY

[0009] The application is carried out to solve the above problems, and aims to provide a transaction flow data generation method capable of generating high-quality synthetic data which not only retains data distribution characteristics but also has a real transaction structure.

[0010] The application provides a transaction flow data generation method using a hybrid generative adversarial network, which has the technical features of using a hybrid generative adversarial network model, the model including a preprocessing module, a graph generation module, a table data generation module and an integration module, the method including the following steps: step S1, the preprocessing module aggregates and structures repeated transactions in original transaction flow data to generate an original transaction network graph; step S2, the graph generation module is trained using a data set composed of the original transaction network graph, and a transaction network graph is generated through the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data; step S3, the integration module integrates the transaction network graph and the transaction flow data to obtain optimized transaction flow data.

[0011] The transaction flow data generation method using a hybrid generative adversarial network provided by the application can also have the technical features that in step S1, the repeated transactions in the original transaction flow data are aggregated, and the number of transactions of the repeated transactions is counted to obtain aggregated data, the structure information in the aggregated data is taken as a node, the transaction relationship is taken as an edge, and the number of transactions is taken as weight information of the edge to construct the original transaction network graph.

[0012] The transaction flow data generation method using a hybrid generative adversarial network provided by the application can also have the technical features that the repeated transactions are multiple transactions between the same group of user identification numbers and merchant identification numbers, the structure information includes the user identification numbers and the merchant identification numbers, and the transaction relationship includes at least transaction amount, transaction time and transaction type.

[0013] The transaction flow data generation method using a hybrid generative adversarial network provided by the application can also have the technical features that the graph generation module includes a sampling unit, which samples the original transaction network graph to obtain a real random walk sequence, and a generator and a discriminator, which perform alternating learning based on the real random walk sequence and the generation result of the generator to enable the generator to capture the topological structure characteristics in the original transaction network graph.

[0014] The flow data generation method using the hybrid generative adversarial network provided by the application can also have the following technical features: in each training round, random noise is input to the generator, a walk sequence is generated using the generator, a loss function for generating the walk sequence is calculated, and the parameters of the generator are updated based on the corresponding loss; in each training round, the sampling unit extracts the real random walk sequence, inputs the real random walk sequence and the walk sequence generated by the generator into the discriminator, calculates a loss function of the discriminator, and updates the parameters of the discriminator based on the corresponding loss.

[0015] The flow data generation method using the hybrid generative adversarial network provided by the application can also have the following technical features: the sampling unit samples the original transaction network graph using a weighted random walk algorithm and sets a walk length, so as to obtain the real random walk sequence.

[0016] The flow data generation method using the hybrid generative adversarial network provided by the application can also have the following technical features: in step S3, the integration module matches the weight information of each edge in the generated transaction network graph with the transaction frequency information in the generated transaction flow data, so as to combine the generation results of the graph generation module and the table data generation module to obtain combined data.

[0017] The flow data generation method using the hybrid generative adversarial network provided by the application can also have the following technical features: in step S2, the table data generation module generates the transaction flow data based on the input aggregated data; and in step S3, the integration module further splits the aggregated edges in the combined data, so as to obtain the optimized transaction flow data.

[0018] Effects of the application

[0019] The flow data generation method using the hybrid generative adversarial network provided by the application aggregates repeated transactions in original transaction flow data and structures them, generates an original transaction network graph, and uses the original transaction network graph for training of a graph generation module, so that the graph generation module can learn topological structure features in a real transaction network structure and generate a transaction network structure close to reality; further, the transaction network graph and the transaction flow data are generated by the graph generation module and the table data generation module respectively, and then the transaction network graph and the transaction flow data are integrated to obtain optimized data, so that high-quality synthetic data that retains data distribution features and has a real transaction structure can be generated through organic combination of the two modules, the short board that a deep learning model in the prior art is difficult to capture structure features of transaction flow data is made up for, and the synthesis quality of transaction flow data is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a schematic diagram of the key steps of the method for generating transaction data streams by using a hybrid generative adversarial network in an embodiment of the present application;

[0021] Figure 2 is a flowchart of the method for generating transaction data streams by using a hybrid generative adversarial network in an embodiment of the present application;

[0022] Figure 3 is an example diagram of aggregating and counting repeated transactions in an embodiment of the present application;

[0023] Figure 4 is a comparison diagram of the transaction network graph of the original transaction network graph and the transaction data streams generated by the three models in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the method for generating transaction data streams by using a hybrid generative adversarial network of the present application will be specifically described below in combination with embodiments and drawings.

[0025] [EMBODIMENT]

[0026] Figure 1 is a schematic diagram of the key steps of the method for generating transaction data streams by using a hybrid generative adversarial network in the present embodiment, Figure 2 is a flowchart of the method for generating transaction data streams by using a hybrid generative adversarial network in the present embodiment.

[0027] As shown in Figure 1 and Figure 2 , the hybrid generative adversarial network model includes a preprocessing module, a graph generation module, a table data generation module and an integration module. The method for generating transaction data streams by using a hybrid generative adversarial network includes the following three key steps:

[0028] Step S1, the preprocessing module aggregates and structures the repeated transactions in the original transaction data stream set to generate an original transaction network graph.

[0029] Step S2, the graph generation module is trained by using the data set composed of the original transaction network graph, and the transaction network graph is generated by the trained graph generation module, and the table data generation module generates the transaction data stream based on the input original transaction data stream set.

[0030] Step S3, the integration module integrates the transaction network graph and the transaction data stream to obtain the optimized transaction data stream.

[0031] The above steps will be described in detail below.

[0032] Step S1, the preprocessing module aggregates and structures the repeated transactions in the original transaction flow data set to generate an original transaction network graph.

[0033] In order to better process and analyze the data, the original transaction flow data set is divided into two parts: a structure information part and a transaction information part. The structure information mainly includes user ID (unique identification number) and merchant ID. The transaction information covers transaction amount, transaction time, transaction type and other contents.

[0034] Due to the large number of repeated transactions between the same node pairs (user ID and merchant ID node pairs) in the transaction flow data, in order to better generate data in the subsequent step of the hybrid generative adversarial network model, in step S1, the preprocessing module aggregates the repeated transactions in the original transaction flow data, and counts the number of repeated transactions between the user ID and the merchant ID, to obtain aggregated data. Then, the user ID and merchant ID node pairs in the aggregated data are taken as a node set, and other transaction information is taken as an edge set, to obtain an original transaction network graph composed of the structure information in the original transaction flow data, and a transaction network data set can be composed of the original transaction network graph.

[0035] Figure 3 is an example graph for aggregating and counting repeated transactions in this embodiment.

[0036] As Figure 3 As shown in the table above, in the original flow data, the user with user ID (uid) of 31xxxx46 and the merchant with merchant ID (mer_no) of 808xxxxxxxxx000 have repeated transactions. According to the uid-cid pair, the repeated multiple transactions are aggregated, and the fields such as uid, mer_no, card ID (cid), card type (card-type), merchant type (mer_type), transaction type (trans_type), and transaction date (pt_dt) and their data are retained after aggregation, and transaction weight (weight) and transaction times (times) fields are added, wherein the transaction weight can be calculated based on the amount of each transaction and the number of repeated transactions, thereby generating aggregated data.

[0037] Then, the uid and mer_no in the aggregated data are taken as nodes to form a node set, the other transaction information (transaction records) in the aggregated data are taken as aggregated edges to form an edge set, and the corresponding transaction times are taken as weight information of each aggregated edge, to build an original transaction network graph, thereby obtaining a transaction network data set.

[0038] Step S2, the graph generation module is trained by using the data set composed of the original transaction network graph, and a transaction network graph is generated by the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data set.

[0039] The table data generation module can use an existing table data generation model. Since the input is an aggregated data set, the generated transaction flow data is also an aggregated data set with similar composition, which contains the above-mentioned transaction repetition times.

[0040] The graph generation module is specially designed to process complex transaction network structures, which includes a sampling unit, a generator, and a discriminator, and the generator and the discriminator constitute a generative adversarial network. The sampling unit uses a weighted random walk algorithm to sample the original transaction network graph composed of structure information to obtain a real random walk sequence. Then, based on the real random walk sequence and the generation result of the generator, the generator and the discriminator are alternately learned, and the parameters of the generator and the discriminator are updated respectively, so that the generator can capture the topological structure features in the real transaction network graph. Finally, the updated generator is used to finally generate the transaction network graph. In the generated transaction network graph, the nodes are user IDs and merchant IDs for transaction, the edges are their corresponding transaction records, and the edges have weight information, which is the above-mentioned transaction times.

[0041] The following Table 1 shows the pseudo code of the graph generation module.

[0042] Table 1 Pseudo code of the graph generation module

[0043]

[0044]

[0045] Step S3, the generated transaction network graph and the transaction flow data are integrated by the integration module to obtain optimized transaction flow data.

[0046] The core idea of the integration is to combine the transaction network structure graph generated by the graph generation module with the table feature data output by the table data generation module.

[0047] Specifically, the weight information of each edge in the transaction network graph is matched with the transaction times information previously added in the generated transaction flow data, so as to combine the generation results of the graph generation module and the table data generation module, and obtain the combined data. Since the repeated multiple transactions are aggregated in step S1, the aggregated edges in the combined data also need to be split, so as to obtain the final structure-optimized transaction flow data.

[0048] This integration approach ensures that the generated transaction log data is not only structurally very similar to real data, but also closely approximates the actual transaction situation in terms of structural feature distribution. Compared to existing tabular data generation models, the model and method in this embodiment can better maintain the diversity and high repeatability of the generated transaction log data in terms of topological structure.

[0049] The inventors compared the model and method of this embodiment with two models and methods in the prior art.

[0050] Figure 4 This is a comparison diagram of the original transaction network diagram and the transaction network diagrams generated by the three models in this embodiment.

[0051] like Figure 4 As shown, the two existing models are CTGAN and TVAE. Transaction flow data were generated based on the same original transaction flow data using these two existing models and methods, as well as the TransGAN model and method of this embodiment. Then, structural analysis and visualization were performed on the original transaction flow data, the data generated by the two existing models, and the data generated by the TransGAN model of this embodiment, thereby forming… Figure 4 .

[0052] from Figure 4 It can be intuitively seen that the data generated by CTGAN and TVAE show more one-to-one transaction relationships. Their distribution characteristics (structural characteristics) are quite different from the distribution characteristics of the original data. Furthermore, the data they generate contains some oversimplified transaction patterns, such as random transactions between a user and a large number of merchants. These features rarely appear in real transaction flow data.

[0053] In comparison, the distribution characteristics of the data generated by TransGAN in this embodiment are significantly more consistent with the distribution characteristics of the original data. The data shows a high variety of transactions between multiple users and the same merchant. This high-density connection pattern indicates that TransGAN can effectively learn the diversity and complexity of user-merchant transactions in real transaction flow data, avoiding the oversimplification problem mentioned above in CTGAN and TVAE, and making the generated data have a more realistic transaction network structure.

[0054] In addition, it is worth noting that the generation of the numerical features of the hybrid generative adversarial network model of the embodiment is mainly based on the existing table data generation model. In the process of restoring the structural information, the model of the embodiment affects the generation quality of the numerical features to some extent, resulting in that the data distribution similarity of the generated data may be slightly lower than the corresponding results of the existing similar models and methods. However, through the comparison of the inventors' multiple experiments, the gap of the data distribution similarity is very small. Moreover, since the structural features often contain important behavior information and relationship information, they are usually more critical than the numerical features. Therefore, the overall effect of the model and method of the embodiment will be significantly better than the similar models and methods in the prior art.

[0055] Effects of the embodiments

[0056] According to the flow data generation method using the hybrid generative adversarial network provided in the embodiment, the repeated transactions in the original transaction flow data are aggregated and structured, and the original transaction network graph is generated, which is used for the training of the graph generation module. Therefore, the graph generation module can learn the topological structure features in the real transaction network structure and generate a transaction network structure close to the real transaction network structure. Further, since the transaction network graph and the transaction flow data are generated by the graph generation module and the table data generation module respectively, and then integrated to obtain the optimized data, through the organic combination of the two modules, high-quality synthetic data that retains the data distribution features and has the real transaction structure can be generated, which makes up for the short board of the existing deep learning model that is difficult to capture the structural feature of the transaction flow data, and further improves the synthesis quality of the transaction flow data.

[0057] In the embodiment, the original transaction flow data is divided into structural information (user ID and merchant ID) and other transaction information, and the user ID and the merchant ID are taken as nodes and their transaction relationship is taken as an edge to construct an original transaction network graph. The parameters of the generator are updated by taking the original transaction network graph as training data, so that the transaction network graph generated by the generator can well restore the structural information and reflect the diversity and complexity of the user-merchant transaction in the real data, avoiding the problem of over-simplification of the transaction.

[0058] Further, in the training of the generator, the random walk algorithm is used. By analyzing the random walk path between nodes, the structural features of the network and the importance of the nodes can be prompted, so that the generator parameters can be simply and effectively updated, and the training efficiency of the generator can be improved.

[0059] Further, the integration module combines the weight information of the edges in the graph structure with the transaction times in the table data to realize the combination of the generation results. The processing is relatively simple, and the integration efficiency is high. Finally, by splitting each aggregated edge, the final transaction flow table data can be obtained.

[0060] The above examples are only used for illustrating the specific embodiments of the present application, and the present application is not limited to the description range of the above examples. It should be understood by those skilled in the art that the present application is not limited by the above examples, and the above examples and the description in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for generating streaming data using a hybrid generative adversarial network, characterized in that, The method comprises the following steps: Step S1, the preprocessing module aggregates and structures the repeated transactions in the original transaction flow data to generate an original transaction network graph; Step S2, the graph generation module is trained using a data set composed of the original transaction network graph, and a transaction network graph is generated by the trained graph generation module, and the table data generation module generates transaction flow data based on the input original transaction flow data; Step S3, the integration module integrates the transaction network graph and the transaction flow data to obtain optimized transaction flow data, In step S1, the repeated transactions in the original transaction flow data are aggregated, and the number of transactions of the repeated transactions is counted to obtain aggregated data, The structure information in the aggregated data is used as a node, the transaction relationship is used as an edge, and the number of transactions is used as the weight information of the edge to construct the original transaction network graph, The graph generation module comprises: A sampling unit that samples the original transaction network graph to obtain a real random walk sequence; and A generator and a discriminator that perform alternating learning based on the real random walk sequence and the generation result of the generator to enable the generator to capture the topological structure features in the original transaction network graph.

2. The method of claim 1, wherein: wherein, The repeated transactions are multiple transactions between the same group of user identification numbers and merchant identification numbers, The structure information includes the user identification numbers and the merchant identification numbers, The transaction relationship includes at least transaction amount, transaction time, and / or transaction type.

3. The method of claim 1, wherein: wherein In each training round, a random noise is input to the generator, a walk sequence is generated using the generator, a loss function for generating the walk sequence is calculated, and the parameters of the generator are updated based on the corresponding loss, In each training round, the sampling unit extracts the real random walk sequence, inputs the real random walk sequence and the walk sequence generated by the generator into the discriminator, calculates the loss function of the discriminator, and updates the parameters of the discriminator based on the corresponding loss.

4. The method of claim 1, wherein: wherein The sampling unit samples the original transaction network graph using a weighted random walk algorithm and sets a walk length to obtain the real random walk sequence.

5. The method of claim 1, wherein: wherein In step S3, the integration module matches the weight information of each edge in the generated transaction network graph with the transaction frequency information in the generated transaction flow data, thereby combining the results generated by the graph generation module and the table data generation module to obtain combined data.

6. The transaction flow data generation method using a hybrid generative adversarial network according to claim 5, characterized in that: wherein, In step S2, the table data generation module generates the transaction flow data based on the input aggregated data, In step S3, the integration module further splits the aggregated edges in the combined data, thereby obtaining the optimized transaction flow data.

Images