Data simulation system and method
By generating realistic synthetic data through a data simulation system, the challenges of data relationship simulation and privacy protection in existing technologies have been solved, enabling efficient and secure data acquisition and model training.
Patent Information
- Application Number
- CN202511087382.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-05
- Publication Date
- 2025-11-25
AI Technical Summary
Existing data generation tools cannot effectively simulate complex data relationships, such as correlations and causal relationships between data, and data privacy protection is difficult to fully protect, resulting in high data acquisition costs and high risks of privacy leaks.
A data simulation system is provided, including a data input module, an SDV processing module, and a data output module. It utilizes various machine learning algorithms to generate synthetic data, supports multi-table relationships and time series data models, and introduces parallel computing and security management mechanisms to ensure data quality and privacy protection.
Generating realistic synthetic data reduces data acquisition costs, improves data acquisition efficiency and flexibility, protects privacy, and enhances the accuracy and generalization ability of machine learning models.
Smart Images
Figure CN121011301A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, specifically to a data simulation system and method. Background Technology
[0002] With the rapid development of big data and artificial intelligence technologies, data is being applied more and more deeply in various fields, from traditional financial risk assessment to emerging medical and health disease prediction, and then to precision marketing in e-commerce. The value of data is being continuously explored and amplified. However, in the actual use of data, the issues of insufficient data volume and data privacy protection are becoming increasingly prominent. In the machine learning training stage, sufficient and high-quality data is the cornerstone for training high-performance models.
[0003] However, in practice, data acquisition often faces numerous obstacles. For example, data collection is costly, requiring significant investment of human, material, and time resources to collect, organize, and label data. Data privacy is difficult to fully protect; data leaks can cause serious losses to individuals, businesses, and even nations. Traditional data generation tools like Faker and Mimesis can only generate simple pseudo-data, such as names, addresses, and phone numbers. This data is merely a random combination according to a fixed format and cannot effectively simulate complex data relationships, such as correlations and causal relationships between data. Summary of the Invention
[0004] The purpose of this invention is to provide a data simulation system and method to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a data simulation system, comprising a data input module, an SDV processing module, and a data output module;
[0006] The data input module is used to receive actual data from various data sources and perform preliminary cleaning and preprocessing on the input data, including data deduplication, outlier detection, and invalid data deletion.
[0007] The SDV processing module is built on Python's SDV components. It uses a variety of machine learning algorithms to comprehensively learn from the input real data, extract patterns and features from the data, and generate synthetic data based on the learned patterns. It can process and simulate data models containing multi-table relationships, and provides adapted models and tools for data containing time series data or data models that need to simulate complex relationships.
[0008] The data output module is used to output the generated synthetic data to the specified target location, and can also perform format conversion and further processing of the synthetic data according to the user's personalized needs.
[0009] Preferably, the data input module has multiple built-in data reading drivers, is compatible with various data sources such as MySQL, Oracle databases, Hadoop distributed file system, and Amazon S3 storage service, and supports both real-time data acquisition and batch data import modes; it also provides RestAPI service interface and GRPC service interface, and uses the FatsAPI component to implement the RestAPI interface. After users follow the API interface protocol and GRPC interface protocol and pass the authentication, they can use the interface to obtain source data.
[0010] Preferably, when the data input module obtains source data using the RestAPI and GRPC interfaces, it automatically identifies common data types to construct complete metadata, uses parallel computing technology to accelerate the data cleaning and preprocessing process, and introduces the IsolationForest algorithm from machine learning algorithms to assist in outlier detection.
[0011] Preferably, the SDV processing module automatically selects the most suitable SDV model based on the characteristics of the input data and business requirements. For example, it prioritizes the Gaussian conjugate model for data with obvious linear relationships and uses deep neural network-generative adversarial network-related models for data with complex nonlinear relationships. It uses automated hyperparameter tuning techniques, such as random search and Bayesian optimization, to find the optimal parameter combination of the model. It also uses model fusion technology to integrate multiple different SDV models and comprehensively consider the advantages of different models to generate more realistic synthetic data.
[0012] Preferably, the data output module supports outputting the synthesized data in multiple formats such as CSV, JSON, Parquet, and Avro; during the data output process, a strict security and access control mechanism is introduced, supporting multiple encryption algorithms such as AES and RSA to encrypt the output data, and restricting the access level of the synthesized data according to user roles and permissions.
[0013] A method for a data simulation system includes the following steps:
[0014] Data Input Steps: The system receives actual data from various data sources and performs preliminary cleaning and preprocessing. This includes removing duplicate records using deduplication algorithms, eliminating noisy data using outlier detection algorithms, and identifying and deleting invalid data according to business rules. To ensure compatibility with different data sources, multiple data reading drivers are built-in, supporting both real-time data acquisition and batch data import modes. It also provides RestAPI and GRPC service interfaces. The RestAPI interface is implemented using the FatsAPI component, and users can call the services after adhering to and being authenticated according to the API and GRPC protocols. When acquiring source data using the interfaces, common data types are automatically identified to construct complete metadata. Parallel computing technology is used to accelerate the data cleaning and preprocessing process, and the IsolationForest algorithm from machine learning is introduced to assist in outlier detection.
[0015] SDV processing steps: Based on Python's SDV component, various machine learning algorithms are used to comprehensively learn from the input real data, extracting patterns and features from the data, and generating synthetic data based on the learned patterns; it processes and simulates data models containing multi-table relationships, providing adapted models and tools for data models containing time series data or data models that need to simulate complex relationships; according to the characteristics of the input data and business needs, the most suitable SDV model is automatically selected, and the optimal parameter combination of the model is found using automated hyperparameter tuning technology; model fusion technology is used to integrate multiple different SDV models to generate more realistic synthetic data;
[0016] Data output steps: Output the generated synthetic data to the specified target location, and perform format conversion or further processing on the synthetic data according to the user's personalized needs; Supports outputting synthetic data in multiple formats such as CSV, JSON, Parquet, and Avro; During the data output process, a strict security and access control mechanism is introduced, supporting multiple encryption algorithms to encrypt the output data, and restricting the access level of the synthetic data according to the user's role and permissions.
[0017] Preferably, in the data input step, the multiple data reading drivers are compatible with various data sources such as MySQL, Oracle databases, Hadoop distributed file systems, and Amazon S3 storage services; the RestAPI service interface and the GRPC service interface follow the corresponding interface protocols, and after user authentication, they can use the interface to obtain actual data from different types of data sources.
[0018] Preferably, in the SDV processing step, for data with obvious linear relationships, the Gaussian conjugate model is selected for data processing; for data with complex nonlinear relationships, deep neural network-generative adversarial network-related models are used; automated hyperparameter tuning techniques include random search and Bayesian optimization; model fusion technology is to fuse the results of multiple different SDV models, taking into account the advantages of different models, such as fusing the results of the Gaussian conjugate model and the results of the deep learning-based generative model.
[0019] Preferably, in the data output step, the format conversion includes converting CSV format data to JSON format, or converting the data to a Parquet or Avro efficient binary format; further processing includes encrypting the synthesized data; multiple encryption algorithms including AES and RSA; and a security and access control mechanism that sets different access levels according to user roles and permissions to ensure that only authorized users can obtain and use the corresponding synthesized data.
[0020] Preferably, the method as a whole is implemented through the collaborative implementation of a data input module, an SDV processing module, and a data output module. The modules interact with each other through an efficient data transmission interface to ensure the smooth flow of data in the method process and to have good scalability, making it easy to add new functional modules or optimize existing modules according to business needs in the future.
[0021] Compared with the prior art, the beneficial effects of the present invention are:
[0022] The data simulation system and method proposed in this invention utilize synthetic data to replace real data for analysis and training, effectively protecting sensitive information in the original data and significantly reducing the risk of data leakage. In the medical field, using synthetic patient medical record data for medical research can meet research needs while protecting patient privacy. There is no need to expend significant resources collecting and organizing real data; this device can quickly generate synthetic data that meets the requirements, greatly improving the efficiency and flexibility of data acquisition. In market research, previously, large sums of money were spent on questionnaires to collect consumer data; now, by generating synthetic data through this device, the required consumer behavior data can be obtained quickly. The generated synthetic data can accurately simulate the patterns and characteristics of real data, providing rich data support for machine learning models and helping to improve the accuracy and generalization ability of the models. Attached Figure Description
[0023] Figure 1 This is a system block diagram of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Example 1, please refer to Figure 1 This invention provides a technical solution: a data simulation system comprising three main parts: a data input module, an SDV processing module, and a data output module. These three modules cooperate to complete the entire process from actual data input to synthesized data output, with each module playing an indispensable role. The modules interact through efficient data transmission interfaces, ensuring smooth data flow within the device and possessing good scalability, facilitating the addition of new functional modules or optimization of existing modules according to business needs.
[0026] Data input module
[0027] This module is responsible for receiving actual data from various data sources (such as databases and file systems). In practical applications, data sources may include internal relational databases storing customer information, transaction records, and other data; or log files in a distributed file system recording user behavior data. Simultaneously, it performs preliminary cleaning and preprocessing of the input data, removing duplicate records using deduplication algorithms, eliminating noisy data using outlier detection algorithms, and identifying and deleting invalid data according to business rules, thus providing a high-quality data foundation for subsequent SDV processing modules.
[0028] Data reception adaptation: To ensure compatibility with different data sources, the data input module has built-in multiple data reading drivers, enabling seamless integration with common databases such as MySQL and Oracle, as well as Hadoop Distributed File System (HDFS) and Amazon S3 storage services. It also supports both real-time data acquisition and batch data import modes to meet the data acquisition needs of different business scenarios.
[0029] Multiple services are provided: In addition to the data reception and adaptation functions mentioned above, this invention also provides RestAPI and GRPC service interfaces to adapt to various application scenarios. This invention uses the FatsAPI component to implement the RestAPI interface. Users only need to follow the API interface protocol and GRPC interface protocol to quickly call the service (provided that authentication is successful).
[0030] Data preprocessing optimization: When acquiring source data using the RESTAPI and GRPC interfaces, the device automatically identifies common data types (such as phone numbers, addresses, and ID cards) to facilitate the construction of complete metadata. Furthermore, the device employs parallel computing technology to accelerate the data cleaning and preprocessing process, significantly reducing processing time for large-scale datasets. Simultaneously, the device incorporates machine learning algorithms to assist in outlier detection, such as the Isolation Forest algorithm, which effectively identifies outliers in the data. Compared to traditional statistical methods for outlier detection, the Isolation Forest algorithm demonstrates higher accuracy and adaptability.
[0031] SDV processing module
[0032] This module, serving as the core of the entire device, is built upon Python's SDV component. It utilizes various machine learning algorithms within the SDV component. Taking the Gaussian Copula model as an example, this model learns the correlation of data by constructing a joint probability distribution between variables, comprehensively learning from the input real data to accurately extract patterns and features. Subsequently, synthetic data is generated based on the learned patterns.
[0033] For multi-table data, this module can process and simulate data models that include relationships between multiple tables, such as user tables, order tables, and product tables in e-commerce scenarios.
[0034] In addition, for data models that include time series data or need to simulate complex relationships, the SDV processing module provides suitable models and tools, such as the time series model PAR (Probabilistic Autoregressive Model), which takes into account the autocorrelation and trend of time series data.
[0035] Model selection and tuning: Based on the characteristics of the input data and business requirements, the most suitable SDV model is automatically selected. For example, for data with obvious linear relationships, the Gaussian conjugate model is preferred; for data with complex nonlinear relationships, deep neural network generative adversarial networks (GANs) are used. Simultaneously, automated hyperparameter tuning techniques, such as random search and Bayesian optimization, are used to find the optimal parameter combination of the model to improve the quality of the synthesized data.
[0036] Model fusion and integration: To further improve the accuracy and reliability of synthetic data, model fusion technology is employed to integrate multiple different SDV models. For example, the results of Gaussian conjugate models and deep learning-based generative models are fused to comprehensively consider the advantages of different models and generate more realistic synthetic data.
[0037] Data output module
[0038] This module is responsible for outputting the generated synthetic data to a specified target location, such as a database or file system, for convenient subsequent data use and analysis. It can also perform format conversions on the synthetic data according to the user's personalized needs, such as converting CSV data to JSON format to suit different application scenarios; or perform further processing, such as encrypting the synthetic data to enhance data security.
[0039] Output format expansion: In addition to common CSV and JSON formats, the data output module also supports outputting synthesized data in efficient binary formats such as Parquet and Avro. These formats have better performance in big data storage and processing scenarios and can effectively reduce data storage space and data transmission time.
[0040] Security and Access Control: A strict security and access control mechanism is implemented during data output. Multiple encryption algorithms, such as AES (Advanced Encryption Standard) and RSA, are supported to encrypt the output data, ensuring its security during transmission and storage. Furthermore, access levels to the synthesized data are restricted based on user roles and permissions; only authorized users can obtain and use the corresponding data.
[0041] Example 2, based on Example 1, proposes a method for a data simulation system, including the following steps:
[0042] Data Input Steps: The system receives actual data from various data sources, performs preliminary cleaning and preprocessing, including removing duplicate records using deduplication algorithms, eliminating noisy data using outlier detection algorithms, and identifying and deleting invalid data according to business rules. To ensure compatibility with different data sources, multiple built-in data reading drivers are provided, supporting both real-time data acquisition and batch data import modes. RestAPI and GRPC service interfaces are also provided. The RestAPI interface is implemented using the FatsAPI component, and users can call the services after adhering to the API and GRPC interface protocols and passing authentication. When using the interfaces to obtain source data, common data types are automatically identified to construct complete metadata. Parallel computing technology is used to accelerate the data cleaning and preprocessing process, and the IsolationForest algorithm from machine learning is introduced to assist in outlier detection. Multiple data reading drivers are compatible with MySQL, Oracle databases, Hadoop Distributed File System, Amazon S3 storage services, and other data sources. The RestAPI and GRPC service interfaces follow their respective interface protocols, and users can obtain actual data from different data sources after authentication.
[0043] SDV processing steps: Based on Python's SDV components, various machine learning algorithms are used to comprehensively learn from the input real-world data, extracting patterns and features, and generating synthetic data based on the learned patterns; Data models containing multi-table relationships are processed and simulated, providing suitable models and tools for data containing time series or data models requiring the simulation of complex relationships; Based on the characteristics of the input data and business needs, the most suitable SDV model is automatically selected, and automated hyperparameter tuning techniques are used to find the optimal parameter combination of the model; Model fusion techniques are used to integrate multiple different SDV models to generate more realistic synthetic data; For data with obvious linear relationships, Gaussian conjugate models are prioritized for data processing; For data with complex nonlinear relationships, deep neural network and generative adversarial network models are used; Automated hyperparameter tuning techniques include random search and Bayesian optimization; Model fusion technology combines the results of multiple different SDV models, comprehensively considering the advantages of different models, such as fusing the results of Gaussian conjugate models and deep learning-based generative models.
[0044] Data Output Steps: The generated synthetic data is output to the specified target location. The synthetic data undergoes format conversion or further processing based on the user's personalized needs. It supports outputting synthetic data in multiple formats, including CSV, JSON, Parquet, and Avro. During the data output process, a strict security and access control mechanism is implemented, supporting multiple encryption algorithms to encrypt the output data. Access levels to the synthetic data are restricted based on user roles and permissions. Format conversion includes converting CSV data to JSON format, or converting data to efficient binary formats such as Parquet and Avro. Further processing includes encrypting the synthetic data. Multiple encryption algorithms include AES and RSA. The security and access control mechanism sets different access levels based on user roles and permissions, ensuring that only authorized users can access and use the corresponding synthetic data.
[0045] The method as a whole is implemented through the collaborative efforts of the data input module, SDV processing module, and data output module. The modules interact with each other through efficient data transmission interfaces to ensure the smooth flow of data in the method process and have good scalability, making it easy to add new functional modules or optimize existing modules according to business needs in the future.
[0046] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A data simulation system, characterized in that: It includes a data input module, an SDV processing module, and a data output module; The data input module is used to receive actual data from various data sources and perform preliminary cleaning and preprocessing on the input data, including data deduplication, outlier detection, and invalid data deletion. The SDV processing module is built on Python's SDV components. It uses a variety of machine learning algorithms to comprehensively learn from the input real data, extract patterns and features from the data, and generate synthetic data based on the learned patterns. It can process and simulate data models containing multi-table relationships, and provides adapted models and tools for data containing time series data or data models that need to simulate complex relationships. The data output module is used to output the generated synthetic data to the specified target location, and can also perform format conversion and further processing of the synthetic data according to the user's personalized needs.
2. The data simulation system according to claim 1, characterized in that: The data input module has multiple built-in data reading drivers, is compatible with various data sources such as MySQL, Oracle databases, Hadoop distributed file system, and Amazon S3 storage service, and supports both real-time data acquisition and batch data import modes. It also provides RestAPI and GRPC service interfaces. The RestAPI interface is implemented using the FatsAPI component. After users comply with the API interface protocol and GRPC interface protocol and pass the authentication, they can use the interface to obtain source data.
3. The data simulation system according to claim 2, characterized in that: When the data input module obtains source data using the RestAPI and GRPC interfaces, it automatically identifies common data types to construct complete metadata, uses parallel computing technology to accelerate the data cleaning and preprocessing process, and introduces the IsolationForest algorithm from machine learning algorithms to assist in outlier detection.
4. A data simulation system according to claim 3, characterized in that: The SDV processing module automatically selects the most suitable SDV model based on the characteristics of the input data and business requirements. For example, it prioritizes the Gaussian conjugate model for data with obvious linear relationships and uses deep neural network-generative adversarial network-related models for data with complex nonlinear relationships. It uses automated hyperparameter tuning techniques, such as random search and Bayesian optimization, to find the optimal parameter combination of the model. It also uses model fusion technology to integrate multiple different SDV models and comprehensively consider the advantages of different models to generate more realistic synthetic data.
5. A data simulation system according to claim 4, characterized in that: The data output module supports outputting synthesized data in multiple formats, including CSV, JSON, Parquet, and Avro. During the data output process, a strict security and access control mechanism is introduced, supporting multiple encryption algorithms such as AES and RSA to encrypt the output data, and restricting the access level of the synthesized data according to user roles and permissions.
6. A method for a data simulation system according to claim 5, characterized in that: Includes the following steps: Data Input Steps: The system receives actual data from various data sources and performs preliminary cleaning and preprocessing. This includes removing duplicate records using deduplication algorithms, eliminating noisy data using outlier detection algorithms, and identifying and deleting invalid data according to business rules. To ensure compatibility with different data sources, multiple data reading drivers are built-in, supporting both real-time data acquisition and batch data import modes. It also provides RestAPI and GRPC service interfaces. The RestAPI interface is implemented using the FatsAPI component, and users can call the services after adhering to and being authenticated according to the API and GRPC protocols. When acquiring source data using the interfaces, common data types are automatically identified to construct complete metadata. Parallel computing technology is used to accelerate the data cleaning and preprocessing process, and the IsolationForest algorithm from machine learning is introduced to assist in outlier detection. SDV processing steps: Based on Python's SDV component, various machine learning algorithms are used to comprehensively learn from the input real data, extracting patterns and features from the data, and generating synthetic data based on the learned patterns; it processes and simulates data models containing multi-table relationships, providing adapted models and tools for data models containing time series data or data models that need to simulate complex relationships; according to the characteristics of the input data and business needs, the most suitable SDV model is automatically selected, and the optimal parameter combination of the model is found using automated hyperparameter tuning technology; model fusion technology is used to integrate multiple different SDV models to generate more realistic synthetic data; Data output steps: Output the generated synthetic data to the specified target location, and perform format conversion or further processing on the synthetic data according to the user's personalized needs; Supports outputting synthetic data in multiple formats such as CSV, JSON, Parquet, and Avro; During the data output process, a strict security and access control mechanism is introduced, supporting multiple encryption algorithms to encrypt the output data, and restricting the access level of the synthetic data according to the user's role and permissions.
7. A method according to claim 6, characterized in that: In the data input step, multiple data reading drivers are compatible with various data sources, including MySQL, Oracle databases, Hadoop Distributed File System, and Amazon S3 storage services. The RestAPI service interface and GRPC service interface follow the corresponding interface protocols. After authentication, users can use the interface to obtain actual data from different types of data sources.
8. A method according to claim 7, characterized in that: In the SDV processing steps, for data with obvious linear relationships, Gaussian conjugate models are preferred for data processing; for data with complex nonlinear relationships, deep neural network-generative adversarial network-related models are used; automated hyperparameter tuning techniques include random search and Bayesian optimization; model fusion technology combines the results of multiple different SDV models, taking into account the advantages of different models, such as combining the results of Gaussian conjugate models and deep learning-based generative models.
9. A method according to claim 8, characterized in that: In the data output step, format conversion includes converting CSV format data to JSON format, or converting data to efficient binary formats such as Parquet and Avro; further processing includes encrypting the synthesized data; multiple encryption algorithms including AES and RSA; and a security and access control mechanism sets different access levels based on user roles and permissions to ensure that only authorized users can obtain and use the corresponding synthesized data.
10. A method according to claim 9, characterized in that: The method as a whole is implemented through the collaborative efforts of the data input module, SDV processing module, and data output module. The modules interact with each other through efficient data transmission interfaces to ensure the smooth flow of data in the method process and have good scalability, making it easy to add new functional modules or optimize existing modules according to business needs in the future.