Multi-source heterogeneous data generation system and method based on production data model

Abstract

The embodiment of the invention provides a multi-source heterogeneous data generation system and method based on a production data model, and relates to the technical field of data generation. The system comprises a base layer used for providing data generation template rendering, data storage and pressure generation functions through corresponding modules in the base layer; the service layer is used for providing basic operation, intelligent data generation, relational database data construction and file / HIVE data construction functions through a corresponding module in the service layer; and the application layer is used for providing a data construction scene arrangement function, a multi-source database information acquisition function, a production data model acquisition function, a data preview generation function and a recording execution function through a corresponding module in the application layer. According to the implementation mode provided by the invention, an orchestrable framework of a high-performance load test tool is integrated, real-time dynamic generation and efficient injection of the test data are realized through the design of an automatic test assembly line, and powerful data support is provided for performance test data construction.

Description

Technical Field

[0001] This application relates to the field of data generation technology, specifically to a method for generating multi-source heterogeneous data based on a production data model, a system for generating multi-source heterogeneous data based on a production data model, an electronic device, and a corresponding storage medium. Background Technology

[0002] In existing performance testing practices, test data construction typically relies on static datasets or data generation methods based on simple randomization algorithms. This data is often generated through specific scripts or database stored procedures. However, these traditional methods have significant limitations: Firstly, they struggle to effectively simulate data with specific distribution characteristics and dynamic changes characteristic of real-world business scenarios. For instance, in high-concurrency systems such as finance and e-commerce, user information, transaction records, and time series data are not only massive in volume but also contain complex business relationships and statistical characteristics. Data generated by traditional methods lacks diversity, authenticity, and scalability, leading to significant discrepancies between the test and production environments and affecting the accuracy and reliability of test conclusions. Secondly, script-based data construction methods are costly to maintain, have poor reusability, and are difficult to adapt to complex and ever-changing test scenario requirements, thus hindering improvements in test coverage and efficiency.

[0003] On the other hand, traditional data generation technologies typically rely heavily on a single type of database system, lacking effective support for multi-source, heterogeneous data environments. As enterprises deepen their digital transformation, to meet the needs of different business scenarios, most have built data ecosystems encompassing various data storage and processing technologies. Typical components include, but are not limited to, relational databases (such as PostgreSQL, Oracle, and MySQL), data warehouse infrastructure (such as Hive), real-time SQL query engines (such as Impala), and real-time analytical databases (such as Doris). Against this backdrop, achieving efficient and consistent test data generation across multiple databases has become a key challenge for enterprises implementing end-to-end performance testing.

[0004] Existing data generation solutions are mostly based on predefined rule templates or limited data pools, making it difficult to dynamically adapt to the needs of different testing scenarios. For example, when simulating typical business processes such as user registration, order generation, and payment transactions, traditional methods struggle to automatically generate correlated data that conforms to business logic constraints, resulting in deficiencies in the completeness and authenticity of test data. Furthermore, when faced with large-scale data generation tasks, traditional solutions often encounter performance bottlenecks, failing to efficiently generate terabytes of high-quality test data within a limited timeframe, thus severely restricting the progress of performance testing and the reliability of test results. Summary of the Invention

[0005] The purpose of this application is to provide a multi-source heterogeneous data generation system and method based on a production data model. It adopts a hierarchical design pattern and a modular testing framework, supports adjusting the data scale according to test requirements or production data volume, and adjusts the data characteristics according to the test scenario, so that the data generation has dynamic adaptability, thereby at least solving some of the problems in the background technology.

[0006] To achieve the above objectives, this application provides a system for generating multi-source heterogeneous data based on a production data model. The system includes: a base layer, which provides data generation template rendering, data storage, and stress generation functions through corresponding modules within the base layer; a service layer, which provides basic operations, intelligent data generation, relational database data construction, and file / HIVE data construction functions through corresponding modules within the service layer; and an application layer, which provides data construction scene orchestration, multi-source database information acquisition, production data model acquisition, data preview generation, and execution record functions through corresponding modules within the application layer.

[0007] Optionally, the base layer includes: a data generation template rendering engine, a database, and a stress generator; the data generation template rendering engine is used to call template rendering during process orchestration; the database is used to store documented data models, rendered template data, and execution records; and the stress generator is used to construct data according to different configured concurrency levels and concurrency counts.

[0008] Optionally, the service layer includes: a common module, an intelligent data generation module, a relational database data construction module, and a file / HIVE data construction module; the common module encapsulates basic operations, including: database environment configuration, preset functions, database data retrieval, process orchestration control, debugging control, and startup control; the intelligent data generation module is used to complete production network security access, production data sampling, data feature statistical analysis, and data table association analysis, then perform intelligent data recommendation, and finally document the model; the relational database data construction module is used to handle the complete lifecycle of multi-source heterogeneous data construction, including from metadata collection, data generation, dynamic code generation based on templates, to data construction and quality verification; the file / HIVE data construction module is used to complete the conversion from file-type data to HIVE data table data.

[0009] Optionally, the application layer includes: a data construction scenario orchestration module, a multi-source database information acquisition module, a production data model acquisition module, a data preview generation module, and an execution record module; the data construction scenario orchestration module is used to visually display the target data table to be constructed based on the user's drag-and-drop operation commands, displaying the table in a node manner, and supporting the copying, acquisition, and modification of the production data model; the multi-source database information acquisition module is used to acquire key information of the table, making it easier for the user to finally determine the production data model based on the key information; the production data model acquisition module is used to recommend data construction rules according to the production data model by analyzing production data; the data preview generation module is used to provide a preview of the generated data and support the user to adjust the preview; the execution record module is used to summarize the execution records, and the summarized execution records are used to trace back the execution status at that time.

[0010] Optionally, the intelligent data generation module includes: a production network security access module, a production data sampling module, a data feature statistical analysis module, a data table association analysis module, a model documentation module, and a data recommendation engine module; the production network security access module is used to perform production network security access; the production data sampling module is used to perform production data sampling; the data feature statistical analysis module is used to perform data feature statistical analysis; the data table association analysis module is used to perform data table association analysis; the model documentation module is used to perform model documentation; and the data recommendation engine module includes intelligent recommendation based on a preset function library and intelligent recommendation based on data table data sources.

[0011] Optionally, the relational database data construction module includes: a data table metadata acquisition module for acquiring data table metadata; a smart parameter dynamic acquisition module for dynamically acquiring smart parameters; a smart insertion template engine for generating executable data construction scripts; a unified database encapsulation module for using an abstract design concept to perform underlying unified encapsulation of different types of databases; and a data-driven execution engine for implementing unified interface adaptation for multi-source databases.

[0012] Optionally, the file / HIVE data construction module includes: a file data generation module for generating file data; a file compression module for compressing the generated batch files using a specified compression algorithm; a file upload SWIFT module for directly uploading data files in a predetermined format to the HDFS distributed file system; and a file LOAD to HIVE module for loading data using HIVE's load command.

[0013] This application also provides a method for generating multi-source heterogeneous data based on a production data model, implemented based on the aforementioned multi-source heterogeneous data generation system based on a production data model. The method includes: obtaining a data generation request through the application layer; calling the corresponding module in the service layer according to the data generation request; the corresponding module in the service layer converting the constructed data generated by the base layer into a structure corresponding to the data generation request; and returning the constructed data after structure conversion to the application layer as a response to the data generation request.

[0014] This application also provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the aforementioned multi-source heterogeneous data generation method based on a production data model by executing the instructions stored in the memory.

[0015] This application also provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned method for generating multi-source heterogeneous data based on a production data model.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for generating multi-source heterogeneous data based on a production data model.

[0017] The above technical solution has the following beneficial effects: This paper proposes an innovative solution for generating test data from multi-source heterogeneous databases. Its core lies in intelligent and efficient dynamic data construction based on a production environment data model. This solution offers the following key advantages: First, by reusing the production environment data model, it ensures the high authenticity of test data in terms of data structure, relationships, and business logic, significantly improving the accuracy and reliability of the test data. Furthermore, intelligent data recommendation enables dynamic data generation, guaranteeing the authenticity of business logic. Second, the solution designs a unified database operation abstraction layer, achieving transparent support for various heterogeneous databases, including relational, non-relational, and data warehouse databases. Standardized metadata acquisition and data insertion interface encapsulation effectively solve the data construction challenges in multi-source database environments. Third, in terms of intelligence, this solution innovatively performs intelligent data filtering based on the logical relationships between database tables, combined with a parameterized dynamic selection mechanism, ensuring that the generated test data not only meets format requirements but also maintains complete business logic consistency. Finally, the underlying technology integrates a programmable architecture for high-performance load testing tools. Through the design of an automated testing pipeline, it achieves real-time dynamic generation and efficient injection of test data, providing strong data support for performance test data construction.

[0018] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 This illustration schematically shows a hierarchical diagram of a multi-source heterogeneous data generation system based on a production data model according to an embodiment of this application; Figure 2 This illustration schematically shows a structural diagram of a multi-source heterogeneous data generation system based on a production data model according to an embodiment of this application; Figure 3 This illustration schematically shows a process diagram for generating intelligent data according to an embodiment of this application; Figure 4 This illustration schematically shows a process for constructing relational database data according to an embodiment of this application; Figure 5 This illustration schematically shows a process diagram of file / HIVE data construction according to an embodiment of this application; Figure 6 The diagram schematically illustrates the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0020] The specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the embodiments of this application.

[0021] Figure 1 The illustration shows a schematic diagram of the steps in a multi-source heterogeneous data generation method based on a production data model according to an embodiment of this application. For example... Figure 1 As shown, a multi-source heterogeneous data generation system based on a production data model includes: The base layer is used to provide data generation template rendering, data storage, and stress generation functions through corresponding modules within the base layer. The service layer is used to provide basic operations, intelligent data generation, relational database data construction, and file / HIVE data construction functions through corresponding modules within the service layer; The application layer is used to provide functions such as data construction scenario orchestration, multi-source database information acquisition, production data model acquisition, data preview generation, and execution record generation through the corresponding modules within the application layer.

[0022] Figure 2 The diagram illustrates the structure of a multi-source heterogeneous data generation system based on a production data model according to an embodiment of this application. For example... Figure 2 As shown, the foundation layer includes: a data generation template rendering engine, a database, and a stress generator. The data generation template rendering engine is used to call template rendering during workflow orchestration. The database is used to store the documented data model, rendered template data, and execution records. The stress generator is used to construct data based on different configured concurrency levels and concurrency counts. Specifically, the foundation layer, as the cornerstone of the entire architecture, mainly provides data generation template rendering, data storage, and stress generation. This system's workflow orchestration uses template rendering; the documented data model, rendered template data, and execution records all need to be stored in the database; after configuring different concurrency levels and concurrency counts, the system will construct data through the stress generator.

[0023] In some optional embodiments of this application, the service layer includes: a common module, an intelligent data generation module, a relational database data construction module, and a file / HIVE data construction module. The common module encapsulates basic operations, including: database environment configuration, preset functions, database data retrieval, process orchestration control, debugging control, and startup control. The intelligent data generation module is used to complete production network security access, production data sampling, data feature statistical analysis, and data table association analysis, then perform intelligent data recommendation, and finally document the model. The relational database data construction module is used to handle the complete lifecycle of constructing multi-source heterogeneous data, including from metadata collection, data generation, dynamic code generation based on templates, to data construction and quality verification. The file / HIVE data construction module is used to complete the conversion from file-type data to HIVE data table data. Specifically, the service layer, as the core hub of the system, is mainly responsible for supporting the functions of the application layer, and mainly includes four modules: the common module, the intelligent data generation module, the relational database data construction module, and the file / HIVE data construction module. The common modules encapsulate basic operations, such as database environment configuration (responsible for acquiring and managing environment information), preset functions and database data retrieval (used for generating simulation data), and a process orchestration control module (concatenating / parallelizing multiple data tables to ensure data correlation or independence between tables). The debugging and startup modules control concurrency and the number of concurrent processes during data construction, allowing for adjustments to the amount of data generated. The intelligent data generation module handles network security access for production, data sampling, data feature statistical analysis, data table correlation analysis, and intelligent data recommendation, ultimately documenting the model. The data construction module handles the entire lifecycle of constructing multi-source heterogeneous data, from metadata collection, data generation, dynamic code generation based on templates, to data construction and quality verification. Specifically, it converts file-based data to Hive table data.

[0024] In some optional embodiments of this application, the application layer includes: a data construction scenario orchestration module, a multi-source database information acquisition module, a production data model acquisition module, a data preview generation module, and an execution record module. The data construction scenario orchestration module is used to visually display the target data table to be constructed based on the user's drag-and-drop operation commands, displaying the table as nodes and supporting the copying, acquisition, and modification of the production data model. The multi-source database information acquisition module is used to acquire key information from the table, facilitating the user to ultimately determine the production data model based on this key information. The production data model acquisition module is used to recommend data construction rules according to the production data model by analyzing production data. The data preview generation module is used to provide a preview of the generated data and supports user adjustments to the preview. The execution record module is used to summarize the execution records, which are used to retrospectively analyze the execution status at the time of execution. Specifically, the application layer is primarily user-oriented and mainly includes five modules: the data construction scenario orchestration module, the multi-source database information acquisition module, the production data model acquisition module, the data preview generation module, and the execution record module.

[0025] The data construction scenario orchestration module visualizes the target data table to be constructed through drag-and-drop, displaying the table as nodes and supporting copying, acquiring production data models, and modifying data models. It provides a visual process orchestration tool, supporting the construction of serial and parallel processing pipelines via drag-and-drop. A directed acyclic graph model is used for process topology management, providing scalability and flexibility in process orchestration. It supports inserting data from multiple tables simultaneously while maintaining the relationships between the data tables.

[0026] The multi-source database information acquisition module will obtain key information such as table fields, comments, types, and primary keys, making it easier for users to determine the data model based on the relevant information.

[0027] The production data model acquisition module analyzes production data and recommends data construction rules based on the production data model to improve the accuracy of the data model.

[0028] To ensure consistency between the generated data and the expected data, the data preview module will generate several data entries, allowing for easy manual evaluation to determine if the model meets expectations and supporting adjustments. It also supports scenario debugging, displaying the value of each variable in real time, facilitating quick verification of whether the inserted field values ​​meet expectations, and enabling data generation preview.

[0029] The execution record module summarizes the execution records, facilitating subsequent backtracking of the execution status. After the data is constructed, a corresponding data generation record is generated. This record visually displays the number of records before insertion, the number of records after insertion, and the number of newly added records, along with log viewing functions, making it easy to determine whether the table data volume meets expectations and whether there are any abnormal errors during the process, such as data generation anomalies caused by insufficient database disk space or insufficient table space.

[0030] In some embodiments of this application, the intelligent data generation module includes: a production network security access module, a production data sampling module, a data feature statistical analysis module, a data table correlation analysis module, a model documentation module, and a data recommendation engine module. The functions of each of the above models are described below.

[0031] The production network security access module is used to perform production network security access. For security reasons, it queries backup database information from the production environment configuration center based on system coding. Strict identity authentication and authorization are implemented for the query interface. An encrypted channel is used to transmit backup database connection information, sensitive information is encrypted, and recording the complete connection string in the log is prohibited. It automatically obtains the structure definition of the data tables and extracts key metadata such as field types, constraints, and primary key constraints.

[0032] The production data sampling module is used to perform production data sampling. After connecting to the backup database, data sampling is performed on multiple selected target tables. To ensure the validity and randomness of the sampled data, and to maintain the foreign key relationships between multiple tables, this patent adopts a time-window-based stratified sampling strategy and cascaded sampling based on foreign keys. The system configures a sampling upper limit threshold N, and performs stratified sampling based on the time window for the main table with a time window. The stratified sampling is performed by quarter, and N = the number of samples in the current quarter + the number of samples in the previous quarter + the number of samples in the previous two quarters + the number of samples in the previous three quarters. The number of samples in each quarter is 1 / 4 of the threshold N, thereby ensuring that data from all four seasons is sampled and reducing the impact of seasonal factors. For related tables, sampling is performed through foreign key join queries; if no join relationship exists, sampling is then performed according to the time window.

[0033] The data feature statistical analysis module is used to perform data feature statistical analysis. First, it performs statistical analysis on the distribution characteristics of field values ​​in the production sample data. The underlying layer uses Pandas technology and the Z-Score method to detect outliers in numerical fields, discarding outliers and fitting the data distribution to determine the data distribution characteristics, deriving different intervals and probabilities. Next, it analyzes character field values, using regular expression matching to identify data with specific formats such as ID card numbers, names, mobile phone numbers, address information, bank card numbers, and customer numbers. For enumerated fields, it calculates the frequency distribution of enumerated values. Finally, it identifies fields such as time series IDs and auto-incrementing primary keys using the difference method.

[0034] The data table association analysis module is used to perform data table association analysis. After completing data feature analysis, the data table association analysis stage will systematically construct the association network between data entities through multi-dimensional technical means. Explicit foreign key constraints are automatically extracted based on database metadata, targeting the same data types (such as INT). INT, VARCHAR For VARCHAR fields without explicit foreign keys, potential associations are deduced by combining field name similarity and value range overlap (comparison of deduplicated value sets); a weighted entity relationship graph is established by quantifying the association strength (weight value = field name similarity * weight 1 + value range overlap * weight 2).

[0035] The model documentation module is used to perform model documentation. It outputs a complete relational model through a structured language, which includes both explicit relationships between tables and implicit relationships inferred from data features, providing a strict reference system for subsequent data construction.

[0036] The data recommendation engine module includes intelligent recommendation based on a preset function library and intelligent recommendation based on data table data sources. It mainly includes two core modules: 1. Intelligent Recommendation Preset Function Library: When there is no obvious relationship between table fields and external objects, the preset function library is used to generate data by intelligently matching corresponding functions. The preset function library mainly includes address functions, ID card number functions, name functions, mobile phone number functions, customer number functions, bank card number functions, enumeration value functions, account number functions, timestamp functions, random number functions, unique number functions, auto-incrementing sequence functions, Date type functions, random character functions, etc. After deriving the production data model based on the distribution of key production data, the corresponding functions are intelligently recommended, and request parameters are recommended according to the production data model, thereby realizing intelligent data recommendation from analysis. For example, if the field regular expression matches to an ID card number, the "ID Card Number Generation" function can be called to generate data that meets the requirements. Generating this part of the data through functions protects data privacy, preventing production environment data from leaking to the test environment, and also ensures the validity of data generation. 2. Intelligent Recommendation Source from Data Table Data Source: Through the automated table relationship analysis proposed in this patent, a foreign key dependency graph between entity tables is established. A real-data referencing strategy is adopted to ensure that generated data strictly adheres to the inherent constraints between business entities, achieving cross-table logical consistency verification. Taking user table A, order table B, and payment record table C as an example: the system automatically analyzes and identifies the primary key constraint relationship between the user ID field in tables B and C and table A; when generating data, intelligent recommendation selects valid user IDs from the existing data in table A (e.g., recommended acquisition method: SELECT cust_no FROM A). When both intelligent function recommendation and association analysis data table fields exist, the priority of association analysis recommendation content is higher than that of function recommendation. This implementation method effectively solves the referential integrity problem of cross-table data generation by directly extracting associated data from existing data tables in the test environment as the model input source. By using a preset function library, fields without association relationships are effectively simulated, effectively improving the consistency between test environment data and production data models.

[0037] In some embodiments of this application, the relational database data construction module includes the following modules.

[0038] The data table metadata acquisition module is used to obtain data table metadata. It establishes a standardized database connection channel and uses a JDBC / ODBC multi-protocol adapter to achieve cross-platform data access. Through automatic metadata parsing technology, it dynamically obtains the field structure information of the target data table, automatically discovers the relational database schema, intelligently identifies field attributes, and visually displays the field information of the structured data.

[0039] The intelligent parameter dynamic acquisition module is used to dynamically acquire intelligent parameters. It employs the innovative methods mentioned above for generating production environment data models and using an intelligent data recommendation engine to dynamically generate data that conforms to business logic and the production data model. Based on the production model's recommended test data generation scheme, for parameters that need to be dynamically acquired, each inserted data entry is either regenerated using a function or extracted from the database, ensuring that the parameters are dynamically generated and logically consistent.

[0040] The intelligent insertion template engine generates executable data construction scripts. Based on the correspondence between fields and field values ​​in different tables, it intelligently generates syntactically correct INSERT statements and, in conjunction with the underlying framework's syntax rules, produces executable data construction scripts. This template engine automatically handles constants and parameterized variables, calls a unified API to insert SQL statements, and dynamically determines the concurrency level and number of concurrent connections at startup based on user input, ensuring scalability across large datasets.

[0041] The unified database encapsulation module employs an abstract design philosophy to provide a unified underlying encapsulation for different types of databases. Its core innovation lies in using this abstract design philosophy to provide a unified underlying encapsulation for relational databases, Impala, and Doris systems, shielding the syntactic differences between different databases through a syntax adaptation layer. This architecture boasts excellent scalability; when adding new data sources, only the corresponding database driver (such as JDBC) needs to be configured for automatic adaptation, significantly reducing the workload of system modifications.

[0042] A data-driven execution engine is used to achieve unified interface adaptation for multiple database sources. It integrates different database JDBC and ODBC driver packages at the underlying level to achieve unified interface adaptation for various databases, supporting different data sources such as MySQL, PostgreSQL, Oracle, Doris, and Impala. Applications can easily introduce new databases without modifying business logic, ensuring consistent database access. It provides unified methods for commonly used functions, such as methods for retrieving table metadata and unified methods for inserting data into the database.

[0043] In some embodiments of this application, the file / HIVE data construction module includes: a file data generation module, used to generate file data; specifically used to generate delimiters, file headers, data record values, and filenames.

[0044] Separator: The header separator and the content separator may be inconsistent, so it is necessary to distinguish between the header separator and the file content separator. The setting of dual separators ensures the flexibility and compatibility of the format.

[0045] File Header: The file header is customizable, supporting both header-free and header-prefixed file configurations. When a file has no header, data is directly generated using content delimiters. If a file has a header, the first line defines the header, followed by the data definition. Considering that some file data follows a convention of N records per file, and when the number of records exceeds N, it is split into multiple files, with only the first file having a header and subsequent files not, this solution effectively addresses the generation requirements for such file data.

[0046] Data record values: Some fields can be set to fixed values, while others need to be random numbers, current time, name, mobile phone number, or specific fields of a table in the database. For these types of data requirements, this solution adopts a parameterized approach consistent with relational databases, supporting three methods: function generation, data retrieval from the database according to user-defined logic, or datasets provided by the user, so that the generated file content conforms to business logic.

[0047] Filename generation: Filenames are divided into three parts: filename prefix, filename concatenation, and filename suffix. Both the filename prefix and filename concatenation support fixed values ​​and parameterized methods. Considering the reusability of file generation scenarios, if the filename is fixed, multiple executions will overwrite files due to identical filenames. Therefore, concatenating a timestamp with the filename allows for easy differentiation of file generation time, and prevents file overwriting during multiple script executions. The filename suffix determines the file format and structure. The system abstracts commonly used file formats, such as txt and csv, and supports the generation of files in these formats.

[0048] The file compression module is used to compress batches of generated files using a specified compression algorithm. After generating a large number of files, compression techniques are often employed to optimize storage and transmission performance, using a specified compression algorithm. In addition to common preview and download functions, the module often provides features such as uploading to SFTP, uploading to SWIFT, and loading to Hive to store the files on specific storage devices.

[0049] For data construction scenarios in data warehouses (Hive), traditional JDBC methods suffer from significant performance bottlenecks. To address this, this invention innovatively proposes a batch processing scheme based on a standardized file format. Specifically, a SWIFT file upload module directly uploads data files in a predefined format to the HDFS distributed file system; a HIVE file load module loads data using HIVE's load commands. This method effectively avoids the line-by-line processing overhead of the JDBC interface, achieving an order-of-magnitude improvement in data throughput efficiency.

[0050] Based on the same inventive concept, this application also provides a method for generating multi-source heterogeneous data based on a production data model. This method is implemented based on the aforementioned multi-source heterogeneous data generation system based on a production data model. The method includes: obtaining a data generation request through the application layer; calling the corresponding module in the service layer according to the data generation request; the corresponding module in the service layer converting the constructed data generated by the base layer into a structure corresponding to the data generation request; and returning the structured constructed data to the application layer as a response to the data generation request. Through the above method, a method for using the multi-source heterogeneous data generation system of the production data model is provided. By operating and applying the multi-source heterogeneous data generation system of the production data model, the generation of multi-source heterogeneous data is realized.

[0051] To better understand the multi-source heterogeneous data generation system based on the production data model in this application, some modules or processes in the system are described through the following implementation methods.

[0052] Figure 3 A schematic diagram illustrating the process of intelligent data generation according to an embodiment of this application is shown. Figure 3 As shown, the main steps of the intelligent data generation module are as follows: 1. Detect outliers in the sampled data, identifying and removing empty or null values; 2. Determine the field type. If it is numeric, perform distribution fitting to obtain different intervals and probabilities; 3. For character types, perform regular expression matching to match data in specific formats; 4. For enumerated types, perform frequency analysis; 5. Recommend corresponding functions based on feature analysis results; 6. Analyze the data in related tables, extracting explicit foreign key constraints for tables with them; 7. Traverse tables with the same data type and calculate field name similarity and value range overlap; 8. Calculate the comprehensive weight to quantify the correlation strength between field values; 9. Based on the comprehensive weight, if the weight exceeds a certain threshold, derive data generation rules; 10. Recommend values ​​for related table fields using SQL statements; 11. Output a complete relational model using a structured language to provide intelligent recommendation results.

[0053] Figure 4 The illustration shows a schematic diagram of the relational database data construction process according to an embodiment of this application. Figure 4 This illustrates the relational database data construction process, corresponding to the aforementioned relational database data construction module. This process achieves high reusability of data construction scenarios, shields users from differences in underlying database operations, eliminates the need to write data construction scripts, and can generate multi-source database construction scenario templates through process orchestration. Based on a load testing framework, it efficiently generates data, specifically including: 1. Add and copy data construction scenarios, add data table nodes, and select the target table for data construction; 2. Obtain data table metadata, displaying metadata information such as fields, comments, and types; 3. Default value of a field is obtained from a data record in the database; 4. The intelligent data recommendation engine provides recommended values ​​for some fields based on feature statistical analysis or data table relationships; 5. Intelligently generate data construction templates, containing a directed acyclic graph of relationships between multiple data construction tables; 6. The execution engine resolves dependencies, generates an executable load testing script, and waits for the plan to be triggered for execution; 7. The execution plan calls the unified insertion interface API to operate on different underlying data sources; 8. Based on the stress generator, it supports data construction with different concurrency levels and concurrency counts; 9. Write to the target data system to complete the unified data construction of multiple source databases.

[0054] This implementation supports flexible definition and execution of processes ranging from single-table insertions to complex cross-database workflows, ensuring the correlation of data between multiple tables. For example, a data construction scenario could include the following process: extracting customer ID and national ID number from a MySQL table; using the customer ID and national ID number as parameters to construct an order table in Hive; and using the national ID number to construct a logistics table in PostgreSQL.

[0055] Figure 5 The schematic diagram illustrates a process diagram for constructing file / HIVE data according to an embodiment of this application. Figure 5 This illustrates the file generation and HIVE data table import process, corresponding to the aforementioned file / HIVE data construction module. The process includes: 1. For structured text files, such as TXT and CSV formats, configure header delimiters. Common header delimiters include commas and semicolons. Support for selecting common delimiters or inputting specific delimiters enhances delimiter support and ensures flexible file generation. 2. Configure content delimiters, similar to configuring header delimiters, supporting customization. 3. Configure headers, supporting both header-less and header-without header configurations. Without headers, the first row contains data; with headers, the first row declares the header, and the second row contains data. 4. Define data, supporting four methods: fixed values, function library retrieval, data table extraction, and user-defined datasets. 5. Configure filenames by prefixing, concatenating filenames, and adding filename suffixes to form complete filenames. 6. To generate HIVE table data, select the HIVE cluster address and the target table. 7. Transfer the generated file to the HIVE cluster. 8. Use a system with HDFS. Users with put privileges upload files to the HDFS distributed file system; 9. Load data using the HIVE LOAD command to complete the construction of HIVE table data.

[0056] The specific limitations of each functional module in the aforementioned multi-source heterogeneous data generation system based on the production data model can be found in the limitations of the multi-source heterogeneous data generation method based on the production data model described above, and will not be repeated here. Each module in the aforementioned system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in an electronic device, or stored in the memory of an electronic device in software form, so that the processor can call and execute the corresponding operations of each module. It achieves the same beneficial effects as the multi-source heterogeneous data generation system based on the production data model.

[0057] In some embodiments of this application, an electronic device is also provided, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which executes the aforementioned multi-source heterogeneous data generation method based on a production data model. Its internal structure diagram can be as follows: Figure 6 As shown. Figure 6 This schematic diagram illustrates the internal structure of an electronic device according to an embodiment of this application. The electronic device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a multi-source heterogeneous data generation method based on a production data model.

[0058] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0059] In one embodiment provided in this application, a machine-readable storage medium is provided, on which instructions are stored, which, when executed by a processor, cause the processor to be configured to perform the aforementioned multi-source heterogeneous data generation method based on a production data model.

[0060] In one embodiment provided in this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned method for generating multi-source heterogeneous data based on a production data model.

[0061] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0066] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0067] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0069] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multi-source heterogeneous data generation system based on a production data model, characterized in that, The system includes: The base layer is used to provide data generation template rendering, data storage, and stress generation functions through corresponding modules within the base layer. The service layer is used to provide basic operations, intelligent data generation, relational database data construction, and file / HIVE data construction functions through corresponding modules within the service layer; The application layer is used to provide functions such as data construction scenario orchestration, multi-source database information acquisition, production data model acquisition, data preview generation, and execution record generation through the corresponding modules within the application layer.

2. The system according to claim 1, characterized in that, The base layer includes: a data generation template rendering engine, a database, and a stress generator; The data generation template rendering engine is used to call template rendering during process orchestration; The database is used to store the document-generated data model, rendered template data, and execution records; The pressure generator is used to construct data based on different configured concurrency levels and concurrency counts.

3. The system according to claim 1, characterized in that, The service layer includes: a public module, an intelligent data generation module, a relational database data construction module, and a file / HIVE data construction module; The common module encapsulates basic operations, which include: database environment configuration, preset functions, database data retrieval, process orchestration control, debugging control, and startup control. The intelligent data generation module is used to complete network security access for production networks, production data sampling, data feature statistical analysis and data table correlation analysis, then perform intelligent data recommendation, and finally document the model. The relational database data construction module is used to handle the complete lifecycle of constructing multi-source heterogeneous data, including from metadata collection, data generation, dynamic code generation based on templates, to data construction and quality verification. The file / HIVE data construction module is used to convert file-type data into HIVE data table data.

4. The system according to claim 1, characterized in that, The application layer includes: a data construction scenario orchestration module, a multi-source database information acquisition module, a production data model acquisition module, a data preview generation module, and an execution record module; The data construction scenario orchestration module is used to visualize the target data table to be constructed based on the user's drag-and-drop operation instructions, display the table in the form of nodes, and support the copying, acquisition and modification of production data models; The multi-source database information acquisition module is used to acquire key information from the table, so that users can ultimately determine the production data model based on the key information. The production data model acquisition module is used to analyze production data and recommend data construction rules according to the production data model. The generated data preview module is used to provide a preview of the generated data and supports users in adjusting the preview; The execution record module is used to summarize the execution records, and the summarized execution records are used to trace back the execution status at that time.

5. The system according to claim 3, characterized in that, The intelligent data generation module includes: a production network security access module, a production data sampling module, a data feature statistical analysis module, a data table correlation analysis module, a model documentation module, and a data recommendation engine module; The production network security access module is used to perform production network security access. The production data sampling module is used to perform production data sampling. The data feature statistical analysis module is used to perform data feature statistical analysis; The data table association analysis module is used to perform data table association analysis; The model documentation module is used to perform model documentation; The data recommendation engine module includes intelligent recommendations based on a preset function library and intelligent recommendations based on data sources from data tables.

6. The system according to claim 3, characterized in that, The relational database data construction module includes: The data table metadata retrieval module is used to retrieve data table metadata. The intelligent parameter dynamic acquisition module is used to dynamically acquire intelligent parameters; A smart template engine for generating executable data construction scripts; The database unified encapsulation module is used to provide a unified underlying encapsulation for different types of databases using an abstract design concept; A data-driven execution engine is used to achieve unified interface adaptation for multi-source databases.

7. The system according to claim 3, characterized in that, The file / HIVE data construction module includes: The file data generation module is used to generate file data. The file compression module is used to compress the generated batch files using a specified compression algorithm; The file upload SWIFT module is used to directly upload data files in a predefined format to the HDFS distributed file system. The file is loaded into the HIVE module, which is used to load data via HIVE's load command.

8. A method for generating multi-source heterogeneous data based on a production data model, characterized in that, Based on the implementation of the multi-source heterogeneous data generation system based on a production data model according to any one of claims 1 to 7, the method includes: A request is generated by obtaining data from the application layer; A request is generated based on the data, and the corresponding module in the service layer is invoked. The corresponding modules in the service layer convert the constructed data generated by the base layer into the structure corresponding to the data generation request. The constructed data after structural transformation is returned to the application layer as a response to the data generation request.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When executed by a processor, the computer program / instructions implement the multi-source heterogeneous data generation system based on a production data model as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the multi-source heterogeneous data generation system based on a production data model as described in any one of claims 1 to 7.

Images