Data processing test method and system based on simulation data generation

By defining configuration files and Json splicing components, the efficiency and flexibility of database Kafka consumer testing is achieved, solving the problem of insufficient time and flexibility of test chain building, providing automated, accurate and repeatable test results, and supporting a variety of tests Data synchronization scenarios and database integration.

CN118820119BActive Publication Date: 2024-12-03JIANGSU HUAKU DATA TECH CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411305079.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-19
Publication Date
2024-12-03
Estimated Expiration
2044-09-19

AI Technical Summary

Technical Problem

In the Kafka consumer test of the database, it takes a lot of time and labor costs to build a complete test chain, and the data flexibility is not enough, making it difficult to deal with various data types and multi-store and multi-table concurrent testing scenarios.

Method used

By defining configuration files, fine-grained target table structure, primary key column and library table schema, use Json stitching component to generate test Json messages, and use Json sending component to perform data synchronization simulation processing, simulate data generation, processing and synchronization operations, including insertion, Update and delete to realize data synchronization and test results analysis.

Benefits of technology

Improves testing efficiency and flexibility, reduces the need to write test scripts manually, reduces the risk of errors, ensures the consistency and repeatability of test results, and provides a tool close to the real data processing environment, suitable for a variety of data Synchronize scenarios and support the integration of different databases and data processing systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118820119B_ABST
    Figure CN118820119B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data processing technology, and discloses a data processing test method and system based on simulated data generation, the method comprising: defining a configuration file, defining a target table structure based on the configuration file; configuring a primary key column and a library table mode, defining a POS number and its starting value; generating a test Json message according to the configuration file through a Json splicing component; performing data synchronization simulation processing through a Json sending component to obtain simulated processing data; performing data synchronization on data to be synchronized through simulated processing data to obtain target data; performing synchronization test result analysis on the target data to obtain a target analysis result, and transmitting the target analysis result to a preset data processing terminal. The present invention adjusts test parameters according to specific test requirements, and this flexibility means that the same set of test frameworks can be applied to a variety of different data synchronization scenarios, thereby greatly enhancing the applicability and efficiency of the test work.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing test method and system based on simulation data generation. Background Art

[0002] Currently, there is a data flow service that uses Kafka to achieve data synchronization. Golden Gate extracts committed transaction information from the source database log, generates a Trail file, and then converts the information in the Trail file into the target format (Json, XML, etc.) and sends it to Kafka as a message. After that, the target end receives the message in Kafka as a Kafka consumer and parses it into the required information.

[0003] Previously, we faced two main problems in testing Kafka consumers for databases: First, a complete test chain requires the construction of a complete environment, from producer to Kafka to consumer, which takes a lot of time and manpower, and is also relatively complicated to operate. Second, the data is not flexible enough. When dealing with various data types, multi-database and multi-table concurrency and other test scenarios, a large number of operations are required, which is very cumbersome and difficult to control. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data processing test method and system based on simulation data generation, which are used to improve the efficiency of data testing based on simulation data generation.

[0005] The present invention provides a data processing test method based on simulation data generation, comprising: defining a configuration file, and defining a target table structure based on the configuration file; configuring a primary key column and a library table mode, and defining a POS number and a starting value of the POS number; generating a test Json message according to the configuration file through a preset Json splicing component; performing data synchronization simulation processing through a preset Json sending component to obtain simulation processing data; performing data synchronization on preset data to be synchronized through the simulation processing data to obtain target data; performing synchronization test result analysis on the target data to obtain a target analysis result, and transmitting the target analysis result to a preset data processing terminal.

[0006] In the present invention, the step of defining a configuration file and, at the same time, defining a target table structure based on the configuration file, includes: performing parameter matching on a database to be tested to obtain corresponding database parameters, performing parameter matching on a target library table corresponding to the database to be tested to obtain library table parameters; defining a configuration file based on the library table parameters and the database parameters, and, at the same time, defining the target table structure based on the configuration file.

[0007] In the present invention, the database parameters include: data style, number of data items and data sending method, and the library table parameters include: library table name, library table structure data, wherein the library table structure data includes: custom column length, custom column name and custom column data type.

[0008] In the present invention, the configuration file is defined based on the library table parameters and the database parameters, and at the same time, the step of defining the target table structure based on the configuration file includes: based on the library table parameters and the database parameters, data is defined for the Kafka end to obtain the first IP address, topic name, and number of partitions corresponding to the Kafka end; based on the library table parameters and the database parameters, data is defined for the consumer end to obtain the second IP address and number of library tables of the consumer end; the configuration file is defined according to the database parameters, the library table parameters, the first IP address, the topic name, the number of partitions, the second IP address, and the number of library tables; and the target table structure is defined through the configuration file.

[0009] In the present invention, the step of generating a test Json message according to the configuration file through a preset Json splicing component includes: extracting the file type of the configuration file to obtain file type data; segmenting the configuration file through the file type data to obtain multiple segmentation fields; constructing a Json message for multiple segmentation fields through the Json splicing component to obtain multiple sub-Json messages; and generating a test Json message based on the multiple sub-Json messages.

[0010] In the present invention, the steps of performing data synchronization simulation processing through a preset Json sending component to obtain simulated processing data include: constructing a simulation implementation logic, wherein the simulation implementation logic includes: insert operation, update operation and delete operation; based on the simulation implementation logic, performing data synchronization simulation processing through the Json sending component to obtain simulated processing data.

[0011] In the present invention, the step of synchronizing the preset data to be synchronized through the simulated processing data to obtain the target data includes: matching the synchronization type of the preset data to be synchronized through the simulated processing data to obtain the target synchronization type; and synchronizing the data to be synchronized through the target synchronization type to obtain the target data.

[0012] The present invention also provides a data processing test system based on simulation data generation, comprising:

[0013] A definition module, used to define a configuration file and define a target table structure based on the configuration file;

[0014] A configuration module, used to configure the primary key column and the library table mode, and define the POS number and the starting value of the POS number;

[0015] A generation module, used to generate a test Json message according to the configuration file through a preset Json splicing component;

[0016] The processing module is used to perform data synchronization simulation processing through the preset Json sending component to obtain simulation processing data;

[0017] A synchronization module, used for synchronizing preset data to be synchronized through the simulated processing data to obtain target data;

[0018] The analysis module is used to analyze the target data synchronization test results, obtain target analysis results, and transmit the target analysis results to a preset data processing terminal.

[0019] In the technical solution provided by the present invention, by finely defining the configuration file, the solution provides a high degree of customizability and flexibility for the test process. The configuration file contains detailed instructions on the target table structure, data synchronization rules and data generation, which enables testers to adjust test parameters such as data types, synchronization frequency and data processing logic according to specific test requirements. This flexibility means that the same set of test frameworks can be applied to a variety of different data synchronization scenarios, thereby greatly enhancing the applicability and efficiency of the test work. Secondly, the technical feature of generating test Json messages based on the configuration file makes the data testing process more automated and accurate. By automatically converting the parameters and rules in the configuration file into specific test data and operation steps, the need for manually writing test scripts is reduced, and the risk of introducing errors due to manual operations is also reduced. This increase in the degree of automation not only speeds up the test process, but also ensures the consistency and repeatability of the test results. In addition, the technical feature of performing data synchronization simulation processing through the Json splicing component provides an effective tool for simulating a real data processing environment. This component can simulate the generation, processing and synchronization operations of data, including the insertion, update and deletion of data, so that the test environment is closer to the actual data processing scenario. This simulation, which is close to the real operating environment, can help testers identify potential problems and bottlenecks more accurately, so that they can be solved in advance during the testing phase, rather than after the product is launched. Finally, this solution also has good scalability and compatibility. Because it is based on the common Json format and standard data processing interface, the test framework can be easily integrated with different databases and data processing systems. Whether it is a traditional relational database or a modern NoSQL data storage, effective data synchronization testing can be performed through appropriate configuration files. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 The present invention is a flowchart of a data processing test method based on simulation data generation in an embodiment of the present invention.

[0022] Figure 2 The flowchart of the configuration file is defined based on the library table parameters and the database parameters in the embodiment of the present invention.

[0023] Figure 3 Schematic diagram of a data processing test system based on simulation data generation in an embodiment of the present invention.

[0024] Reference numerals:

[0025] 301. Definition module; 302. Configuration module; 303. Generation module; 304. Processing module; 305. Synchronization module; 306. Analysis module. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In the description of the present invention, the terms “first”, “second” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0028] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , Figure 1 is a flow chart of a data processing test method based on simulation data generation according to an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0030] S101, define a configuration file, and at the same time, define a target table structure based on the configuration file;

[0031] S102, configure the primary key column and the library table mode, and define the POS number and the starting value of the POS number;

[0032] S103, generating a test Json message according to the configuration file through a preset Json splicing component;

[0033] S104, performing data synchronization simulation processing through a preset Json sending component to obtain simulation processing data;

[0034] S105, synchronizing the preset data to be synchronized by simulating the processed data to obtain target data;

[0035] S106, performing synchronous test result analysis on the target data to obtain a target analysis result, and transmitting the target analysis result to a preset data processing terminal.

[0036] It should be noted that, first of all, the whole process starts with a carefully designed configuration file, which not only defines the structure of the target table, including table name, column name, data type and constraints, but also configures the primary key column, library table mode and the starting value of the POS number, laying the foundation for data generation, processing and synchronization. For example, assuming there is a project aimed at testing the efficiency of database synchronization on an e-commerce platform, the configuration file will define the structure of the Products table in detail, including fields such as product ID (primary key), name, price, etc., and specify the starting value of the POS number as 1000, ensuring that each generated test data can be uniquely identified in the simulation environment. Based on this configuration file, the next step is to use the developed script or software tool to automatically generate Json messages for testing, which simulate real business data according to the definition in the configuration file. In the example, the generated Json messages may simulate operations such as adding new products, updating the price of existing products, or deleting products. These Json messages are then used to generate a Json splicing component, which is responsible for not only processing data in Json format, but also simulating the synchronization process of data, such as simulating data synchronization by performing insert, update or delete operations. Entering the data synchronization simulation processing stage, the Json splicing component processes the input Json message according to the pre-configured rules and logic to generate simulated processed data. In the example of the e-commerce platform, if a Json message represents an update to the price of a certain product, the splicing component will simulate this update operation and generate a simulated processed data reflecting the update. These simulated processed data are generated according to the real business logic and are intended to simulate the synchronization process of data in the actual production environment. Next, the simulated processed data is used to perform actual synchronization operations on the preset data to be synchronized to generate the final target data. In this process, the preset data represents the state of the e-commerce platform database before synchronization, and by applying simulated processed data, such as adding new products to the product list or updating the price of existing products, the target data reflecting all these changes can be obtained, that is, the expected state of the database after synchronization. Finally, the analysis of the synchronization test results of the target data is the key to verifying whether the entire data synchronization simulation process is successful. By comparing the data before and after synchronization and evaluating the accuracy, completeness and efficiency of data synchronization, we can deeply understand the problems and challenges that may exist in the synchronization process. In the case of the e-commerce platform, this analysis phase helped the team discover latency issues in the data synchronization process and successfully reduced the time required for data synchronization by further optimizing the synchronization algorithm. By transmitting these target analysis results to the pre-set data processing terminal, the team can better share and discuss test results and jointly find solutions.

[0037] By executing the above steps and carefully defining the configuration file, this solution provides a high degree of customizability and flexibility for the testing process. The configuration file contains detailed instructions on the target table structure, data synchronization rules, and data generation, which enables testers to adjust test parameters such as data types, synchronization frequency, and data processing logic for specific testing needs. This flexibility means that the same set of test frameworks can be applied to a variety of different data synchronization scenarios, greatly enhancing the applicability and efficiency of testing work. Secondly, the technical feature of generating test Json messages based on configuration files makes the data testing process more automated and accurate. By automatically converting the parameters and rules in the configuration file into specific test data and operation steps, the need for manual writing of test scripts is reduced, and the risk of errors introduced by manual operations is also reduced. This increased degree of automation not only speeds up the testing process, but also ensures the consistency and repeatability of the test results. In addition, the technical feature of data synchronization simulation processing through the Json splicing component provides an effective tool for simulating a real data processing environment. This component can simulate data generation, processing, and synchronization operations, including data insertion, update, and deletion, making the test environment closer to the actual data processing scenario. This simulation, which is close to the real operating environment, can help testers identify potential problems and bottlenecks more accurately, so that they can be solved in advance during the testing phase, rather than after the product is launched. Finally, this solution also has good scalability and compatibility. Because it is based on the common Json format and standard data processing interface, the test framework can be easily integrated with different databases and data processing systems. Whether it is a traditional relational database or a modern NoSQL data storage, effective data synchronization testing can be performed through appropriate configuration files.

[0038] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0039] (1) Match the parameters of the database to be tested to obtain the corresponding database parameters, and match the parameters of the target table corresponding to the database to be tested to obtain the table parameters;

[0040] (2) Define the configuration file based on the library table parameters and database parameters, and define the target table structure based on the configuration file.

[0041] In a specific embodiment, database parameters include: data style, number of data items and data sending method, and library table parameters include: library table name and library table structure data, wherein the library table structure data includes: custom column length, custom column name and custom column data type.

[0042] In a specific embodiment, if Figure 2As shown, the process of executing the step of defining the configuration file based on the library table parameters and the database parameters may specifically include the following steps:

[0043] S201. Based on the library table parameters and database parameters, define the data of the Kafka end to obtain the first IP address, topic name, and number of partitions corresponding to the Kafka end;

[0044] S202: Based on the library table parameters and the database parameters, define the data of the consumer end to obtain the second IP address and the number of library tables of the consumer end;

[0045] S203, define a configuration file according to database parameters, library table parameters, first IP address, topic name, number of partitions, second IP address, and number of library tables;

[0046] S204. Define the target table structure through a configuration file.

[0047] First, defining the data on the Kafka side based on the database parameters and database parameters is the basis for ensuring that data can flow correctly. This includes determining the first IP address, topic name, and number of partitions corresponding to the Kafka side. In the e-commerce system example, it is assumed that the source database contains product information, such as product ID, name, price, etc. This information needs to be synchronized to the target database in real time to ensure the consistency of system data. By analyzing the structure and synchronization requirements of the source database, the Kafka topic name is determined to be "product_sync" to reflect its purpose, and the number of partitions is determined to be 3 based on the data volume and synchronization frequency to improve the efficiency and reliability of data processing. In addition, the IP address of the server where Kafka is deployed is selected as the first IP address to ensure that the data source can be seamlessly connected to Kafka. Next, it is also crucial to define the data for the consumer side. This step involves determining the second IP address and number of tables on the consumer side. In the e-commerce system, the consumer side is responsible for receiving product information from Kafka and updating it synchronously to the target database. Therefore, based on the configuration of the target database and the expected number of synchronized tables, the second IP address of the consumer side is defined, and the number of tables to be synchronized is determined. For example, if the target database is designed with dedicated product information tables and inventory tables, the number of tables must be at least 2. Furthermore, all of these key information—database parameters, table parameters, first IP address, topic name, number of partitions, second IP address, and number of tables—need to be defined and integrated in a configuration file. This configuration file is not only a blueprint for data synchronization, but also the key to ensuring that the data synchronization process is controllable and manageable. In the example of an e-commerce system, the configuration file describes in detail the entire data flow process from the source database to the target database, including how to transfer through Kafka, the precise details of data synchronization, etc., to ensure the accurate construction of the test environment and the effectiveness of data synchronization. The last step is to define the target table structure through the configuration file. In an e-commerce system, this means accurately creating or updating the product information table and inventory table in the target database according to the information in the configuration file. This step ensures that the target database can receive and correctly process the data synchronized from the source database through Kafka. By executing the definition in the configuration file, the table structure is automatically built or updated in the target database, making the entire data synchronization test process efficient and reliable.

[0048] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0049] (1) Extract the file type of the configuration file to obtain the file type data;

[0050] (2) Segment the configuration file according to the file type data to obtain multiple segmentation fields;

[0051] (3) Use the Json splicing component to construct Json messages for multiple segmented fields to obtain multiple sub-Json messages;

[0052] (4) Generate a test Json message based on multiple sub-Json messages, and generate a Json splicing component through multiple sub-Json messages.

[0053] Specifically, the file type is extracted from the configuration file. This step is key to understanding the content and format of the configuration file because it determines the way subsequent operations are performed. In the example of the electronic medical record system, it is assumed that the configuration file is provided in YAML format. The YAML format is selected for its readability and simplicity, making the setting of test parameters intuitive. By analyzing the file type, the organizational structure of the data in the configuration file and how to parse the data to adapt to the subsequent processing flow are determined. Next, based on the file type data, the configuration file is segmented and broken into multiple segmentation fields. This step is the process of breaking down the large content of the configuration file into manageable parts, each of which represents a specific set of test parameters or instructions. In the scenario of the electronic medical record system, this may include segmenting fields representing different data sources, target formats, synchronization frequencies, etc. Such segmentation not only makes the configuration file easier to manage, but also allows specific data processing logic to be customized for different test scenarios. Next, these segmentation fields are used to build Json messages. Each segmentation field is converted into a sub-Json message, which details the data conversion and synchronization logic from the source to the target. For example, one sub-Json message may specify how to convert patient information in an electronic medical record from XML format to JSON format, while another describes the data mapping rules when synchronizing patient records to another database. In this way, each sub-Json message becomes a blueprint for a specific operation in the test process. Finally, a test Json message is generated based on multiple sub-Json messages, and a Json splicing component is generated through these sub-Json messages. This component is responsible for integrating all sub-Json messages into a complete test sequence to simulate the data processing and synchronization process of the entire electronic medical record system. In this process, the test Json message is used as input to instruct the Json splicing component on how to process and synchronize data according to the parameters and rules defined in the configuration file. For example, the Json splicing component extracts data from the source database, converts it to the target format, and synchronizes it to the target database according to the instructions in the sub-Json message. In this way, not only can complex data conversion and synchronization operations be simulated, but also each step can be ensured to accurately follow the definition in the configuration file, thereby improving the accuracy and reliability of the test.

[0054] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0055] (1) Construct simulation implementation logic, where the simulation implementation logic includes: insert operation, update operation and delete operation;

[0056] (2) Based on the simulation implementation logic, the data synchronization simulation processing is performed through the Json sending component to obtain the simulated processing data.

[0057] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0058] (1) Match the synchronization type of the preset data to be synchronized by simulating the processed data to obtain the target synchronization type;

[0059] (2) Through the target synchronization type, synchronize the data to be synchronized to obtain the target data.

[0060] It should be noted that the configuration file: specifies the style, number of data, and sending method; the name and structure of the target library table are also specified in the configuration file, and the column length and column name are customized to support most common data types; specifies the IP address, topic name, and number of partitions on the Kafka side; and specifies the IP and number of libraries and tables on the consumer side.

[0061] Json splicing component: splice Json messages according to the style, table structure and other configurations in the configuration file. In order to ensure that the test data is not repeated, the following rules are adopted: the value of the numeric type (int, decimal, etc.) in each data is incremented, and the value of the string type is a splicing of fixed characters and incrementing numbers. In this way, if delete (delete) or update (update) operations are performed, it can be guaranteed that there will be no misses or hits on multiple rows. For example, this program is used to send 100 Json messages, of which 70 are inserted (insert), 20 are deleted, and 10 are updated. After inserting 70 more data, the delete operation will delete the first 20, and the update operation will update the last inserted 10. Compared with the previous data, the updated data has no change except the primary key, and the remaining values ​​are all reduced by 1, which can still ensure the uniqueness of the data.

[0062] Sending component: According to the number of entries, sending method and other configurations in the configuration file, the Json splicing component is called repeatedly, and the Kafka-python interface is called once each time a message is generated. This can simulate full synchronization of the source database without stopping, and can also simulate the generation of incremental content during the synchronization process.

[0063] Table creation and deletion script: According to the table structure, number of database tables, IP address of the consumer end, etc. in the configuration file, complete the operations of creating databases and tables, and deleting databases and tables in the target database. The database table structure corresponds to the content in Json.

[0064] Automated regression framework: You can choose to include the generated data in the automated regression framework. This framework requires users to manually write SQL statements to create tables, create Kafka consumers, check results, and clean up the environment, and add the parameters required by the database to the framework. After the preparations are completed, it takes about three minutes to execute a consumption process. At the end of each execution, the logs generated during the execution process and the results of the result correctness check will be output to facilitate the analysis of testers.

[0065] The following is a detailed description of the implementation details of various configurations:

[0066] Table structure part: define a dictionary type object to customize column names, column data types and column attributes. Testers can directly add entries to this dictionary in column order. For example:

[0067] column["c1"] = "A"

[0068] column["c1_type"] = "int"

[0069] column["c2"] = "B"

[0070] column["c2_type"] = "datetime"

[0071] …

[0072] You can continue writing according to the above style to customize column length, column name, and column properties.

[0073] Primary key part: specified in list format, one or more columns of string or numeric type can be specified. The column specified as the primary key is used as the content of the "primary_keys" field when splicing Json, and the value of the primary key column remains unchanged when updating, which is mentioned in the Json splicing component above.

[0074] Library and table mode: Two modes are added to control the number of libraries and tables: single library and single table mode and multi-library and multi-table mode. In single library and single table mode, the Json string will be directly spliced ​​according to the configured library name and table name. In multi-library and multi-table mode, Arabic numerals will be added after the configured library name and table name to distinguish different libraries and tables. The table creation script splices SQL statements according to this part of the configuration and calls the connection interface between Python and the database to create the table.

[0075] POS number part: This involves a duplicate message judgment mechanism. When judging duplicate messages on the consumer side, the largest POS number in the currently parsed Json will be recorded, and messages less than or equal to this POS number will be regarded as duplicate messages. Simulate OGG to generate the POS number in Json. Here, two ten-digit integers are concatenated into a 20-digit POS number. The number of digits in the POS number can also be specified to cope with more test scenarios. Set the starting value of the POS number in the configuration file. When sending messages to Kafka, the POS number is increased by 1 for each message. If you need to send another message to Kafka again, just increase the starting value of the POS number.

[0076] Starting value part: As mentioned in the Json splicing component above, the value of the numeric type in each data is incremented, and the value of the string type is a concatenation of fixed characters and incremented numbers, and the characters can also be specified. In addition, the initial value of the time type can also be specified in the configuration file, and these values ​​are fixed.

[0077] Multi-partition part: A multi-partition switch is set in the configuration file. For multi-partition scenarios, the embodiment of the present invention is designed as follows: the multi-partition mode must be used in conjunction with the multi-library and multi-table mode, and each partition corresponds to each library one-to-one. For example, the Json corresponding to the test1 library will only be sent to the first partition of kafka (partition 0), and so on. The reason for this is that the consumer consumes multiple partitions at the same time. If the libraries and tables contained in different partitions overlap, it may cause abnormal situations such as confusion in the order of warehousing and data errors.

[0078] Large message part: After the large message switch is turned on, different from the original one Json sent once, multiple Jsons can be accumulated and sent to one offset. The number of Jsons contained in this large message can also be specified, but if the message is too large and exceeds the setting in Kafka, an error will be reported and the sending will fail. When the large message switch and the multi-partition switch are turned on at the same time, the program will randomly send the message to a partition, and the library table information in it is also random, which can be used for some performance testing scenarios.

[0079] Message sending control part: Under different library-table mode configurations, the way to call the Json splicing component is different. By using multiple loop judgments, you can manually adjust the proportions of various insert, delete, and update messages. In the multi-library and multi-table mode, set two corresponding library-table number identifiers. After each message is sent to a table, the value of the table identifier is +1. After the messages of all tables in this library are sent, the library identifier is +1, and the table identifier is reset to 1, so as to control the message sending of multiple libraries and tables. In the multi-partition mode, the partition number is the library identifier-1, ensuring that each partition corresponds to only one library.

[0080] The embodiment of the present invention also provides a data processing test system based on simulation data generation, such as Figure 3 As shown, the data processing test system based on simulation data generation specifically includes:

[0081] A definition module 301 is used to define a configuration file and define a target table structure based on the configuration file;

[0082] Configuration module 302, used to configure primary key columns and library table modes, and define POS numbers and starting values ​​of the POS numbers;

[0083] A generating module 303 is used to generate a test Json message according to the configuration file through a preset Json splicing component;

[0084] The processing module 304 is used to perform data synchronization simulation processing through a preset Json sending component to obtain simulation processing data;

[0085] A synchronization module 305 is used to synchronize preset data to be synchronized through the simulated processing data to obtain target data;

[0086] The analysis module 306 is used to analyze the target data synchronization test result, obtain a target analysis result, and transmit the target analysis result to a preset data processing terminal.

[0087] Through the collaborative work of the above modules and the detailed definition of configuration files, this solution provides a high degree of customizability and flexibility for the testing process. The configuration file contains detailed instructions on the target table structure, data synchronization rules, and data generation, which enables testers to adjust test parameters such as data types, synchronization frequency, and data processing logic for specific test requirements. This flexibility means that the same set of test frameworks can be applied to a variety of different data synchronization scenarios, greatly enhancing the applicability and efficiency of testing work. Secondly, the technical feature of generating test Json messages based on configuration files makes the data testing process more automated and accurate. By automatically converting the parameters and rules in the configuration file into specific test data and operation steps, the need for manual writing of test scripts is reduced, and the risk of errors introduced by manual operations is also reduced. This increase in automation not only speeds up the testing process, but also ensures the consistency and repeatability of the test results. In addition, the technical feature of data synchronization simulation processing through the Json splicing component provides an effective tool for simulating a real data processing environment. This component can simulate data generation, processing, and synchronization operations, including data insertion, update, and deletion, making the test environment closer to the actual data processing scenario. This simulation, which is close to the real operating environment, can help testers identify potential problems and bottlenecks more accurately, so that they can be solved in advance during the testing phase, rather than after the product is launched. Finally, this solution also has good scalability and compatibility. Because it is based on the common Json format and standard data processing interface, the test framework can be easily integrated with different databases and data processing systems. Whether it is a traditional relational database or a modern NoSQL data storage, effective data synchronization testing can be performed through appropriate configuration files.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the embodiments, a person skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data processing test method based on simulated data generation, characterized in that: include: Define a configuration file, and at the same time, define a target table structure based on the configuration file, including: Perform parameter matching on the database to be tested to obtain corresponding database parameters, the database parameters include: data style, number of data items and data sending mode; perform parameter matching on the target library table corresponding to the database to be tested to obtain library table parameters, the library table parameters include: library table name and library table structure data, wherein the library table structure data includes: custom column length, custom column name and custom column data type; The step of defining a configuration file based on the library table parameters and the database parameters and defining the target table structure based on the configuration file includes: Based on the library table parameters and the database parameters, define the data of the Kafka end to obtain the first IP address, topic name, and number of partitions corresponding to the Kafka end; Based on the library table parameters and the database parameters, data definition is performed on the consumer end to obtain the second IP address and the number of library tables of the consumer end; Define a configuration file according to the database parameters, the library table parameters, the first IP address, the topic name, the number of partitions, the second IP address, and the number of library tables; Defining the target table structure through the configuration file; Configure the primary key column and the database table mode, and define the POS number and the starting value of the POS number; Generate a test Json message according to the configuration file through a preset Json splicing component; Perform data synchronization simulation processing through the preset Json sending component to obtain simulated processing data; The preset data to be synchronized is synchronized by using the simulated processing data to obtain target data; A synchronous test result analysis is performed on the target data to obtain a target analysis result, and the target analysis result is transmitted to a preset data processing terminal.

2. The data processing test method based on simulation data generation according to claim 1, characterized in that: The step of generating a test Json message according to the configuration file by using a preset Json splicing component includes: Extracting the file type of the configuration file to obtain file type data; Segment the configuration file according to the file type data to obtain a plurality of segmentation fields; Constructing a Json message for the plurality of segmented fields through the Json concatenation component to obtain a plurality of sub-Json messages; Generate a test Json message based on the multiple sub-Json messages.

3. The data processing test method based on simulation data generation according to claim 1, characterized in that: The step of performing data synchronization simulation processing through the preset Json sending component to obtain the simulated processing data includes: Constructing simulation implementation logic, wherein the simulation implementation logic includes: insert operation, update operation and delete operation; Based on the simulation implementation logic, data synchronization simulation processing is performed through the Json sending component to obtain simulation processing data.

4. The data processing test method based on simulation data generation according to claim 3, characterized in that: The step of synchronizing the preset data to be synchronized by the simulation processing data to obtain the target data includes: Matching the preset data to be synchronized with the synchronization type through the simulated processing data to obtain the target synchronization type; The data to be synchronized is synchronized using the target synchronization type to obtain target data.

5. A data processing test system based on simulation data generation, used to execute the data processing test method based on simulation data generation according to any one of claims 1 to 4, characterized in that: include: The definition module is used to define a configuration file and define a target table structure based on the configuration file, including: Perform parameter matching on the database to be tested to obtain corresponding database parameters, the database parameters include: data style, number of data items and data sending mode; perform parameter matching on the target library table corresponding to the database to be tested to obtain library table parameters, the library table parameters include: library table name and library table structure data, wherein the library table structure data includes: custom column length, custom column name and custom column data type; The step of defining a configuration file based on the library table parameters and the database parameters and defining the target table structure based on the configuration file includes: Based on the library table parameters and the database parameters, define the data of the Kafka end to obtain the first IP address, topic name, and number of partitions corresponding to the Kafka end; Based on the library table parameters and the database parameters, data definition is performed on the consumer end to obtain the second IP address and the number of library tables of the consumer end; Define a configuration file according to the database parameters, the library table parameters, the first IP address, the topic name, the number of partitions, the second IP address, and the number of library tables; Defining the target table structure through the configuration file; A configuration module, used to configure the primary key column and the library table mode, and define the POS number and the starting value of the POS number; A generation module, used to generate a test Json message according to the configuration file through a preset Json splicing component; The processing module is used to perform data synchronization simulation processing through the preset Json sending component to obtain simulation processing data; A synchronization module, used for synchronizing preset data to be synchronized through the simulated processing data to obtain target data; The analysis module is used to analyze the target data synchronization test results, obtain target analysis results, and transmit the target analysis results to a preset data processing terminal.

Citation Information

Patent Citations

  • Lightweight network simulation method using node agent

    CN114024986A

  • Data synchronization test method and device, electronic equipment and storage medium

    CN115017054A

  • Simulation system test method and device, electronic equipment and storage medium

    CN115617661A

  • Processing method and device of APP watch function simulator and electronic equipment

    CN118012440A

  • System and method for performing end-to-end simulation and testing of an IoT application

    US20220365868A1