Test data generation method and device

Abstract

The invention discloses a test data generation method and device. The method comprises the steps that test metadata are periodically obtained, the test metadata comprise test data construction rules and test data verification rules, and the test data construction rules comprise universal test data construction rules and personalized test data construction rules corresponding to different test scenes; generating universal test data based on the universal test data construction rule; for each test scene, generating personalized test data based on the personalized test data construction rule, and forming a first test data set of the scene with the general test data; and for each first test data set, performing verification based on a test data verification rule, forming a second test data set corresponding to the test scene by the test data passing the verification, and storing the second test data set in the target database. The technical problem that an existing test data generation tool lacks customization ability and is difficult to meet diversified test requirements is solved.

Description

Technical Field

[0001] This application relates to the field of big data testing technology, and more specifically, to a test data generation method and apparatus. Background Technology

[0002] In the context of big data analytics and processing, the generation and correction of test data play a crucial role. Traditional manual methods for inputting test data are extremely inefficient and struggle to ensure data quality, becoming a major bottleneck restricting big data platform testing. Given the rapid development of big data technology and the increasing testing demands, the industry urgently calls for a more efficient and accurate test data generation and correction technology. Automated tools, such as ETL (Extract, Transform, Load) and Apache Trafodion, have become ideal choices for big data testing due to their ability to significantly improve work efficiency and ensure data accuracy and reliability. However, while existing automated test data generation tools have improved test quality and efficiency to some extent, they also reveal significant shortcomings. First, due to the complex learning curve, users often need to invest a lot of time to familiarize themselves with the tool's internal mechanisms, resulting in high initial usage costs. Second, these tools typically lack customization capabilities for personalized testing scenarios, making it difficult to comprehensively cover various testing needs, especially when facing frequently iterating systems, where tool maintenance and updates become new challenges. Therefore, there is an urgent need to develop new technologies to solve the customization problems of current test data construction tools, simplify configuration processes, and reduce maintenance burdens, thereby ensuring that big data testing quality assurance is both accurate and timely.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a test data generation method and apparatus to at least solve the technical problem that existing test data generation tools lack customization capabilities and are unable to meet diverse testing needs.

[0005] According to one aspect of the embodiments of this application, a test data generation method is provided, comprising: periodically acquiring test metadata, wherein the test metadata includes: test data construction rules and test data verification rules, the test data construction rules including: general test data construction rules and personalized test data construction rules corresponding to different test scenarios; generating general test data based on the general test data construction rules; for each test scenario, generating personalized test data corresponding to the test scenario based on the personalized test data construction rules corresponding to the test scenario, and combining the personalized test data and the general test data to form a first test dataset corresponding to the test scenario; for each first test dataset, verifying the test data in the first test dataset based on the test data verification rules, and combining the verified test data to form a second test dataset corresponding to the test scenario, and storing the second test dataset in a target database.

[0006] Optionally, periodically obtaining test metadata includes: periodically obtaining rule data configured by the target object based on the rule configuration entry, wherein the type of the rule configuration entry includes at least one of the following: a graphical user interface integrated with the Vue framework, a custom rule engine; and using a backend framework to parse the rule data to obtain test metadata, wherein the type of the backend framework includes: the Django framework, which is used to communicate with the Vue framework through a RESTful interface.

[0007] Optionally, generating general test data based on general test data construction rules includes: calling a first test data construction tool to generate general test data based on general test data construction rules, wherein the type of the first test data construction tool includes at least one of the following: an AI tool for generating unstructured test data, a Faker tool for generating structured test data, and the Faker tool integrates a Python library.

[0008] Optionally, generating personalized test data corresponding to the test scenario based on personalized test data construction rules includes: calling a second test data construction tool to generate personalized test data based on personalized test data construction rules, wherein the second test data construction tool is a custom executable script, which is generated based on custom rules configured in the graphical user interface of the target object, or generated based on computer language written in the custom rule engine of the target object.

[0009] Optionally, the test data in the first test dataset is validated based on the test data validation rules, including: preprocessing each test data in the first test dataset to obtain a third test dataset, wherein the preprocessing includes at least one of the following: data format conversion, missing value handling; generating a test data validation script based on the test data validation rules; and validating each test data in the third test dataset based on the test data validation script.

[0010] Optionally, after validating the test data in the first test dataset based on the test data validation rules, the method further includes: for each test data that fails validation, calling a test data repair tool to repair the test data, and re-validating the repaired test data; if the repaired test data passes validation, adding the repaired test data to the second test dataset; if the repaired test data fails validation, determining the test data as abnormal data, adding the test data to the abnormal test dataset, and generating an abnormal prompt message.

[0011] Optionally, the method further includes: using a data visualization tool to statistically analyze the verification results of each test data in the first test dataset and generating a data visualization report, wherein the data visualization tool includes at least one of the following: Echarts tool, Tableau tool, and the data visualization report includes at least one of the following: the test data that passed verification and their corresponding quantity, and the test data that failed verification and their corresponding quantity; in response to a report viewing instruction, displaying the data visualization report in a preset format, wherein the preset format includes at least one of the following: chart format, dashboard format.

[0012] Optionally, storing the second test dataset in the target database includes: determining the data type of the test data in the second test dataset, and using a target transmission mechanism to transfer all test data in the second test dataset to the target database corresponding to the data type for storage. The target transmission mechanism includes at least one of the following: Kafka message queue transmission, WebSocket protocol transmission, and the target database includes at least one of the following: a MySQL database for storing structured test data, and a MongoDB database for storing unstructured test data.

[0013] Optionally, each second test dataset has a unique version identifier. After storing the second test dataset in the target database, the method further includes: in response to a management instruction for the second test dataset with the target version identifier, performing a corresponding operation on the second test dataset with the target version identifier, wherein the type of management instruction includes at least one of the following: adding data, deleting data, modifying data, querying data, version modification, and data rollback.

[0014] Optionally, the method further includes: using a backend framework to call a RESTful interface to share each second test dataset in the target database to a third-party platform, wherein the RESTful interface integrates access control technology, and the access control technology includes at least one of the following: OAuth2 authentication technology and JWT token technology.

[0015] According to another aspect of the embodiments of this application, a test data generation apparatus is also provided, comprising: an acquisition module, configured to periodically acquire test metadata, wherein the test metadata includes: test data construction rules and test data verification rules, the test data construction rules including: general test data construction rules and personalized test data construction rules corresponding to different test scenarios; a first generation module, configured to generate general test data based on the general test data construction rules; a second generation module, configured to generate personalized test data corresponding to each test scenario based on the personalized test data construction rules corresponding to the test scenario, and combine the personalized test data and the general test data to form a first test dataset corresponding to the test scenario; and a verification module, configured to verify the test data in the first test dataset based on the test data verification rules for each first test dataset, and combine the verified test data to form a second test dataset corresponding to the test scenario, and store the second test dataset in a target database.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described test data generation method.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described test data generation method through the computer program.

[0018] In this embodiment, by periodically acquiring test metadata, it is ensured that the construction and verification rules for test data can reflect the latest test requirements and environmental changes in a timely manner, thereby improving the accuracy and coverage of the tests. By using general test data construction rules, a large amount of structured and unstructured test data can be quickly generated, covering the basic testing requirements of the big data platform. Using a custom rule engine, test data that is more closely aligned with actual business scenarios is generated based on personalized test data construction rules, making the test data more accurate. This enables in-depth testing of specific functions or performance of the big data platform, improving the targeting and effectiveness of the tests. By defining test data verification rules and making... Verification tools are used to validate each first test dataset, automatically checking the accuracy and completeness of the test data. This ensures that only test data that meets quality requirements can be used for subsequent big data testing, greatly improving the usability of the test data and reducing the error rate and omissions during the testing process. The validated test data is then combined into a second test dataset and stored in the target database, achieving persistent storage and version control of the test data. It also provides secure data sharing and access capabilities, promoting the reuse and collaboration of test data, reducing the waste of test resources and repetitive work, and thus solving the technical problem that existing test data generation tools lack customization capabilities and are unable to meet diverse testing needs. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart illustrating an optional test data generation method according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the structure of an optional test data generation system according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the implementation process of an optional test data generation method according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an optional test data generation device according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Example 1

[0028] According to an embodiment of this application, a test data generation method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 1 This is a flowchart illustrating a test data generation method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0030] Step S102: Periodically acquire test metadata, which includes test data construction rules and test data verification rules. The test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios.

[0031] Step S104: Generate general test data based on general test data construction rules;

[0032] Step S106: For each test scenario, generate personalized test data corresponding to the test scenario based on the personalized test data construction rules, and combine the personalized test data and general test data to form the first test dataset corresponding to the test scenario.

[0033] Step S108: For each first test dataset, the test data in the first test dataset is verified based on the test data verification rules, and the verified test data is used to form a second test dataset for the corresponding test scenario. The second test dataset is then stored in the target database.

[0034] The following section explains each step of the test data generation method in conjunction with the specific implementation process.

[0035] Test metadata is periodically retrieved. The test metadata includes test data construction rules and test data verification rules. The test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios.

[0036] As an optional implementation, periodically obtaining test metadata can be achieved in the following ways: periodically obtaining rule data configured by the target object based on the rule configuration entry point, wherein the type of the rule configuration entry point includes at least one of the following: a graphical user interface integrated with the Vue framework, or a custom rule engine; and using a backend framework to parse the rule data to obtain test metadata, wherein the type of the backend framework includes: the Django framework, which is used to communicate with the Vue framework through a RESTful interface.

[0037] For example, to ensure the timely receipt of the latest test data construction and validation rules, thus adapting to constantly changing testing needs and environments, two main rule configuration entry points are provided: a Vue-integrated graphical user interface and a custom rule engine. The Vue-integrated graphical user interface is the direct interface for testers and administrators, allowing configuration of general and personalized test data construction rules through intuitive UI components (such as dropdown menus, text input boxes, checkboxes, etc.). Vue.js provides a reactive and component-based development approach, making the user interface more flexible and user-friendly. The custom rule engine is used for more complex or rule-specific test scenarios. This typically involves programming, allowing users to define personalized test data construction rules by writing Python scripts, providing greater customization and flexibility. The backend uses the Django framework to receive rule data from the Vue interface or rule scripts from the custom rule engine. Through Django's model parsing and RESTful interface communication capabilities, the received rule data is parsed into a system-understandable test metadata format, preparing it for subsequent steps.

[0038] Specifically, when the graphical user interface integrated with the Vue framework serves as the entry point for rule configuration, it features an intuitive rule configuration area on the front-end interface. This area includes a series of forms, dropdown menus, and draggable elements to collect metadata information from the user, such as rules for constructing and validating test data. The rule configuration area supports the following main functions:

[0039] Configure data types: Users can specify the type of test data, including structured data (such as numbers, dates, strings, etc.) and unstructured data (such as text, images, audio, etc.).

[0040] Set data size: Allows users to set the size of the generated test data, including the number of records, file size, etc., to meet the needs of different test scenarios.

[0041] Define data distribution: Users can define data distribution patterns, such as normal distribution, uniform distribution or other complex distributions, by dragging and dropping, to generate test data that is closer to real-world scenarios.

[0042] Customized validation rules: Users can set data validation rules, such as data format, range, uniqueness, etc., to ensure that the generated test data meets specific validation requirements.

[0043] In addition to the graphical user interface, a custom rule engine configuration entry can be provided, allowing users to define test rules using more advanced configuration methods. The custom rule engine supports the following features:

[0044] Writing Python scripts: Users can write Python scripts to define more complex test logic and data construction rules, which provides greater flexibility for advanced users.

[0045] Rule Template Library: Provides a set of rule templates, which users can directly select templates for fine-tuning to quickly generate test data for specific scenarios.

[0046] Real-time preview: During the rule editing process, a real-time preview function can be provided, allowing users to instantly view the effect of the rule configuration and ensure that the data construction meets expectations.

[0047] After obtaining the general test data construction rules from the test metadata, general test data is generated based on the general test data construction rules.

[0048] As an optional implementation, generating general test data based on general test data construction rules can be achieved by calling a first test data construction tool to generate general test data based on general test data construction rules. The first test data construction tool includes at least one of the following types: an AI tool for generating unstructured test data, or a Faker tool for generating structured test data, wherein the Faker tool integrates a Python library.

[0049] For example, the Django framework reads common test data construction rules from a MySQL database. These rules are common across test scenarios and define the basic structure, type, and format of the data. Django uses Python code to parse these rules, converting them into a format that the Faker and AI tools can understand. Based on the parsed rules, appropriate Python scripts are written or called. These scripts integrate the Faker library and define the logic for calling methods in the Faker library, such as generating structured data like random names, addresses, and phone numbers. Once the script is executed, the Faker tool generates structured test data according to the rules. If the test scenario requires unstructured test data, such as random text, images, or videos, the AI ​​tool will be called. The AI ​​tool may be pre-trained or trained on-site according to the rules to generate unstructured data that meets the needs of the scenario. The data generated by the AI ​​tool will also be formatted according to the common construction rules to ensure compatibility with the overall dataset.

[0050] After obtaining the personalized test data construction rules from the test metadata, for each test scenario, personalized test data corresponding to the test scenario is generated based on the personalized test data construction rules corresponding to the test scenario, and the personalized test data and general test data are combined to form the first test dataset corresponding to the test scenario.

[0051] As an optional implementation, personalized test data corresponding to the test scenario can be generated by constructing rules based on personalized test data corresponding to the test scenario. This can be achieved by calling a second test data construction tool to generate personalized test data based on personalized test data construction rules. The second test data construction tool is a custom executable script. The executable script is generated based on the custom rules configured by the target object in the graphical user interface, or based on the computer language written by the target object in a custom rule engine.

[0052] In big data testing, to cover various possible scenarios, especially personalized scenarios with specific business logic and requirements, two types of test data need to be generated: general test data and personalized test data. General test data is generated to meet common testing needs, while personalized test data is customized for the specific requirements of each test scenario. The following process describes how to generate this personalized test data and merge it with general data to form a complete test dataset: A custom test data construction tool is called. This tool can be a script specifically written for generating personalized data. It generates data based on rules submitted by the user through the interface or a custom rule engine. The script runs in Python. Executed in a Python environment, leveraging Python's powerful data processing and generation capabilities, the system ensures that the data meets the requirements of personalized scenarios. This data can be generated completely randomly or modified or expanded based on existing data, ensuring data diversity and authenticity. Once the personalized test data is generated, it is integrated with the general test data previously generated using tools like Faker to form the first test dataset for a specific test scenario. The system automatically identifies which data belongs to a particular test scenario and which is general data, then merges them according to a preset method. For example, a basic data skeleton can be generated first, and then personalized data can be populated into it to fully cover various situations in the test scenario.

[0053] Before integration is complete, personalized and general data may be temporarily stored in Kafka or MongoDB for management and subsequent processing. Each test dataset will be tagged with a version number, which helps to track data changes and allows for quick backtracking or updating of the dataset when the test scenario changes.

[0054] After obtaining the first test dataset, for each first test dataset, the test data in the first test dataset is verified based on the test data verification rules, and the verified test data is used to form the second test dataset corresponding to the test scenario. The second test dataset is then stored in the target database.

[0055] As an optional implementation, the test data in the first test dataset is validated based on the test data validation rules. This can be achieved by: preprocessing each test data in the first test dataset to obtain a third test dataset, wherein the preprocessing includes at least one of the following: data format conversion, missing value handling; generating a test data validation script based on the test data validation rules; and validating each test data in the third test dataset based on the test data validation script.

[0056] For example, since the generated test data may come from different tools (such as Faker or AI tools), the formats may differ. One task of the preprocessing stage is to convert the data format to unify it and make it compatible with the big data platform being tested. For example, converting date strings to the correct date and time format, or converting text data to a specific encoding. Test data may contain missing values, which is unacceptable in big data testing because real-world data is often complete. The preprocessing stage uses imputation methods (such as mean, mode, values ​​predicted by machine learning, etc.) to fill in missing values ​​to ensure data integrity. The preprocessed dataset is called the third test dataset. Based on the test data validation rules, the system automatically generates validation scripts. These rules can be specifications regarding data type, range, uniqueness, correlation, etc. The generated scripts use libraries such as Pandas to perform rule checks, ensuring that each data point meets the requirements. The validation scripts run on the third test dataset, performing a detailed check on each data entry. This step may involve multiple levels of validation, from simple format verification to complex business logic verification.

[0057] As an optional implementation, after validating the test data in the first test dataset based on the test data validation rules, the method further includes: for each test data that fails validation, calling a test data repair tool to repair the test data, and re-validating the repaired test data; if the repaired test data passes validation, adding the repaired test data to the second test dataset; if the repaired test data fails validation, determining the test data as abnormal data, adding the test data to the abnormal test dataset, and generating an abnormal prompt message.

[0058] For example, once data verification detects non-compliant data, the system will activate an automatic repair mechanism to attempt to repair the data according to preset rules. For instance, if the data exceeds the allowed range, it can be cropped; if the data format is incorrect, it can be converted; if the data can be repaired using a preset algorithm, the system will automatically repair it. The repaired data will be verified again to confirm whether it meets the standards. If the data cannot be repaired, or is still non-compliant after repair, it will be marked as abnormal data, stored separately in an abnormal test dataset, and an abnormal prompt message will be generated to remind testers to pay attention.

[0059] As an optional implementation, storing the second test dataset in the target database can be achieved by: determining the data type of the test data in the second test dataset, and using a target transmission mechanism to transfer all test data in the second test dataset to the target database corresponding to the data type for storage. The target transmission mechanism includes at least one of the following: Kafka message queue transmission, WebSocket protocol transmission, and the target database includes at least one of the following: a MySQL database for storing structured test data, and a MongoDB database for storing unstructured test data.

[0060] For example, the data that passes the verification is called the second test dataset. This data will be stored in the target database to prepare for subsequent big data testing. The database to be stored depends on the type of data (structured and unstructured). Structured data, such as user information and transaction records, will be stored in a MySQL database; unstructured data, such as images and audio files, will be stored in a MongoDB database. By using Kafka message queues or WebSocket protocols, data can be efficiently transmitted to the target database, ensuring real-time and reliable data transmission even with large amounts of data.

[0061] As an optional implementation, the method further includes: using a data visualization tool to statistically analyze the verification results of each test data in the first test dataset and generating a data visualization report, wherein the data visualization tool includes at least one of the following: Echarts tool, Tableau tool, and the data visualization report includes at least one of the following: the test data that passed verification and their corresponding quantity, and the test data that failed verification and their corresponding quantity; in response to a report viewing instruction, displaying the data visualization report in a preset format, wherein the preset format includes at least one of the following: chart format, dashboard format.

[0062] For example, data visualization is an important tool for interpreting test data validation results. It transforms complex data into intuitive charts or dashboards, helping testers quickly understand and analyze data quality. Echarts or Tableau can be chosen as data visualization tools. Echarts is an open-source charting library based on JavaScript, suitable for rapid development and lightweight deployment, while Tableau is a professional enterprise-level data visualization software, better suited for large and complex datasets. The system automatically calculates the validation results for each test data point, including the number of data points that passed and failed validation, and converts this information into a data visualization report in the form of charts or dashboards. The report can also include additional information such as key data attributes and distribution. Users can request to view the data visualization report by sending a "report view command," and the system will display the report in a preset format (such as line charts, bar charts, heatmaps, dashboards, etc.), allowing users to quickly understand the overall status and trends of the test data.

[0063] As an optional implementation, each second test dataset has a unique version identifier. After storing the second test dataset in the target database, the method further includes: in response to a management instruction for the second test dataset with the target version identifier, performing a corresponding operation on the second test dataset with the target version identifier, wherein the type of management instruction includes at least one of the following: adding data, deleting data, modifying data, querying data, version modification, and data rollback.

[0064] For example, to ensure the traceability and consistency of test data, each qualified test dataset (second test dataset) is assigned a unique version identifier. This identifier can be a timestamp, hash value, or a custom version number, uniquely identifying the state of a test dataset. Testers or administrators can add, delete, modify, or query specific versions of test data by sending "management commands," and can even perform version modifications or data rollbacks to a previous data state. This mechanism is similar to version control systems in software development, such as Git, but applied in the context of test data management. After receiving management commands, the system executes the corresponding operations based on the target version identifier of the second test dataset. All these operations are recorded for future auditing or recovery.

[0065] As an optional implementation, the above method further includes: using a backend framework to call a RESTful interface to share each second test dataset in the target database to a third-party platform, wherein the RESTful interface integrates access control technology, and the access control technology includes at least one of the following: OAuth2 authentication technology and JWT token technology.

[0066] For example, to enable test data to be shared among various teams or collaborating organizations, the system uses RESTful interfaces to send datasets to third-party platforms while ensuring data security. RESTful interfaces are an interface design pattern based on the HTTP protocol, allowing data exchange between different systems. By calling these interfaces, third-party systems can obtain specific versions of the test dataset. The system integrates OAuth2 authentication or JWT token technology to ensure that only users or systems with legitimate access rights can obtain the data. OAuth2 is an authorization protocol that enables applications to obtain limited permissions on behalf of users, while JWT (JSON Web Token) is an open standard for authentication, proving the holder's identity by generating an encrypted token. Under secure conditions, the test dataset is transmitted to third parties through RESTful interfaces, which not only reduces data redundancy and improves testing efficiency but also promotes collaboration across teams.

[0067] Data visualization reports should be automatically updated to reflect the latest verification results to maintain the timeliness and accuracy of the reports. The system should support multi-level access control, assigning different permissions according to user roles (such as development, testing, and management) to ensure that sensitive parts of the data are only visible to authorized users. In addition to version control, a data lifecycle management strategy should be developed to clarify the data retention period and deletion mechanism to avoid excessive data backlog and unnecessary storage space occupation.

[0068] Figure 2 The components of a test data generation system according to this embodiment are shown, such as... Figure 1The system integrates the Vue framework's graphical user interface to configure rule data. Alternatively, a custom rule engine can be used for configuration. The backend framework Django parses the rule data, generates test metadata, and communicates with the frontend Vue framework via a RESTful interface. The Data Generation Engine (DGE), including AI tools, Faker, and a custom rule engine, is responsible for generating test data. General test data can be generated using AI tools or Faker, while personalized test data is generated using custom scripts. Based on the rules configured by the user in the UI or custom rule engine, the test data validation module, including Pytest and Pandas, validates the generated first test dataset, including preprocessing, generating validation scripts, and executing validation. Data that fails validation can be repaired and re-validated, ultimately forming the second test dataset. The test data storage and management module includes MySQL, MongoDB, and Kafka. The second test dataset is stored in the target database (e.g., MySQL or MongoDB) according to data type, and data transmission and version management are performed using mechanisms such as Kafka. The test data import / export module includes an API Gateway, which shares the test dataset stored in the target database to third-party platforms, supporting access control technologies such as OAuth2 authentication and JWT tokens.

[0069] Figure 3 The specific implementation process of the test data generation system is given, such as... Figure 3 As shown, users configure rule data through a graphical user interface integrated with the Vue framework, which is then parsed and managed by the Django backend framework. Rules are constructed based on general test data, and general test data is generated using AI tools or generation tools like Faker. This generated general test data is temporarily stored for later processing and verification. Personalized test data is generated using a custom rule engine based on personalized rules for different test scenarios. This personalized test data is also temporarily stored for later processing and verification. Tools such as Pytest and Pandas are used to verify both the generated general and personalized test data. Data that fails verification is corrected until all data meets the rule requirements. The verified and corrected general test data is stored, forming the final general test dataset. Similarly, the verified and corrected personalized test data is also stored, forming the final personalized test dataset. Finally, the verified and passed test data (including general and personalized test data) is imported into the target database or exported for use by other systems via an API interface. The entire process begins with the user configuring rules on the front-end interface, proceeds through data generation, verification, storage, and management, and ultimately achieves automated management and sharing of test data.

[0070] In this embodiment, by periodically acquiring test metadata, it is ensured that the construction and verification rules for test data can reflect the latest test requirements and environmental changes in a timely manner, thereby improving the accuracy and coverage of the tests. By using general test data construction rules, a large amount of structured and unstructured test data can be quickly generated, covering the basic testing requirements of the big data platform. Using a custom rule engine, test data that is more closely aligned with actual business scenarios is generated based on personalized test data construction rules, making the test data more accurate. This enables in-depth testing of specific functions or performance of the big data platform, improving the targeting and effectiveness of the tests. By defining test data verification rules and making... Verification tools are used to validate each first test dataset, automatically checking the accuracy and completeness of the test data. This ensures that only test data that meets quality requirements can be used for subsequent big data testing, greatly improving the usability of the test data and reducing the error rate and omissions during the testing process. The validated test data is then combined into a second test dataset and stored in the target database, achieving persistent storage and version control of the test data. It also provides secure data sharing and access capabilities, promoting the reuse and collaboration of test data, reducing the waste of test resources and repetitive work, and thus solving the technical problem that existing test data generation tools lack customization capabilities and are unable to meet diverse testing needs.

[0071] Example 2

[0072] According to an embodiment of this application, a test data generation apparatus for implementing the test data generation method in Embodiment 1 is also provided, such as... Figure 4 As shown, the test data generation device includes at least: an acquisition module 41, a first generation module 42, a second generation module 43, and a verification module 43, wherein:

[0073] The acquisition module 41 is used to periodically acquire test metadata, which includes test data construction rules and test data verification rules. The test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios.

[0074] The first generation module 42 is used to generate general test data based on general test data construction rules;

[0075] The second generation module 43 is used to generate personalized test data corresponding to each test scenario based on the personalized test data construction rules corresponding to the test scenario, and to combine the personalized test data and the general test data to form the first test dataset corresponding to the test scenario.

[0076] The verification module 44 is used to verify the test data in each first test dataset based on the test data verification rules, and to form a second test dataset corresponding to the test scenario by combining the verified test data, and to store the second test dataset in the target database.

[0077] The following section describes the functions of each module of the test data generation device in conjunction with the specific implementation process.

[0078] The module periodically retrieves test metadata, which includes test data construction rules and test data verification rules. The test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios.

[0079] As an optional implementation, periodically obtaining test metadata can be achieved in the following ways: periodically obtaining rule data configured by the target object based on the rule configuration entry point, wherein the type of the rule configuration entry point includes at least one of the following: a graphical user interface integrated with the Vue framework, or a custom rule engine; and using a backend framework to parse the rule data to obtain test metadata, wherein the type of the backend framework includes: the Django framework, which is used to communicate with the Vue framework through a RESTful interface.

[0080] The first generation module generates general test data based on general test data construction rules.

[0081] As an optional implementation, generating general test data based on general test data construction rules can be achieved by calling a first test data construction tool to generate general test data based on general test data construction rules. The first test data construction tool includes at least one of the following types: an AI tool for generating unstructured test data, or a Faker tool for generating structured test data, wherein the Faker tool integrates a Python library.

[0082] For each test scenario, the second generation module generates personalized test data corresponding to the test scenario based on the personalized test data construction rules corresponding to the test scenario, and combines the personalized test data and the general test data to form the first test dataset corresponding to the test scenario.

[0083] As an optional implementation, personalized test data corresponding to the test scenario can be generated by constructing rules based on personalized test data corresponding to the test scenario. This can be achieved by calling a second test data construction tool to generate personalized test data based on personalized test data construction rules. The second test data construction tool is a custom executable script. The executable script is generated based on the custom rules configured by the target object in the graphical user interface, or based on the computer language written by the target object in a custom rule engine.

[0084] For each first test dataset, the verification module verifies the test data in the first test dataset based on the test data verification rules, and combines the verified test data into a second test dataset corresponding to the test scenario, and stores the second test dataset in the target database.

[0085] As an optional implementation, the test data in the first test dataset is validated based on the test data validation rules. This can be achieved by: preprocessing each test data in the first test dataset to obtain a third test dataset, wherein the preprocessing includes at least one of the following: data format conversion, missing value handling; generating a test data validation script based on the test data validation rules; and validating each test data in the third test dataset based on the test data validation script.

[0086] As an optional implementation, after validating the test data in the first test dataset based on the test data validation rules, the method further includes: for each test data that fails validation, calling a test data repair tool to repair the test data, and re-validating the repaired test data; if the repaired test data passes validation, adding the repaired test data to the second test dataset; if the repaired test data fails validation, determining the test data as abnormal data, adding the test data to the abnormal test dataset, and generating an abnormal prompt message.

[0087] As an optional implementation, the method further includes: using a data visualization tool to statistically analyze the verification results of each test data in the first test dataset and generating a data visualization report, wherein the data visualization tool includes at least one of the following: Echarts tool, Tableau tool, and the data visualization report includes at least one of the following: the test data that passed verification and their corresponding quantity, and the test data that failed verification and their corresponding quantity; in response to a report viewing instruction, displaying the data visualization report in a preset format, wherein the preset format includes at least one of the following: chart format, dashboard format.

[0088] As an optional implementation, storing the second test dataset in the target database can be achieved by: determining the data type of the test data in the second test dataset, and using a target transmission mechanism to transfer all test data in the second test dataset to the target database corresponding to the data type for storage. The target transmission mechanism includes at least one of the following: Kafka message queue transmission, WebSocket protocol transmission, and the target database includes at least one of the following: a MySQL database for storing structured test data, and a MongoDB database for storing unstructured test data.

[0089] As an optional implementation, each second test dataset has a unique version identifier. After storing the second test dataset in the target database, the method further includes: in response to a management instruction for the second test dataset with the target version identifier, performing a corresponding operation on the second test dataset with the target version identifier, wherein the type of management instruction includes at least one of the following: adding data, deleting data, modifying data, querying data, version modification, and data rollback.

[0090] As an optional implementation, the above method further includes: using a backend framework to call a RESTful interface to share each second test dataset in the target database to a third-party platform, wherein the RESTful interface integrates access control technology, and the access control technology includes at least one of the following: OAuth2 authentication technology and JWT token technology.

[0091] It should be noted that each module in the test data generation device in this application embodiment corresponds one-to-one with each implementation step of the test data generation method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0092] Example 3

[0093] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the test data generation method in embodiment 1.

[0094] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the test data generation method in Embodiment 1 by running the computer program.

[0095] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the test data generation method in Embodiment 1 during runtime.

[0096] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the test data generation method of Embodiment 1 through the computer program.

[0097] Specifically, the computer program executes the following steps during runtime: periodically acquiring test metadata, which includes test data construction rules and test data verification rules. The test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios; generating general test data based on the general test data construction rules; for each test scenario, generating personalized test data corresponding to the test scenario based on the personalized test data construction rules corresponding to the test scenario, and combining the personalized test data and the general test data to form a first test dataset corresponding to the test scenario; for each first test dataset, verifying the test data in the first test dataset based on the test data verification rules, and combining the verified test data to form a second test dataset corresponding to the test scenario, and storing the second test dataset in the target database.

[0098] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 5 A hardware block diagram of an electronic device for implementing a test data generation method is shown. Figure 5 As shown, the electronic device 50 may include one or more processors 502 (shown as 502a, 502b, ..., 502n in the figure) 502 (processor 502 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 504 for storing data, and a transmission device 506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, electronic device 50 may also include... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.

[0099] It should be noted that the aforementioned one or more processors 502 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 50. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0100] The memory 504 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the test data generation method in this embodiment. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, thereby implementing the above-mentioned application vulnerability detection method. The memory 504 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 504 may further include memory remotely located relative to the processor 502, and these remote memories can be connected to the electronic device 50 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] The transmission device 506 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 50. In one example, the transmission device 506 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 506 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0102] The display may be, for example, a touchscreen liquid crystal display (LCD), which allows a user to interact with the user interface of the electronic device 50.

[0103] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0104] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0106] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0109] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for generating test data, characterized in that, include: Periodically acquire test metadata, which includes test data construction rules and test data verification rules. The test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios. Generate general test data based on the rules for constructing the general test data; For each test scenario, personalized test data corresponding to the test scenario is generated based on the personalized test data construction rules corresponding to the test scenario, and the personalized test data and the general test data are combined to form the first test dataset corresponding to the test scenario. For each first test dataset, the test data in the first test dataset is verified based on the test data verification rules, and the verified test data is used to form a second test dataset for the corresponding test scenario. The second test dataset is then stored in the target database.

2. The method according to claim 1, characterized in that, Periodically retrieve test metadata, including: Periodically obtain rule data configured for the target object based on the rule configuration entry, wherein the type of the rule configuration entry includes at least one of the following: a graphical user interface integrated with the Vue framework, or a custom rule engine; The rule data is parsed using a backend framework to obtain the test metadata. The type of the backend framework includes: Django framework, which is used to communicate with the Vue framework through a RESTful interface.

3. The method according to claim 1, characterized in that, Generate general test data based on the general test data generation rules, including: The first test data construction tool is invoked to generate the general test data based on the general test data construction rules. The first test data construction tool is of at least one of the following types: an AI tool for generating unstructured test data, or a Faker tool for generating structured test data, wherein the Faker tool integrates a Python library.

4. The method according to claim 2, characterized in that, Personalized test data corresponding to the test scenario is generated based on the personalized test data construction rules, including: The second test data construction tool is invoked to generate the personalized test data based on the personalized test data construction rules. The second test data construction tool is a custom executable script, which is generated based on the custom rules configured by the target object in the graphical user interface, or based on a computer language written by the target object in the custom rule engine.

5. The method according to claim 1, characterized in that, The test data in the first test dataset is validated based on the test data validation rules, including: Each test data in the first test dataset is preprocessed to obtain a third test dataset, wherein the preprocessing includes at least one of the following: data format conversion and missing value handling; A test data verification script is generated based on the aforementioned test data verification rules; The test data verification script verifies each test data in the third test dataset.

6. The method according to claim 1, characterized in that, After validating the test data in the first test dataset based on the test data validation rules, the method further includes: For each test data that fails the verification, the test data repair tool is invoked to repair the test data, and the repaired test data is re-verified; If the repaired test data passes the verification, the repaired test data will be added to the second test dataset; If the repaired test data fails the verification, the test data is determined to be abnormal data. The test data is then added to the abnormal test dataset, and an abnormal prompt message is generated.

7. The method according to claim 1, characterized in that, The method further includes: The validation results of each test data in the first test dataset are statistically analyzed using a data visualization tool, and a data visualization report is generated. The data visualization tool includes at least one of the following: Echarts tool and Tableau tool. The data visualization report includes at least one of the following: the test data that passed validation and the corresponding quantity, and the test data that failed validation and the corresponding quantity. In response to a report viewing command, the data visualization report is displayed in a preset format, wherein the preset format includes at least one of the following: chart format and dashboard format.

8. The method according to claim 1, characterized in that, Storing the second test dataset into the target database includes: The data type of the test data in the second test dataset is determined, and all test data in the second test dataset is transferred to the target database corresponding to the data type for storage using the target transmission mechanism. The target transmission mechanism includes at least one of the following: Kafka message queue transmission and WebSocket protocol transmission. The target database includes at least one of the following: a MySQL database for storing structured test data and a MongoDB database for storing unstructured test data.

9. The method according to claim 1, characterized in that, Each of the second test datasets has a unique version identifier. After storing the second test datasets in the target database, the method further includes: In response to a management instruction for a second test dataset with a target version identifier, a corresponding operation is performed on the second test dataset with the target version identifier, wherein the type of the management instruction includes at least one of the following: adding data, deleting data, modifying data, querying data, version modification, and data rollback.

10. The method according to claim 1, characterized in that, The method further includes: The backend framework calls a RESTful interface to share each of the second test datasets in the target database to a third-party platform. The RESTful interface integrates access control technology, which includes at least one of the following: OAuth2 authentication technology and JWT token technology.

11. A test data generation device, characterized in that, include: The acquisition module is used to periodically acquire test metadata, wherein the test metadata includes test data construction rules and test data verification rules, and the test data construction rules include general test data construction rules and personalized test data construction rules corresponding to different test scenarios. The first generation module is used to generate general test data based on the general test data construction rules. The second generation module is used to generate personalized test data corresponding to each test scenario based on the personalized test data construction rules corresponding to the test scenario, and to combine the personalized test data and the general test data to form the first test dataset corresponding to the test scenario. The verification module is used to verify the test data in each first test dataset based on the test data verification rules, and to form a second test dataset corresponding to the test scenario by combining the verified test data, and to store the second test dataset in the target database.

12. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the test data generation method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the test data generation method of any one of claims 1 to 10 through the computer program.

Images