Methods, apparatus, storage media, and electronic devices for preparing test data

Abstract

This application discloses a method, apparatus, storage medium, and electronic device for preparing test data, relating to the field of big data. The method includes: acquiring a target batch script, wherein the target batch script is used to process big data; parsing the target code in the target batch script to obtain a tree structure and operators for the target code, wherein the tree structure represents the structural information between codes in the target code, and the operators represent the mapping relationship between codes in the target code; generating a target data table and target fields in the target data table based on the tree structure and operators; and parsing the target data table and target fields to generate target test data. This application solves the problem of low efficiency in test data preparation caused by manual methods in related technologies.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a method and apparatus for preparing test data, a storage medium, and an electronic device. Background Technology

[0002] Currently, in related technologies, testing large-scale batch scripts primarily involves manually preparing test data and automatically running the program in a test environment for verification. The specific steps are as follows:

[0003] (1) Manual analysis and organization of data retrieval logic list: Write the data retrieval script required for the test big data batch script, which consists of a series of SQL statements. From the big data batch script, it is necessary to accurately summarize and find the source data fields required by the test target script and the operator logic to be tested from the test database.

[0004] (2) Prepare test data manually on the local machine according to the compiled list: Execute the script manually written in step (1) to obtain the source data to be tested;

[0005] (3) Data is manually imported into the test environment for testing;

[0006] (4) Automated loading test of big data scripts: Execute the big data scripts to be tested, observe the execution process, and finally check whether the running results meet expectations.

[0007] However, current big data testing solutions lack automated methods for parsing and generating data; test data preparation remains manual. Furthermore, manual data generation is time-consuming, requiring testers to read batch scripts for large datasets, analyze the code using specialized coding knowledge, and write their own data generation parsing scripts. However, during the analysis of the test object's code, varying levels of understanding of the code logic and differing skill levels among testers can lead to errors in the manually generated data scripts, resulting in rework and low efficiency.

[0008] There is currently no effective solution to the problem of low efficiency in preparing test data due to the use of manual methods in related technologies. Summary of the Invention

[0009] The main objective of this application is to provide a method, apparatus, storage medium, and electronic device for preparing test data, in order to solve the problem that the preparation of test data is inefficient due to the use of manual methods in related technologies.

[0010] To achieve the above objectives, according to one aspect of this application, a method for preparing test data is provided. The method includes: obtaining a target batch script, wherein the target batch script is used to process large amounts of data; parsing target code in the target batch script to obtain a tree structure of the target code and operators of the target code, wherein the tree structure represents structural information between codes in the target code, and the operators represent mapping relationships between codes in the target code; generating a target data table and target fields in the target data table based on the tree structure and the operators; and parsing the target data table and the target fields to generate target test data.

[0011] Furthermore, after parsing the target data table and the target fields to generate target test data, the method further includes: determining a target test file, wherein the target test file includes at least two or more of the target test data; and using the target test file to verify the correctness of the target batch script.

[0012] Furthermore, before verifying the correctness of the target batch script using the target test file, the method further includes: determining the data volume of each target test data in the target test file and the correlation degree between every two target test data; obtaining the expected result of running the target batch script based on the data volume and the correlation degree; determining test cases based on the target test data in the target test file; and executing the target batch script based on the test cases to obtain the test result of the target batch script.

[0013] Furthermore, verifying the correctness of the target batch script using the target test file includes: determining whether the expected result is the same as the test result; if the expected result is the same as the test result, the target batch script is correct; if the expected result is different from the test result, the target batch script is incorrect.

[0014] Further, parsing the target data table and the target field to generate target test data includes: determining the operator of the target field; parsing the associated field in the target source table based on the operator of the target field, wherein the target source table is a table associated with the target data table; extracting data from the target source table based on the associated field; and generating the target test data based on the data extracted from the source table.

[0015] Furthermore, after parsing the target code in the target batch script to obtain the tree structure and operators corresponding to the target code, the method further includes: generating target script one and target script two based on the tree structure and the operators; parsing the target code using target script one to obtain the target data table and the operators of the target fields; and generating the target test data based on target script two and the target fields.

[0016] Further, determining the target test file includes: acquiring target test data one and target test data two according to a preset method, wherein target test data one is target test data that conforms to the target batch script execution scenario, and target test data two is target test data that does not conform to the target batch script execution scenario; and determining the target test file based on target test data one and target test data two.

[0017] Furthermore, before parsing the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code, the method further includes: determining whether there is unparsed code in the target batch script; if there is unparsed code in the target batch script, then the unparsed code is used as the target code; if there is no unparsed code in the target batch script, then the parsing of the target batch script is complete.

[0018] To achieve the above objectives, according to another aspect of this application, a test data preparation apparatus is provided. The apparatus includes: a first acquisition unit for acquiring a target batch script, wherein the target batch script is used to process large amounts of data; a first parsing unit for parsing target code in the target batch script to obtain a tree structure of the target code and operators of the target code, wherein the tree structure represents structural information between codes in the target code, and the operators represent mapping relationships between codes in the target code; a first generation unit for generating a target data table and target fields in the target data table based on the tree structure and the operators; and a second parsing unit for parsing the target data table and the target fields to generate target test data.

[0019] Furthermore, the apparatus further includes: a first determining unit, configured to determine a target test file after parsing the target data table and the target field to generate target test data, wherein the target test file includes at least two or more target test data; and a first verification unit, configured to verify the correctness of the target batch script using the target test file.

[0020] Furthermore, the apparatus further includes: a second determining unit, configured to determine the data volume of each target test data in the target test file and the correlation degree between every two target test data before verifying the correctness of the target batch script using the target test file; a first processing unit, configured to obtain the expected result of running the target batch script based on the data volume and the correlation degree; a third determining unit, configured to determine test cases based on the target test data in the target test file; and a second processing unit, configured to execute the target batch script based on the test cases to obtain the test result of the target batch script.

[0021] Further, the first verification unit includes: a first judgment module, used to judge whether the expected result is the same as the test result; a first processing module, used to indicate that the target batch script is correct if the expected result is the same as the test result; and a second processing module, used to indicate that the target batch script is incorrect if the expected result is not the same as the test result.

[0022] Further, the second parsing unit includes: a first determining module, used to determine the operator of the target field; a first parsing module, used to parse out the associated field in the target source table based on the operator of the target field, wherein the target source table is a table associated with the target data table; a first extraction module, used to extract data from the target source table based on the associated field; and a first generation module, used to generate the target test data based on the data extracted from the source table.

[0023] Furthermore, the apparatus further includes: a second generation unit, configured to, after parsing the target code in the target batch script to obtain the tree structure and operators corresponding to the target code, generate target script one and target script two based on the tree structure and the operators; a third processing unit, configured to parse the target code using target script one to obtain the target data table and the operators of the target fields; and a third generation unit, configured to generate the target test data based on target script two and the target fields.

[0024] Further, the first determining unit includes: a first acquiring module, used to acquire target test data one and target test data two according to a preset method, wherein the target test data one is target test data that conforms to the target batch script running scenario, and the target test data two is target test data that does not conform to the target batch script running scenario; and a second determining module, used to determine the target test file based on the target test data one and the target test data two.

[0025] Furthermore, the device further includes: a first determining unit, configured to determine whether there is unparsed code in the target batch script before parsing the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code; a fourth processing unit, configured to use the unparsed code as the target code if there is unparsed code in the target batch script; and a fifth processing unit, configured to indicate that the parsing of the target batch script is complete if there is no unparsed code in the target batch script.

[0026] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided, the storage medium storing a program, wherein the program executes the test data preparation method described in any one of the above claims.

[0027] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including one or more processors and a memory, the memory being used to store one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the test data preparation method described in any one of the above.

[0028] This application employs the following steps: obtaining a target batch script, which is used to process large amounts of data; parsing the target code in the target batch script to obtain the tree structure and operators of the target code, wherein the tree structure represents the structural information between codes in the target code, and the operators represent the mapping relationship between codes in the target code; generating a target data table and target fields in the target data table based on the tree structure and operators; and parsing the target data table and target fields to generate target test data. This solves the problem of low efficiency in test data preparation caused by manual methods in related technologies. By generating a target data table and target fields based on the tree structure and operators of the target code obtained from parsing the target code in the acquired target batch script, and then parsing the target data table and target fields to generate target test data, test data can be automatically generated, thereby improving the efficiency of test data preparation. Attached Figure Description

[0029] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 This is a flowchart of a method for preparing test data according to an embodiment of this application;

[0031] Figure 2 This is a flowchart of a test batch big data batch script in an embodiment of this application;

[0032] Figure 3 This is a flowchart illustrating the automated parsing of upstream data tables and corresponding fields in an embodiment of this application;

[0033] Figure 4 This is a flowchart of the automated big data batch parsing script in the embodiments of this application;

[0034] Figure 5 This is a schematic diagram of a test data preparation device provided according to an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0040] The present invention will now be described in conjunction with preferred implementation steps. Figure 1 This is a flowchart of a test data preparation method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0041] Step S101: Obtain the target batch script, wherein the target batch script is used to process big data.

[0042] In this application, "Big Data" refers to a collection of data that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insight discovery, and process optimization capabilities. "Batch scripts" refers to a series of programs running on a big data platform used for data extraction and reprocessing (classification, summarization, calculation, etc.), typically including database scripting languages ​​such as SQL and stored procedures. The big data platform is a system for managing big data batch scripts, including program monitoring, alerts, and management of upstream and downstream dependencies between big data batches, ensuring that big data batches execute their steps in the correct order.

[0043] For example, obtaining batch scripts for processing big data means obtaining a series of programs running on a big data platform, and using these programs to extract and reprocess data (classify, summarize, calculate, etc.).

[0044] Step S102: Parse the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code. The tree structure represents the structural information between the code in the target code, and the operators represent the mapping relationship between the code in the target code.

[0045] In this application, a tree structure is a hierarchical nested structure, meaning that the outer and inner layers of a tree structure have similar structures. Therefore, such structures can often be represented recursively, and various tree diagrams in classic data structures are typical tree structures: a tree can be simply represented as a root, left subtree, and right subtree. The left and right subtrees also have their own subtrees. An operator refers to a mapping relationship from one function space to another, and algorithmic relationships include mathematical formula operations on fields, classification, summarization, logical condition judgment, sorting, type conversion, etc. An upstream batch refers to the upper-level batch program that the current batch depends on. The current batch program can only execute when the upstream batch has been executed correctly; otherwise, the current batch program will wait for the upstream batch to finish executing before it can start executing. A large data batch program often has multiple upstream batches; downstream batches are the next level batch programs that the current batch depends on. The downstream batch program can only be executed after the current batch has been executed correctly; the dependency relationship is the sequential execution order of the large data batch program. The dependency relationship indicates the correct execution order of the large data batch tasks, and the large data tasks must be executed in sequence according to a specific upstream and downstream relationship in order to ultimately complete the data program operation task correctly.

[0046] For example, based on parsing the database name and table name, the program parses the table fields, field associations, or filtering conditions used in the most recent code segment, parsing layer by layer, and finally obtains the upstream and downstream tree structure and relational logic operators.

[0047] Step S103: Based on the tree structure and operators, generate the target data table and the target fields in the target data table.

[0048] For example, based on the upstream and downstream tree structure relationship diagram and logical operators obtained above, the upstream data table and corresponding fields are automatically generated in the database system.

[0049] Step S104: Parse the target data table and target fields to generate target test data.

[0050] For example, test data is obtained by parsing the upstream data tables and corresponding fields automatically generated in the database system. Furthermore, the process of generating test data is also known as data creation. Data creation is the method for preparing source values ​​for large-scale data testing; it can be achieved through extraction methods, and the entire process is called data creation.

[0051] Through the above steps S101 to S104, by parsing the target code in the acquired target batch script to obtain the tree structure of the target code and the operators of the target code, a target data table and target fields in the target data table are generated. The target data table and target fields are then parsed to generate target test data, thereby automatically generating test data and improving the efficiency of test data preparation.

[0052] Optionally, in the test data preparation method provided in the embodiments of this application, after parsing the target data table and target fields to generate target test data, the method further includes: determining a target test file, wherein the target test file includes at least two or more target test data; and using the target test file to verify the correctness of the target batch script.

[0053] For example, after confirming the test data, automated testing is performed (importing data, executing batch programs, and generating data results for verification). Specifically, based on two or more test data obtained above, a test file is determined; and the batch script is tested using the test file. That is, after confirming the test file, automated testing is performed, and a series of programs corresponding to the batch script are tested.

[0054] The above solution overcomes the shortcomings of manual test data preparation in existing technologies. It establishes a standardized tree-structure model to automatically generate data creation scripts, which in turn generate the data files needed for batch testing and automate the testing process, while simultaneously verifying the correctness of the batch scheduling logic. Furthermore, after automatically parsing the program logic of the large-scale batch scripts, it automatically forms logical operators by analyzing the upstream and downstream relationships, classification, summarization, mathematical formula calculations, and conditional logic judgments. Based on this, it identifies the tree-structure relationship between upstream and downstream tables, with the upstream node representing the source data and the downstream node representing the final layer of data results from the test object. This automatically generates the required test data files, thereby accelerating test data preparation, improving the accuracy of the data generation and parsing scripts in the testing phase, and increasing the efficiency of test data preparation. This ultimately optimizes the batch automated testing method.

[0055] Optionally, in the test data preparation method provided in the embodiments of this application, before verifying the correctness of the target batch script using the target test file, the method further includes: determining the data volume of each target test data in the target test file and the correlation between every two target test data; obtaining the expected results of running the target batch script based on the data volume and correlation; determining test cases based on the target test data in the target test file; and executing the target batch script based on the test cases to obtain the test results of the target batch script.

[0056] In this embodiment, Figure 2 This is a flowchart of a test batch big data batch script in an embodiment of this application, such as... Figure 2As shown, the process of testing batch big data batch scripts specifically includes: (1) automatically calculating the amount of data stored in the test file and the data correlation degree of the related table through the tool; (2) calculating the expected results of the existing test data based on the obtained amount of data stored in the test file and the data correlation degree of the related table; (3) automatically importing the data stored in the test file into the database through the tool; (4) deploying automated test cases based on the data stored in the test file; (5) automatically executing the big data batch script through the tool based on the test cases to obtain the test results of the big data batch script.

[0057] In summary, by using tools to automatically calculate the amount of data stored in the test files and the correlation degree of the data in the related tables, the calculation results of the test data can be obtained quickly and accurately. Furthermore, based on the data stored in the test files, automated test cases can be deployed, thereby automatically obtaining the test results of the large-scale batch script based on the test cases and the large-scale batch script.

[0058] Optionally, in the test data preparation method provided in the embodiments of this application, verifying the correctness of the target batch script using the target test file includes: determining whether the expected result is the same as the test result; if the expected result is the same as the test result, it means that the target batch script is correct; if the expected result is different from the test result, it means that the target batch script is incorrect.

[0059] For example, by judging the consistency between the calculated expected results and the obtained test results, the generated test results can be automatically verified. Specifically: if the expected results are the same as the test results, it means that the big data batch script is correct; if the expected results are different from the test results, it means that the big data batch script is incorrect.

[0060] The above solutions can improve the verification efficiency of batch scripts for big data.

[0061] Optionally, in the test data preparation method provided in the embodiments of this application, parsing the target data table and target fields to generate target test data includes: determining the operator of the target field; parsing the associated field in the target source table based on the operator of the target field, wherein the target source table is a table associated with the target data table; extracting data from the target source table based on the associated field; and generating target test data based on the data extracted from the source table.

[0062] In this embodiment, Figure 3 This is a flowchart illustrating the automated parsing of upstream data tables and corresponding fields in this application embodiment, such as... Figure 3As shown, the process of automatically parsing upstream data tables and corresponding fields specifically includes: (1) obtaining a piece of SQL (code) from the big data batch script; (2) using a parser to parse the operators of the target table A and the Id and Val fields of table A; (3) using a parser to parse the Id field of the source table B and the source table C, and parse the calculated field Val; (4) using a data file generator to extract the data of the source table B according to the Id field; (5) using a data file generator to extract the data of the source table C according to the Id field based on the data of the source table B, that is, the data in the source table B and the source table C are test data.

[0063] The above solution can automatically parse logical operators to generate a tree structure, and extract test data with business scenarios accordingly, thereby generating test data and improving the efficiency of test data preparation.

[0064] Optionally, in the test data preparation method provided in the embodiments of this application, after parsing the target code in the target batch script to obtain the tree structure and operators corresponding to the target code, the method further includes: generating target script one and target script two based on the tree structure and operators; parsing the target code using target script one to obtain the target data table and operators for the target fields; and generating target test data based on target script two and the target fields.

[0065] For example, the first target script mentioned above could be a parser, and the second target script mentioned above could be a data file generator. For example, such as Figure 3 As shown, the parser can parse the target table A and the operators for the Id and Val fields of table A. The data file generator can extract data from the source table B based on the Id field. The data generator is then used again, referencing the data from source table B, to extract data from source table C based on the Id field. Finally, the data in source tables B and C constitute the test data.

[0066] In summary, by using automated big data batch script parsing to generate test data, the possibility of analysis errors, omissions, and preparation errors during the preparation process is reduced, and the process of manually preparing test data is avoided, thereby improving the accuracy and efficiency of test data preparation.

[0067] Optionally, in the test data preparation method provided in the embodiments of this application, determining the target test file includes: obtaining target test data one and target test data two according to a preset method, wherein target test data one is target test data that conforms to the target batch script running scenario, and target test data two is target test data that does not conform to the target batch script running scenario; and determining the target test file based on target test data one and target test data two.

[0068] For example, based on the parsed tree structure and relational logic operators, test data that meets and does not meet the logical scenarios are generated proportionally from the test environment by extraction and modification, forming test files.

[0069] In summary, the generated test files contain both data that matches and does not match the script execution scenario, thereby expanding the coverage of test data and improving the accuracy of batch testing large datasets.

[0070] Optionally, in the test data preparation method provided in the embodiments of this application, before parsing the target code in the target batch script to obtain the tree structure of the target code and the operator of the target code, the method further includes: determining whether there is unparsed code in the target batch script; if there is unparsed code in the target batch script, then the unparsed code is used as the target code; if there is no unparsed code in the target batch script, then it means that the target batch script has been parsed.

[0071] In this embodiment, Figure 4 This is a flowchart of an automated big data batch parsing script in an embodiment of this application, such as... Figure 4 As shown, the process of automatically parsing large batch scripts includes: (1) inputting any single large batch script; (2) parsing the most recent unparsed SQL code in the script. If there is no unparsed SQL code, the automated parsing process is completed; (3) inputting the unparsed SQL into the parser to obtain the final upstream and downstream tree structure relationship diagram and logical operators; (4) automatically generating the upstream data table and corresponding fields in the database system based on the above upstream and downstream tree structure relationship diagram and logical operators; (5) marking the currently parsed SQL code as parsed and returning to the step of parsing the most recent unparsed SQL code in the script; (6) automated parsing is completed.

[0072] By using the above method to parse big data scripts, logical operators and tree structures can be quickly output.

[0073] In summary, the test data preparation method provided in this application, by acquiring a target batch script (used for processing large amounts of data), parsing the target code in the target batch script to obtain the tree structure and operators of the target code (where the tree structure represents the structural information between codes in the target code, and the operators represent the mapping relationship between codes in the target code), generating a target data table and target fields in the target data table based on the tree structure and operators, and parsing the target data table and target fields to generate target test data, solves the problem of low efficiency in test data preparation caused by manual methods in related technologies. By generating a target data table and target fields based on the tree structure and operators of the target code obtained by parsing the target code in the acquired target batch script, and then parsing the target data table and target fields to generate target test data, test data can be automatically generated, thereby improving the efficiency of test data preparation.

[0074] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0075] This application also provides a test data preparation apparatus. It should be noted that the test data preparation apparatus of this application can be used to execute the test data preparation method provided in this application. The test data preparation apparatus provided in this application is described below.

[0076] Figure 5 This is a schematic diagram of a test data preparation apparatus according to an embodiment of this application. Figure 5 As shown, the device includes: a first acquisition unit 501, a first parsing unit 502, a first generation unit 503, and a second parsing unit 504.

[0077] Specifically, the first acquisition unit 501 is used to acquire a target batch script, wherein the target batch script is used to process big data;

[0078] The first parsing unit 502 is used to parse the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code. The tree structure represents the structural information between the code in the target code, and the operators represent the mapping relationship between the code in the target code.

[0079] The first generation unit 503 is used to generate a target data table and target fields in the target data table based on the tree structure and operators;

[0080] The second parsing unit 504 is used to parse the target data table and target fields to generate target test data.

[0081] In summary, the test data preparation apparatus provided in this application embodiment obtains a target batch script through a first acquisition unit 501, wherein the target batch script is used to process large amounts of data; a first parsing unit 502 parses the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code, wherein the tree structure represents the structural information between the codes in the target code, and the operators represent the mapping relationship between the codes in the target code; a first generation unit 503 generates a target data table and target fields in the target data table based on the tree structure and operators; and a second parsing unit 504 parses the target data table and target fields to generate target test data. This solves the problem of low test data preparation efficiency caused by manual preparation of test data in related technologies. By generating a target data table and target fields based on the tree structure of the target code obtained by parsing the target code in the acquired target batch script, and then parsing the target data table and target fields to generate target test data, test data can be automatically generated, thereby improving the efficiency of test data preparation.

[0082] Optionally, in the test data preparation apparatus provided in the embodiments of this application, the apparatus further includes: a first determining unit, used to determine a target test file after parsing the target data table and target fields to generate target test data, wherein the target test file includes at least two or more target test data; and a first verification unit, used to verify the correctness of the target batch script using the target test file.

[0083] Optionally, in the test data preparation apparatus provided in the embodiments of this application, the apparatus further includes: a second determining unit, configured to determine the data volume of each target test data in the target test file and the correlation between every two target test data before verifying the correctness of the target batch script using the target test file; a first processing unit, configured to obtain the expected result of running the target batch script based on the data volume and correlation; a third determining unit, configured to determine test cases based on the target test data in the target test file; and a second processing unit, configured to execute the target batch script based on the test cases to obtain the test result of the target batch script.

[0084] Optionally, in the test data preparation device provided in the embodiments of this application, the first verification unit includes: a first judgment module, used to judge whether the expected result is the same as the test result; a first processing module, used to indicate that the target batch script is correct if the expected result is the same as the test result; and a second processing module, used to indicate that the target batch script is wrong if the expected result is not the same as the test result.

[0085] Optionally, in the test data preparation apparatus provided in this application embodiment, the second parsing unit includes: a first determining module, used to determine the operator of the target field; a first parsing module, used to parse out the associated field in the target source table based on the operator of the target field, wherein the target source table is a table associated with the target data table; a first extraction module, used to extract data from the target source table based on the associated field; and a first generating module, used to generate target test data based on the data extracted from the source table.

[0086] Optionally, in the test data preparation apparatus provided in the embodiments of this application, the apparatus further includes: a second generation unit, used to generate target script one and target script two based on the tree structure and operators after parsing the target code in the target batch script and obtaining the tree structure and operators corresponding to the target code; a third processing unit, used to parse the target code using target script one to obtain the target data table and the operators of the target fields; and a third generation unit, used to generate target test data based on target script two and the target fields.

[0087] Optionally, in the test data preparation device provided in the embodiments of this application, the first determining unit includes: a first acquiring module, used to acquire target test data one and target test data two according to a preset method, wherein target test data one is target test data that conforms to the target batch script running scenario, and target test data two is target test data that does not conform to the target batch script running scenario; and a second determining module, used to determine a target test file based on target test data one and target test data two.

[0088] Optionally, in the test data preparation device provided in the embodiments of this application, the device further includes: a first judgment unit, used to determine whether there is unparsed code in the target batch script before parsing the target code in the target batch script to obtain the tree structure of the target code and the operator of the target code; a fourth processing unit, used to take the unparsed code as the target code if there is unparsed code in the target batch script; and a fifth processing unit, used to indicate that the parsing of the target batch script is complete if there is no unparsed code in the target batch script.

[0089] The test data preparation device includes a processor and a memory. The first acquisition unit 501, the first parsing unit 502, the first generation unit 503, and the second parsing unit 504 are all stored in the memory as program units. The processor executes the program units stored in the memory to achieve the corresponding functions.

[0090] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the efficiency of test data preparation.

[0091] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0092] This invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a method for preparing test data.

[0093] This invention provides a processor for running a program, wherein the program executes a method for preparing the test data during runtime.

[0094] like Figure 6 As shown, this embodiment of the invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining a target batch script, wherein the target batch script is used to process big data; parsing the target code in the target batch script to obtain a tree structure of the target code and operators of the target code, wherein the tree structure represents the structural information between the codes in the target code, and the operators represent the mapping relationship between the codes in the target code; generating a target data table and target fields in the target data table based on the tree structure and the operators; and parsing the target data table and the target fields to generate target test data.

[0095] When the processor executes the program, it also performs the following steps: after parsing the target data table and the target field to generate target test data, the method further includes: determining a target test file, wherein the target test file includes at least two or more of the target test data; and using the target test file to verify the correctness of the target batch script.

[0096] When the processor executes the program, it also performs the following steps: before verifying the correctness of the target batch script using the target test file, the method further includes: determining the data volume of each target test data in the target test file and the correlation degree between every two target test data; obtaining the expected result of running the target batch script based on the data volume and the correlation degree; determining test cases based on the target test data in the target test file; and executing the target batch script based on the test cases to obtain the test result of the target batch script.

[0097] When the processor executes the program, it also performs the following steps: using the target test file, verifying the correctness of the target batch script includes: determining whether the expected result is the same as the test result; if the expected result is the same as the test result, it means that the target batch script is correct; if the expected result is different from the test result, it means that the target batch script is incorrect.

[0098] When the processor executes the program, it also performs the following steps: parsing the target data table and the target field to generate target test data, including: determining the operator of the target field; parsing the associated field in the target source table based on the operator of the target field, wherein the target source table is a table associated with the target data table; extracting data from the target source table based on the associated field; and generating the target test data based on the data extracted from the source table.

[0099] When the processor executes the program, it also performs the following steps: after parsing the target code in the target batch script to obtain the tree structure and operators corresponding to the target code, the method further includes: generating target script one and target script two based on the tree structure and the operators; parsing the target code using target script one to obtain the target data table and the operators of the target fields; and generating the target test data based on target script two and the target fields.

[0100] When the processor executes the program, it also performs the following steps: Determining the target test file includes: obtaining target test data one and target test data two according to a preset method, wherein target test data one is target test data that conforms to the target batch script running scenario, and target test data two is target test data that does not conform to the target batch script running scenario; determining the target test file based on target test data one and target test data two.

[0101] When the processor executes the program, it also performs the following steps: before parsing the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code, the method further includes: determining whether there is unparsed code in the target batch script; if there is unparsed code in the target batch script, then the unparsed code is used as the target code; if there is no unparsed code in the target batch script, then the parsing of the target batch script is complete. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0102] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: obtaining a target batch script, wherein the target batch script is used to process large data; parsing the target code in the target batch script to obtain a tree structure of the target code and operators of the target code, wherein the tree structure represents the structural information between codes in the target code, and the operators represent the mapping relationship between codes in the target code; generating a target data table and target fields in the target data table based on the tree structure and the operators; and parsing the target data table and the target fields to generate target test data.

[0103] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: after parsing the target data table and the target field to generate target test data, the method further includes: determining a target test file, wherein the target test file includes at least two or more of the target test data; and using the target test file to verify the correctness of the target batch script.

[0104] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: before verifying the correctness of the target batch script using the target test file, the method further includes: determining the data volume of each target test data in the target test file and the correlation degree between every two target test data; obtaining the expected result of running the target batch script based on the data volume and the correlation degree; determining test cases based on the target test data in the target test file; and executing the target batch script based on the test cases to obtain the test result of the target batch script.

[0105] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: using the target test file, verifying the correctness of the target batch script includes: determining whether the expected result is the same as the test result; if the expected result is the same as the test result, it means that the target batch script is correct; if the expected result is different from the test result, it means that the target batch script is incorrect.

[0106] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: parsing the target data table and the target field to generate target test data, including: determining the operator of the target field; parsing the associated field in the target source table based on the operator of the target field, wherein the target source table is a table associated with the target data table; extracting data from the target source table based on the associated field; and generating the target test data based on the data extracted from the source table.

[0107] When executed on a data processing device, it is also suitable to execute an initialization program with the following steps: after parsing the target code in the target batch script to obtain the tree structure and operators corresponding to the target code, the method further includes: generating target script one and target script two based on the tree structure and the operators; parsing the target code using target script one to obtain the target data table and the operators of the target fields; and generating the target test data based on target script two and the target fields.

[0108] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: determining the target test file includes: obtaining target test data one and target test data two according to a preset method, wherein target test data one is target test data that conforms to the target batch script running scenario, and target test data two is target test data that does not conform to the target batch script running scenario; determining the target test file based on target test data one and target test data two.

[0109] When executed on a data processing device, it is also suitable to execute an initialization program with the following method steps: before parsing the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code, the method further includes: determining whether there is unparsed code in the target batch script; if there is unparsed code in the target batch script, then the unparsed code is used as the target code; if there is no unparsed code in the target batch script, then the parsing of the target batch script is complete.

[0110] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0111] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for preparing test data, characterized in that, include: Obtain the target batch script, wherein the target batch script is used to process big data; The target code in the target batch script is parsed to obtain the tree structure of the target code and the operators of the target code. The tree structure represents the structural information between the code in the target code, and the operators represent the mapping relationship between the code in the target code. Based on parsing the library name and table name, the program parses the table fields, field associations or filtering conditions used in the most recent code segment, and parses layer by layer to obtain the upstream and downstream tree structure and relational logic operators. Based on the tree structure and the operator, a target data table and target fields in the target data table are generated; The target data table and the target fields are parsed to generate target test data; A target test file is determined, wherein the target test file includes at least two or more target test data; the data volume of each target test data in the target test file and the correlation between each pair of target test data are determined; based on the data volume and the correlation, the expected result of running the target batch script is obtained; based on the target test data in the target test file, test cases are determined; based on the test cases, the target batch script is executed to obtain the test result of the target batch script.

2. The method according to claim 1, characterized in that, After parsing the target data table and the target fields to generate target test data, the method further includes: The correctness of the target batch script is verified using the target test file.

3. The method according to claim 1, characterized in that, Verifying the correctness of the target batch script using the target test file includes: Determine whether the expected result is the same as the test result; If the expected result is the same as the test result, it means that the target batch script is correct; If the expected result is different from the test result, it means that the target batch script is faulty.

4. The method according to claim 1, characterized in that, Parsing the target data table and the target fields to generate target test data includes: Determine the operator for the target field; Based on the operator of the target field, the associated fields in the target source table are parsed out, wherein the target source table is a table associated with the target data table; Based on the associated fields, data is extracted from the target source table; The target test data is generated based on the data extracted from the source table.

5. The method according to claim 1, characterized in that, After parsing the target code in the target batch script to obtain the tree structure and operators corresponding to the target code, the method further includes: Based on the tree structure and the operator, target script one and target script two are generated; The target code is parsed using the target script to obtain the target data table and the operators for the target fields; The target test data is generated based on the target script and the target field.

6. The method according to claim 2, characterized in that, The target test file includes: According to a preset method, target test data one and target test data two are obtained, wherein target test data one is target test data that conforms to the target batch script running scenario, and target test data two is target test data that does not conform to the target batch script running scenario; The target test file is determined based on the target test data one and the target test data two.

7. The method according to claim 1, characterized in that, Before parsing the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code, the method further includes: Determine whether there is unparsed code in the target batch script; If the unparsed code exists in the target batch script, then the unparsed code will be used as the target code; If the unparsed code does not exist in the target batch script, it means that the target batch script has been parsed successfully.

8. A test data preparation apparatus, characterized in that, include: The first acquisition unit is used to acquire a target batch script, wherein the target batch script is used to process big data; The first parsing unit is used to parse the target code in the target batch script to obtain the tree structure of the target code and the operators of the target code. The tree structure represents the structural information between the code in the target code, and the operators represent the mapping relationship between the code in the target code. Based on parsing the library name and table name, the unit parses the table fields, field associations or filtering conditions used in the most recent part of the program code layer by layer to obtain the upstream and downstream tree structure and relational logic operators. The first generation unit is used to generate a target data table and target fields in the target data table based on the tree structure and the operator; The second parsing unit is used to parse the target data table and the target fields to generate target test data; A first determining unit is configured to determine a target test file after parsing the target data table and the target fields to generate target test data, wherein the target test file includes at least two or more target test data; a second determining unit is configured to determine the data volume of each target test data in the target test file and the correlation degree between every two target test data; a first processing unit is configured to obtain the expected result of running the target batch script based on the data volume and the correlation degree; a third determining unit is configured to determine test cases based on the target test data in the target test file; and a second processing unit is configured to execute the target batch script based on the test cases to obtain the test result of the target batch script.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program, wherein the program executes the test data preparation method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method for preparing test data as described in any one of claims 1 to 7.

Images