A test data preparation method and related device

Abstract

The test data preparation method and the related device provided by the present disclosure can be applied to the field of big data or the field of finance. According to different test requirements of a product to be tested, based on a target table and a target date partition, by using a digital map technology and a test data resource pool, field-level data query is realized, so that test data corresponding to a source table having a data blood relationship with the target table can be quickly obtained, the test data acquisition time is saved, the utilization rate of the test data is improved, and the test efficiency of the product is improved.

Description

Technical Field

[0001] This disclosure relates to the field of big data, and in particular to a test data preparation method and related equipment. Background Technology

[0002] Currently, big data products are characterized by diversified data sources and complex data processing chains. Testing these products requires comprehensive and accurate test data to verify logical errors in the data processing flow.

[0003] However, given the complex data processing chain and the massive number of source tables involved, simply relying on extracting test data and importing it into the test environment for testing can easily lead to the extracted test data no longer meeting the product testing needs at the current time point due to the long test data extraction cycle. This reduces the utilization rate of test data and affects the product testing efficiency.

[0004] Therefore, how to improve the utilization rate of test data has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a test data preparation method and related equipment to overcome or at least partially solve the above problems, the technical solution of which is as follows:

[0006] A test data preparation method, comprising:

[0007] Obtain the target table and target date partitions for the product to be tested;

[0008] Using the data map corresponding to the product under test, query the source tables that have a data lineage relationship with the target table;

[0009] In the pre-built test data resource pool, query whether there is test data corresponding to the source table under the target date partition. If not, query whether there is test data corresponding to the source table under other date partitions besides the target date partition.

[0010] If no test data corresponding to the source table exists in any date partition other than the target date partition, obtain each field in the source table;

[0011] For any of the fields: using the data map, query whether a data table for the field exists. If so, obtain the first field data corresponding to the field in the data table, and copy the first field data to the position corresponding to the field in the source table under the target date partition.

[0012] Optionally, the method further includes:

[0013] If no data table containing the field is found, the second field data is generated automatically according to the data generation pattern that matches the field.

[0014] The second field data is transferred to the location corresponding to the field in the source table under the target date partition.

[0015] Optionally, the step of automatically generating data according to a data generation pattern matching the field to generate the second field data includes:

[0016] Determine whether the field has a preset field input condition. If so, generate the second field data according to the preset field input condition. If not, generate the second field data based on the field type and field length of the field.

[0017] Optionally, the preset field input conditions include field value ranges and / or regular expressions.

[0018] Optionally, the method further includes:

[0019] If test data corresponding to the source table exists in a date partition other than the target date partition, copy the test data to the location corresponding to the source table in the target date partition.

[0020] Optionally, before obtaining the target table and target date partitions of the product to be tested, the method further includes:

[0021] Obtain the data resource information of the product under test, wherein the data resource information includes field information, storage information, description information and data statistics information associated with the product under test;

[0022] Using the data resource information, a data map corresponding to the product to be tested is constructed.

[0023] Optionally, the test data is production data associated with the product under test and anonymized.

[0024] A test data preparation apparatus includes: a first acquisition unit, a first query unit, a second query unit, a third query unit, a second acquisition unit, a fourth query unit, a third acquisition unit, and a field data copying unit.

[0025] The first obtaining unit is used to obtain the target table and target date partition of the product to be tested;

[0026] The first query unit is used to query the source table that has a data lineage relationship with the target table using the data map corresponding to the product under test;

[0027] The second query unit is used to query whether there is test data corresponding to the source table under the target date partition in the pre-built test data resource pool; if not, the third query unit is triggered.

[0028] The third query unit is used to query whether there is test data corresponding to the source table under other date partitions besides the target date partition;

[0029] The second obtaining unit is used to obtain each field in the source table when there is no test data corresponding to the source table in other date partitions besides the target date partition;

[0030] The fourth query unit is used to query whether a data table for any of the fields exists using the data map; if so, the third obtaining unit is triggered.

[0031] The third obtaining unit is used to obtain the first field data corresponding to the field in the data table;

[0032] The field data copying unit is used to copy the first field data to the position corresponding to the field in the source table under the target date partition.

[0033] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the test data preparation method described in any of the preceding claims.

[0034] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the test data preparation method described above.

[0035] Using the above technical solution, the test data preparation method and related equipment provided in this disclosure can be applied to the big data field or the financial field. This disclosure can obtain the target table and target date partition of the product to be tested; using the data map corresponding to the product to be tested, query the source table that has a data lineage relationship with the target table; in the pre-built test data resource pool, query whether there is test data corresponding to the source table under the target date partition; if not, query whether there is test data corresponding to the source table under other date partitions besides the target date partition; if there is no test data corresponding to the source table under other date partitions besides the target date partition, obtain each field in the source table; for any field: using the data map, query whether there is a data table for that field; if so, obtain the first field data corresponding to that field in the data table, and copy the first field data to the position corresponding to that field in the source table under the target date partition. This disclosure addresses the different testing needs of the product under test. Based on the target table and target date partitioning, it utilizes digital map technology and a test data resource pool to achieve field-level data querying. This enables the rapid acquisition of test data from source tables that have a data lineage with the target table, saving test data acquisition time, improving test data utilization, and thus enhancing product testing efficiency.

[0036] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0037] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0038] Figure 1 A flowchart illustrating one implementation of the test data preparation method provided in this disclosure is shown.

[0039] Figure 2 A flowchart illustrating another implementation of the test data preparation method provided in this disclosure is shown.

[0040] Figure 3 A flowchart illustrating another implementation of the test data preparation method provided in this disclosure is shown.

[0041] Figure 4A flowchart illustrating another implementation of the test data preparation method provided in this disclosure is shown.

[0042] Figure 5 A flowchart illustrating another implementation of the test data preparation method provided in this disclosure is shown.

[0043] Figure 6 A logic block diagram of a test data preparation method provided in an embodiment of this disclosure is shown;

[0044] Figure 7 A schematic diagram of the structure of the test data preparation apparatus provided in an embodiment of this disclosure is shown;

[0045] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] To accurately identify logical errors during data processing, comprehensive and accurate test data preparation is crucial. However, the complex processing flow and the massive amount of source tables involved make test data preparation particularly challenging. Test data preparation primarily relies on extracting anonymized production data. In practice, the extracted production data is anonymized and imported into the test environment. Test data acquisition is mainly based on source tables identified through product project requirements analysis. Anonymization scripts are developed to perform data anonymization during production data extraction, ensuring data security.

[0048] In actual production data extraction, due to the sheer volume of data generated during production, it is typically only possible to extract data within a timeframe of a few days and / or a geographical area of ​​a few provinces. Because the data volume and range for extraction are so limited, it cannot fully meet actual testing needs.

[0049] Before any testing phase of a real product project begins, significant manpower and time costs are typically invested. Because there's a considerable time lag between submitting a production data request and the data arriving, and data requirements may change during this period as project requirements analysis and development progress, resubmitting data requests based on these changes makes project timelines unpredictable. Furthermore, due to limited testing environment resources, the environment usually needs to be cleaned up after the previous project ends before importing test data for the next project. Additionally, parallel phases often exist between projects, and different testers responsible for different projects may submit duplicate test data requests, resulting in a waste of test data resources.

[0050] To maximize the utilization of test data resources, the test data preparation method provided in this disclosure utilizes digital map technology and a test data resource pool to perform field-level data analysis and querying of test data, which can efficiently obtain test data and improve the utilization rate of test data.

[0051] like Figure 1 The diagram shows a flowchart of one embodiment of the test data preparation method provided in this disclosure. The test data preparation method may include:

[0052] S100: Obtain the target table and target date partitions for the product to be tested.

[0053] The product to be tested can be a banking product, such as bank deposit products, bank loan products, and bank insurance products. The target table is a data table built to target the product under test, consisting of multiple fields. The target date partition is a region divided according to the data date; date partitioning facilitates recording the test data used in this test.

[0054] This disclosure provides an embodiment that allows users to input the target table and target date partitions required for the test of the product under test, providing a basis for the analysis and preparation of subsequent test data.

[0055] S110. Using the data map corresponding to the product under test, query the source table that has a data lineage relationship with the target table.

[0056] A data map is a graphical data asset management tool. This disclosure allows for the pre-construction of a data map corresponding to the product under test, establishing data lineage relationships between data points. This enables rapid searching and locating within the data map, facilitating efficient querying and retrieval of data assets related to the product under test. Data lineage relationships refer to the relationships formed between data points during their generation, processing, flow, and eventual disappearance—essentially, the data generation chain. Lineage analysis of data lineage relationships enables traceability of data fusion processing.

[0057] This embodiment of the disclosure can use the target table as a condition for lineage analysis to obtain source tables that have a data lineage relationship with the target table from the data map. The source table can also be simply referred to as the source table. There must be at least one pair of field mapping relationships between the target table and any source table.

[0058] S120. In the pre-built test data resource pool, query whether there is test data corresponding to the source table under the target date partition. If not, proceed to step S130.

[0059] The test data resource pool stores multiple test data sets. Optionally, the test data consists of anonymized production data associated with the product under test. This embodiment of the disclosure allows for field-level analysis of existing anonymized production data to construct the test data resource pool, achieving field-level test data asset storage and enabling test data reusability. The test data resource pool can be partitioned and stored according to data date to facilitate the retrieval of required test data.

[0060] This disclosure embodiment utilizes existing anonymized production data to the maximum extent at the field level by constructing a test data resource pool, which helps to cover test data preparation work under various testing requirements.

[0061] It is understandable that by querying the data map to find the various source tables that have a data lineage with the target table, this embodiment of the disclosure can query the test data resource pool in a sequential or parallel manner whether there is test data corresponding to each source table under the target date partition.

[0062] Optionally, in this embodiment of the disclosure, test data for each source table can be prepared sequentially. For example, suppose the source tables corresponding to the target table include tables A1 to An. In this embodiment of the disclosure, the test data resource pool can be searched to see if test data corresponding to table A1 exists under the target date partition. If it exists, the test data for table A1 is confirmed to be prepared. The search continues to be performed in the test data resource pool to see if test data corresponding to table A2 exists under the target date partition, and so on, until the test data corresponding to table An is prepared.

[0063] S130. Query whether test data corresponding to the source table exists in other date partitions besides the target date partition. If no test data corresponding to the source table exists in other date partitions besides the target date partition, proceed to step S140.

[0064] In this embodiment of the disclosure, if the test data corresponding to the source table cannot be found in the target date partition of the test data resource pool, the test data corresponding to the source table can be queried in other date partitions of the test data resource pool besides the target date partition.

[0065] S140. Obtain each field from the source table.

[0066] S150. For any field: Use the data map to query whether the data table for that field exists. If it does, proceed to step S160.

[0067] Specifically, this embodiment of the disclosure can utilize a data map to perform field-level analysis on the source table, obtain each field in the source table, and analyze the field-level relationships between each field and other data tables. For example, suppose the fields in the source table include: field a1, field a2, ..., field an, where field a1 is the same as field b1 in data table B1, and field a2 is the same as field c2 in data table C1. This embodiment of the disclosure queries data tables that have the same fields as the source table so that the field data of that field can be obtained subsequently through that data table.

[0068] S160. Obtain the data of the first field corresponding to this field in the data table.

[0069] The field data consists of the test data corresponding to the field.

[0070] S170. Copy the data of the first field to the location corresponding to that field in the source table under the target date partition.

[0071] Optionally, in this embodiment of the present disclosure, the first field data corresponding to the field in the data table can be queried and obtained from the test data resource pool, and the first field data can be copied to the position corresponding to the field in the source table under the target date partition in the test data resource pool, thereby preparing the test data for the field in the source table.

[0072] Optionally, for data tables that do not store test data in the test data resource pool, this embodiment of the present disclosure can query the source system to which the data table belongs, obtain the first field data corresponding to the field in the data table through the interface provided by the source system, and copy the first field data to the position corresponding to the field in the source table under the target date partition in the test data resource pool, thereby preparing the test data for the field in the source table.

[0073] To facilitate understanding, an example is provided here: In most source systems within a banking system, the data tables have a "Customer Number" field. If any source table containing the "Customer Number" field does not have corresponding test data in the test data resource pool, the data map analysis will match the data table containing the "Customer Number" field. The first field data corresponding to this field in the data table will be copied to the location corresponding to this field in the target date partition of the test data resource pool. This prepares the test data for the "Customer Number" field in the source table so that the first field data can be used for this field in the source table.

[0074] This disclosure provides a test data preparation method applicable to the big data or financial fields. This method allows obtaining the target table and target date partition of the product to be tested; using a data map corresponding to the product to be tested, querying the source table that has a data lineage with the target table; in a pre-built test data resource pool, querying whether test data corresponding to the source table exists under the target date partition; if not, querying whether test data corresponding to the source table exists under other date partitions besides the target date partition; if no test data corresponding to the source table exists under other date partitions besides the target date partition, obtaining each field in the source table; for any field: using the data map, querying whether a data table exists for that field; if so, obtaining the first field data corresponding to that field in the data table, and copying the first field data to the position corresponding to that field in the source table under the target date partition. This disclosure addresses the different testing needs of the product under test. Based on the target table and target date partitioning, it utilizes digital map technology and a test data resource pool to achieve field-level data querying. This enables the rapid acquisition of test data from source tables that have a data lineage with the target table, saving test data acquisition time, improving test data utilization, and thus enhancing product testing efficiency.

[0075] During actual project testing, there may be situations where the source table lacks corresponding test data (e.g., application data is missing or the applied production data is empty). Due to the complexity of data links and processing rules, it is difficult to construct the corresponding test data in a timely manner by relying on manual data generation methods. It is also impossible to guarantee the diversity and relevance of test data. As a result, if test data is missing for individual fields in the same source table, the test data obtained from other fields in the same source table will become outdated, resulting in a waste of test data resources.

[0076] In this embodiment of the disclosure, for fields in the source table where data cannot be obtained by querying the data table, an automated data generation method can be used to complete the field data, thereby preparing the test data for the source table.

[0077] Optional, based on Figure 1 The method shown is as follows: Figure 2 The diagram shows another implementation of the test data preparation method provided in this disclosure. In this disclosure, step S180 can also be executed if the data table containing the field cannot be found.

[0078] S180. Generate data automatically according to the data generation pattern that matches this field, and generate the data for the second field.

[0079] Optionally, the number generation mode can include random number generation and custom number generation.

[0080] Random number generation is used for fields with low requirements for data quality. For example, fields that will not undergo further processing, have no downstream data use, or are only used for connectivity testing. Embodiments of this disclosure can use a random number generation pattern to randomly generate second field data for a field according to its field type and length.

[0081] Custom-defined data generation is used for fields with high requirements for their data. For example, fields with limitations on their business meaning. This embodiment of the disclosure can use a custom-defined data generation mode to generate a second field of data for that field based on the field input conditions.

[0082] S190. Transfer the second field data to the location corresponding to that field in the source table under the target date partition.

[0083] In this embodiment, the automatically generated second field data is transmitted to the location corresponding to the field in the source table under the target date partition in the test data resource pool, thereby preparing the test data for that field.

[0084] This embodiment of the disclosure addresses fields in the source table where data cannot be obtained through querying the data table. It employs automated data generation methods to complete the field data, which helps to prepare test data for all fields in the source table. This avoids the invalidation of already obtained test data due to missing field data, thereby effectively preventing the waste of test data resources and improving the utilization rate of test data.

[0085] This disclosure embodiment can determine the data generation pattern matched by a field by identifying whether a field has preset field input conditions. Optionally, based on Figure 2 The method shown is as follows: Figure 3 As shown, this is a flowchart illustrating another implementation of the test data preparation method provided in this disclosure. Step S180 may include:

[0086] S181. Determine whether the field has a preset field input condition. If yes, execute S182; otherwise, execute S183.

[0087] S182. Generate the second field data according to the preset field input conditions.

[0088] S183. Based on the field type and field length of this field, generate the data for the second field.

[0089] Specifically, in embodiments of this disclosure, when a field is found to have preset field input conditions, a custom-generated number is applied to that field to generate second field data within the range of those preset field input conditions. Alternatively, in embodiments of this disclosure, when a field is found to lack preset field input conditions, a random-generated number is applied to that field to randomly generate second field data that conforms to the field type and field length of that field.

[0090] This embodiment of the disclosure can automatically distinguish the level of field data generation requirements by recognizing preset field input conditions, and then apply the matching data generation mode to complete the automated data generation. It can quickly and accurately generate the field data corresponding to the field, which is conducive to improving the utilization rate of test data and efficiently completing the test data preparation work.

[0091] Optionally, preset field input conditions include field value ranges and / or regular expressions.

[0092] Optional, based on Figure 1 The method shown is as follows: Figure 4 The diagram shows a flowchart of another implementation of the test data preparation method provided in this disclosure. This disclosure can also proceed to step S200 if test data corresponding to the source table exists in date partitions other than the target date partition.

[0093] S200. Copy the test data to the location corresponding to the source table under the target date partition.

[0094] This embodiment of the disclosure can quickly prepare the test data of the source table by copying test data from other date partitions to the target date partition, thereby efficiently completing the test data preparation work.

[0095] Optionally, in practice, this embodiment of the disclosure may find test data corresponding to the source table in multiple other date partitions besides the target date partition. This embodiment of the disclosure can select the test data from other date partitions adjacent to the target date partition and copy the test data to the location corresponding to the source table in the target date partition. By selecting test data from other date partitions adjacent to the target date partition, this embodiment of the disclosure can prepare the latest test data for the source table in the target date partition, thereby ensuring the real-time nature and effectiveness of subsequent product testing results.

[0096] To accurately and quickly prepare test data, it is necessary to construct a data map capable of analyzing the meaning of field-level data and identifying the same field in different tables. This embodiment of the disclosure, by constructing a data map corresponding to the product under test, can analyze all links in the data processing and flow process involved in the product under test according to requirements, including all data resource information and data lineage relationships along the links.

[0097] Optional, based on Figure 1 The method shown is as follows: Figure 5 As shown, this is a flowchart illustrating another embodiment of the test data preparation method provided in this disclosure. Before step S100, the method further includes:

[0098] S010. Obtain the data resource information of the product to be tested, wherein the data resource information includes field information, storage information, description information and data statistics information associated with the product to be tested.

[0099] The information includes: field information (field name, field type, and field description); storage information (storage engine, storage period, and storage path); description information (business information, table introduction, responsible person, business tags, and permission tags); and data statistics information (partitions in the test environment, table size, latest partition, and most recent update time).

[0100] S020. Using data resource information, construct a data map corresponding to the product to be tested.

[0101] Among them, the data map construction technology is an existing technology. The embodiments of this disclosure can utilize the data resource information of the product under test and use the data map construction technology to construct a data map corresponding to the product under test.

[0102] Understandably, once the data map corresponding to the product under test is constructed, if there are no changes to the data scope or data structure of the product under test, there is no need to modify the data map, and the test data preparation work can be completed directly using the data map.

[0103] This disclosure embodiment constructs a data map corresponding to the product under test, which enables efficient querying of the required source tables and corresponding fields of test data. This facilitates the rapid and comprehensive completion of test data preparation, thereby saving a significant amount of human and time resources and helping to improve the utilization rate of test data and the testing efficiency of the product.

[0104] To facilitate understanding of the overall logical process of the test data preparation method provided in the embodiments of this disclosure, this document combines... Figure 6 Explanation: Figure 6 The diagram illustrates a logical flow of a test data preparation method. After inputting the target table and target date partition, all source tables corresponding to the target table are retrieved based on data lineage. The system then queries whether test data exists in any source table under the target date partition. If so, the preparation of test data for that source table is confirmed. The system then queries another source table to see if test data exists under the target date partition. For source tables without test data under the target date partition, the system checks whether test data exists in other date partitions. If so, the test data is copied to the target date partition, confirming the preparation of test data for that source table. If not, the system uses a data map to retrieve the fields of the source table, analyzes the relationships between each field and other data tables, and copies the field data from the related data tables to the target date partition. For fields in the source tables for which no data is available, automatic data generation completes the preparation of test data for that source table. This process continues until test data preparation for all source tables is complete.

[0105] The test data preparation method provided in this disclosure is based on the actual testing process of big data products. It addresses the issue that test data is difficult to fully cover testing needs, resulting in low test data utilization. By using digital map technology and a test data resource pool, it achieves field-level data analysis, maximizing the value of test data. Furthermore, by automatically generating data to supplement field data, it helps to efficiently complete the test data preparation work and improve the utilization rate of test data.

[0106] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0107] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0108] Corresponding to the above method embodiments, this disclosure also provides a test data preparation apparatus, the structure of which is as follows: Figure 7 As shown, it may include: a first obtaining unit 01, a first query unit 02, a second query unit 03, a third query unit 04, a second obtaining unit 05, a fourth query unit 06, a third obtaining unit 07, and a field data copying unit 08.

[0109] The first acquisition unit 01 is used to obtain the target table and target date partition of the product to be tested.

[0110] The first query unit 02 is used to query the source table that has a data lineage relationship with the target table by using the data map corresponding to the product under test.

[0111] The second query unit 03 is used to query whether there is test data corresponding to the source table under the target date partition in the pre-built test data resource pool. If not, the third query unit 04 is triggered.

[0112] The third query unit 04 is used to query whether test data corresponding to the source table exists in other date partitions besides the target date partition.

[0113] The second obtaining unit 05 is used to obtain each field in the source table when there is no test data corresponding to the source table in other date partitions besides the target date partition.

[0114] The fourth query unit 06 is used to query whether a data table exists for any field using a data map. If it does, the third acquisition unit 07 is triggered.

[0115] The third obtaining unit 07 is used to obtain the first field data corresponding to the field in the data table.

[0116] Field data copying unit 08 is used to copy the data of the first field to the location corresponding to that field in the source table under the target date partition.

[0117] Optionally, the test data preparation device may also include a data generation unit and a field data transmission unit.

[0118] The data generation unit is used to automatically generate data for the second field by following a data generation pattern that matches the field when no data table for the field can be found.

[0119] The field data transfer unit is used to transfer the second field data to the location corresponding to that field in the source table under the target date partition.

[0120] Optionally, the number generation unit may include: a condition determination subunit, a custom number generation subunit, and a random number generation subunit.

[0121] The condition determination subunit is used to determine whether the field has a preset field input condition. If it does, the custom data generation subunit is triggered; otherwise, the random data generation subunit is triggered.

[0122] A custom data generation subunit is used to generate second field data according to preset field input conditions.

[0123] The random number generation subunit is used to generate second field data based on the field type and field length of the given field.

[0124] Optionally, preset field input conditions include field value ranges and / or regular expressions.

[0125] Optionally, the test data preparation apparatus may further include a test data copying unit.

[0126] The test data copying unit is used to copy the test data to the location corresponding to the source table in the target date partition, provided that test data corresponding to the source table exists in a date partition other than the target date partition.

[0127] Optionally, the test data preparation device may also include: a data resource information acquisition unit and a data map construction unit.

[0128] The data resource information acquisition unit is used to acquire the data resource information of the product under test, including field information, storage information, description information and data statistics information associated with the product under test.

[0129] The data map construction unit is used to construct a data map corresponding to the product under test using data resource information.

[0130] Optionally, the test data can be production data that is associated with the product under test and has been de-identified.

[0131] This disclosure provides a test data preparation apparatus applicable to the big data or financial fields. This apparatus can obtain the target table and target date partition of the product to be tested; using a data map corresponding to the product to be tested, it queries the source table that has a data lineage with the target table; in a pre-built test data resource pool, it queries whether test data corresponding to the source table exists under the target date partition; if not, it queries whether test data corresponding to the source table exists under other date partitions besides the target date partition; if no test data corresponding to the source table exists under other date partitions besides the target date partition, it obtains each field in the source table; for any field: using the data map, it queries whether a data table exists for that field; if so, it obtains the first field data corresponding to that field in the data table and copies the first field data to the position corresponding to that field in the source table under the target date partition. This disclosure addresses the different testing needs of the product under test. Based on the target table and target date partitioning, it utilizes digital map technology and a test data resource pool to achieve field-level data querying. This enables the rapid acquisition of test data from source tables that have a data lineage with the target table, saving test data acquisition time, improving test data utilization, and thus enhancing product testing efficiency.

[0132] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0133] The test data preparation device includes a processor and a memory. The first acquisition unit 01, the first query unit 02, the second query unit 03, the third query unit 04, the second acquisition unit 05, the fourth query unit 06, the third acquisition unit 07, and the field data copying unit 08 are all stored in the memory as program units. The processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0134] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, based on the target table and target date partitioning, and utilizing digital mapping technology and test data resource pools, field-level data queries can be achieved. This allows for the rapid retrieval of test data from source tables that have a data lineage with the target table, saving test data acquisition time and improving test data utilization and product testing efficiency.

[0135] This disclosure provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the test data preparation method.

[0136] This disclosure provides a processor for running a program, wherein the program executes the test data preparation method during runtime.

[0137] like Figure 8 As shown, this disclosure provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned test data preparation method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0138] This disclosure also provides a computer program product that, when executed on an electronic device, is adapted to perform a program that initializes a test data preparation method step.

[0139] It should be noted that the test data preparation method and related equipment provided in this disclosure can be used in the fields of big data or finance. The above is merely an example and does not limit the application areas of the test data preparation method and related equipment provided in this disclosure.

[0140] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, electronic devices (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0142] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0143] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0144] In the description of this disclosure, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.

[0145] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0146] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A method for preparing test data, characterized in that, include: Obtain the target table and target date partitions for the product to be tested; Using the data map corresponding to the product under test, query the source tables that have a data lineage relationship with the target table; In the pre-built test data resource pool, query whether there is test data corresponding to the source table under the target date partition. If not, query whether there is test data corresponding to the source table under other date partitions besides the target date partition. If no test data corresponding to the source table exists in any date partition other than the target date partition, obtain each field in the source table; For any of the fields: using the data map, query whether a data table for the field exists. If so, obtain the first field data corresponding to the field in the data table, and copy the first field data to the position corresponding to the field in the source table under the target date partition.

2. The method according to claim 1, characterized in that, Also includes: If no data table containing the field is found, the second field data is generated automatically according to the data generation pattern that matches the field. The second field data is transferred to the location corresponding to the field in the source table under the target date partition.

3. The method according to claim 2, characterized in that, The step of automatically generating data according to a data generation pattern matching the field to generate the second field data includes: Determine whether the field has a preset field input condition. If so, generate the second field data according to the preset field input condition. If not, generate the second field data based on the field type and field length of the field.

4. The method according to claim 3, characterized in that, The preset field input conditions include field value ranges and / or regular expressions.

5. The method according to claim 1, characterized in that, Also includes: If test data corresponding to the source table exists in a date partition other than the target date partition, copy the test data to the location corresponding to the source table in the target date partition.

6. The method according to claim 1, characterized in that, Before obtaining the target table and target date partitions of the product to be tested, the method further includes: Obtain the data resource information of the product under test, wherein the data resource information includes field information, storage information, description information and data statistics information associated with the product under test; Using the data resource information, a data map corresponding to the product to be tested is constructed.

7. The method according to claim 1, characterized in that, The test data is production data that has been anonymized and associated with the product under test.

8. A test data preparation apparatus, characterized in that, include: The system comprises a first acquisition unit, a first query unit, a second query unit, a third query unit, a second acquisition unit, a fourth query unit, a third acquisition unit, and a field data copying unit. The first obtaining unit is used to obtain the target table and target date partition of the product to be tested; The first query unit is used to query the source table that has a data lineage relationship with the target table using the data map corresponding to the product under test; The second query unit is used to query whether there is test data corresponding to the source table under the target date partition in the pre-built test data resource pool; if not, the third query unit is triggered. The third query unit is used to query whether there is test data corresponding to the source table under other date partitions besides the target date partition; The second obtaining unit is used to obtain each field in the source table when there is no test data corresponding to the source table in other date partitions besides the target date partition; The fourth query unit is used to query whether a data table for any of the fields exists using the data map; if so, the third obtaining unit is triggered. The third obtaining unit is used to obtain the first field data corresponding to the field in the data table; The field data copying unit is used to copy the first field data to the position corresponding to the field in the source table under the target date partition.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the test data preparation method as described in any one of claims 1 to 7.

10. An electronic device, the electronic device comprising at least one processor, and at least one memory and a bus connected to the processor; wherein, The processor and the memory communicate with each other via the bus; The processor is used to invoke program instructions in the memory to execute the test data preparation method as described in any one of claims 1 to 7.

Images