A method used for mass generation of relational simulation data

Abstract

The invention discloses a method used for mass generation of relational simulation data. The method comprises: at run time, based on the partition field definitions in the data template, generating asequence composed of one or more partition fields, starting from the first partition set of the sequence, generating a local file through the simulator and logical rules, after the local file is imported into the specified location of Hive warehouse, entering the second partition set to start simulation, and generating the second local file at this time. The second local file references the firstlocal file line by line, translates the same key fields, and changes the remaining fields in accordance with the logical rules.

Description

technical field [0001] The invention relates to the technical field of data simulation, in particular to a method for generating relational simulation data in large batches. Background technique [0002] In the prior art, the amount of manually written test data is often small, and there is a gap of several orders of magnitude from the actual data amount of the production environment; the manually written test data is often found to have dirty data after actual testing, that is, it does not meet the constraints or is incomplete data. Manually adjusting these dirty data is costly but has little effect. According to the experience of launching big data projects in the past, after running in the production environment, problems that cannot be reproduced in the test environment often occur. For example, there may be very large datasets with unbalanced distribution that cause computing tasks to fail during shuffle, and it is difficult to detect such hidden problems early on in t...

AI Technical Summary

Problems solved by technology

[0002] In the prior art, the amount of manually written test data is often small, and there is a gap of several orders of magnitude from the actual data amount of the production environment; the manually written test data is often found to have dirty data after actual testing, that is, it does not meet the constraints or is incomplete data

However, by manually adjusting these dirty data, the cost is huge but the effect is very little; according to the experience of the big data project in the past, after running in the production environment, there are often problems that cannot be reproduced in the test environment

For example, there may be very large data sets with uneven distribution that cause computing tasks to fail during shuffle, and it is difficult for the test environment to detect such hidden problems early because such a large number of data sets cannot be manually generated.

Images