A data generation method and system for massive data relationship mining

Abstract

This application discloses a data generation method and system for massive data relationship mining, belonging to the field of data processing technology. The method includes: based on seed data to be processed, calculating the information entropy gain ratio of the splitting feature field to the target field for pairwise ordered combinations of fields in a specified target database table; automatically classifying the relationships between fields based on the comparison result of the information entropy gain ratio and a preset threshold; constructing and storing a data model file based on the classification result, seed data, and its probability distribution; parsing the model file to determine the parent field associated with the target field, and driving data generation according to the association type and conditional probability; or generating data for fields determined to be independent according to their probability distribution. This invention is applied to big data management, realizing full-process automation from data relationship mining to data generation, improving the construction efficiency and quality consistency of structured business data, and reducing reliance on human experience and professional skills.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a data generation method and system for mining relationships in massive datasets. It provides an automated method and system for mining relationships and generating data from massive structured data. Background Technology

[0002] As enterprises deepen their digital transformation, business data is not only expanding in scale but also becoming increasingly complex in its internal structure and the relationships between its fields. In scenarios such as software testing, system verification, and business simulation, there is a rigid demand for high-quality simulation data that can accurately mimic real business logic. However, especially in privacy-sensitive sectors like finance, directly using production data faces significant compliance challenges, requiring an innovative technological balance between data usability and privacy.

[0003] To address this need, the methods the industry has long relied on have significant limitations. Traditionally, users had to manually construct data, a process involving in-depth manual breakdown of business rules, manual writing of complex SQL statements based on table structure constraints, and repeated verification. This method is not only extremely inefficient and time-consuming, but also prone to errors in manually coding business rules, making it difficult to guarantee data consistency. Furthermore, adjustments during business changes are costly, and it demands a high level of expertise in both SQL and business logic from the operator. Although semi-automated solutions combining tools like the Faker library or DataFactory have emerged, simplifying field rule configuration to some extent, they still haven't achieved full automation. Users still need to manually complete the core business rule analysis and complex relational configurations, thus failing to fundamentally solve the problems of overall inefficiency, high maintenance costs with business iterations, and continued heavy reliance on human experience.

[0004] In summary, existing technical solutions fail to achieve automated, low-barrier, and high-quality data construction when dealing with the need to generate massive amounts of complex and interconnected structured business data. Whether using fully manual or semi-automatic methods, it is difficult to generate the required data quickly and cost-effectively while ensuring that the data conforms to real business rules and relational constraints, thus creating a technological gap that urgently needs to be addressed. Summary of the Invention

[0005] This invention provides a data generation method for mining relationships in massive datasets, solving the technical problems of low efficiency, poor consistency, and high professional threshold caused by reliance on manual experience and non-automated processes in the construction of structured business data in existing technologies.

[0006] This invention is achieved through the following technical solution:

[0007] Firstly, this application provides a data generation method for mining relationships in massive datasets, comprising the following steps:

[0008] Based on the seed data to be processed, for each pairwise ordered combination of fields in the specified target database table, calculate the information entropy gain ratio of the split feature field to the target field in each field combination;

[0009] Based on the comparison result between the information entropy gain rate and the preset threshold, the classification result of the association relationship between fields is obtained;

[0010] Based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between each field combination, a data model file is constructed and stored. The data model file includes at least a gain ratio model file that records the information entropy gain ratio between each field combination, an independent field model file that records the fields classified as independent relationships and their data probability distributions, and an association field model file that records the field combinations classified as association relationships and their conditional probabilities and association types.

[0011] Based on the gain ratio model file, the associated parent field is determined for the target field to be generated. Combining the found field combinations, association types, and conditional probabilities, the data of the target field is generated based on the seed data; or, if the gain ratio model file determines that the target field has no parent field, its data is generated based on the independent field model file.

[0012] Further, based on the seed data to be processed, for each pairwise ordered combination of fields in the specified target database table, the information entropy gain ratio of the splitting feature field to the target field in each field combination is calculated, including:

[0013] The seed data is divided into multiple independent groups according to the field list, and the information entropy of each field combination is calculated.

[0014] Perform Cartesian operations on the field list to generate all pairwise ordered field combinations, and label the target field and split feature field in each combination;

[0015] For each field combination, calculate the information entropy gain of the target field in the combination of the split feature fields;

[0016] Based on the information entropy gain and the information entropy of the split feature field itself, the information entropy gain rate of the target field combined with the split feature field is calculated.

[0017] Further, the step of obtaining the classification result of the association relationship between fields based on the comparison result of the information entropy gain rate and the preset threshold includes:

[0018] If a certain field is taken as the target field, and the information entropy gain ratio of its combination with all other fields is less than the preset threshold, then the field is classified as an independent field.

[0019] If the information entropy gain ratio of a combination with a certain field as the target field is greater than or equal to the preset threshold, and the conditional probability of the target field taking a certain value when the split feature field takes a certain value is 1, then the relationship corresponding to this specific value under the combination is classified as a complete association.

[0020] If the information entropy gain ratio is greater than or equal to the preset threshold, and when the split feature field takes a specific value, the conditional probability of each value of the target field is equal, then the relationship corresponding to this specific value under this combination is classified as uniform association.

[0021] If the information entropy gain rate is greater than or equal to the preset threshold, and the correlation is neither complete nor uniform, then it is classified as a strong correlation.

[0022] Furthermore, the association field model file is stored in matrix form. Its outer structure defines the association field combination, and the inner matrix records the seed data value of the association field combination, the conditional probability corresponding to the value, and the association type identifier representing complete association, uniform association, or strong association.

[0023] Further, determining the associated parent field for the target field to be generated based on the gain ratio model file includes:

[0024] Iterate through all information entropy gain rates in the gain rate model file that correspond to the target field combination;

[0025] If all information entropy gain ratios are less than the preset threshold, then the target field is determined to be an independent field without a parent field.

[0026] If there is an information entropy gain rate greater than or equal to the preset threshold, then the splitting feature field corresponding to the maximum value is determined to be the parent field of the target field.

[0027] Furthermore, the step of generating data for the target field based on seed data by combining the found field combinations, association types, and conditional probabilities includes:

[0028] If the association type is a full association, then the current value of the mapped parent field is the value of the target field;

[0029] If the association type is uniform association, then the target field will be randomly selected and assigned a value evenly from all possible values ​​corresponding to the current value of the parent field.

[0030] If the association type is strong association, then a weighted random selection is made from all possible values ​​of the target field based on the conditional probability weight corresponding to the current value of the parent field.

[0031] Secondly, this application provides a data generation system for massive data relationship mining, including:

[0032] The gain ratio calculation module is used to calculate the information entropy gain ratio of the split feature field to the target field in each pairwise ordered combination of fields in the specified target database table, based on the seed data to be processed.

[0033] The relationship classification module is communicatively connected to the gain rate calculation module and is used to obtain the classification result of the association relationship between fields based on the comparison result of the information entropy gain rate and the preset threshold.

[0034] The model building and storage module is communicatively connected to the relationship classification module. It is used to build and store a data model file based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between each field combination. The data model file includes at least a gain ratio model file that records the information entropy gain ratio between each field combination, an independent field model file that records the fields classified as independent relations and their data probability distribution, and an association field model file that records the field combinations classified as association relations and their conditional probabilities and association types.

[0035] The model-driven data generation module is communicatively connected to the model building and storage module. It is used to determine the associated parent field for the target field to be generated based on the gain ratio model file, and generate the data of the target field based on seed data by combining the found field combinations, association types and conditional probabilities; or, if it is determined from the gain ratio model file that the target field has no parent field, then its data is generated based on the independent field model file.

[0036] Furthermore, the data model construction module includes:

[0037] The gain ratio calculation unit is used to calculate the information entropy gain ratio of the split feature field to the target field in each pairwise ordered combination of fields in the specified target database table, based on the seed data to be processed.

[0038] The relationship classification unit is communicatively connected to the gain rate calculation unit and is used to classify the relationship between fields into independent relationship, complete relationship, uniform relationship or strong relationship based on the comparison result of the information entropy gain rate and the preset threshold.

[0039] The model file generation unit is communicatively connected to the relationship classification unit and is used to construct and store the gain rate model file, the independent field model file, and the associated field model file based on the classification results, seed data and their probability distribution, and the calculated information entropy gain rate between the various field combinations.

[0040] Furthermore, the model-driven data generation module includes:

[0041] The model parsing unit is used to read and parse the data model file;

[0042] The association decision unit is communicatively connected to the model parsing unit and is used to determine the associated parent field and association relationship type for the target field based on the gain ratio model file;

[0043] The data generation unit is communicatively connected to the model parsing unit and the association decision unit. It is used to generate data for the target field based on seed data by calling the corresponding data generation strategy according to the association relationship type, combining the found field combinations, association types and conditional probabilities.

[0044] Thirdly, this application provides a computer-readable storage medium storing a data generation program for massive data relationship mining, wherein when the data generation program for massive data relationship mining is executed by a processor, it implements the steps of the data generation method for massive data relationship mining as described above.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] By automating the calculation of information entropy gain rate to quantify and classify the relationships between fields, and based on this, a structured data model file is constructed, driving the system to intelligently select the corresponding probability model to generate data. This achieves full automation from data relationship mining to data generation, improves data construction efficiency, ensures the consistency between generated data and real business rules, and significantly reduces reliance on human experience and professional skills.

[0047] By decoupling the data model construction from the data generation process and making the generation process an independent unit of data tables, supporting concurrent processing, it is possible to further meet the high-performance generation requirements in massive data scenarios. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] In the attached diagram:

[0050] Figure 1A flowchart of a data generation method for massive data relationship mining provided in an embodiment of this application;

[0051] Figure 2 A complete flowchart of model-driven data generation provided in the embodiments of this application;

[0052] Figure 3 This is a diagram of the model-driven data generation system architecture provided in the embodiments of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0054] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0055] DDL: Data Definition Language;

[0056] SQL: Structured Query Language;

[0057] Data rules: These refer to the rules that constrain table data, including the relationship logic between fields (such as foreign key dependencies and business equivalence relationships) and the type and format constraints of the fields themselves (such as data type, length, value range, and validation rules).

[0058] Firstly, such as Figure 1 As shown, this application provides a data generation method for massive data relationship mining, used for big data management, including the following steps:

[0059] Step S1: Based on the seed data to be processed, for each pairwise ordered combination of fields in the specified target database table, calculate the information entropy gain ratio of the splitting feature field to the target field in each field combination;

[0060] Step S2: Based on the comparison result between the information entropy gain rate and the preset threshold, obtain the classification result of the association relationship between fields;

[0061] Step S3: Based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between each field combination, construct and store a data model file. The data model file includes at least a gain ratio model file that records the information entropy gain ratio between each field combination, an independent field model file that records the fields classified as independent relationships and their data probability distributions, and an association field model file that records the field combinations classified as association relationships and their conditional probabilities and association types.

[0062] Step S4: Determine the associated parent field for the target field to be generated based on the gain ratio model file, and generate the data of the target field based on the seed data by combining the found field combinations, association types and conditional probabilities; or, if it is determined from the gain ratio model file that the target field has no parent field, then generate its data based on the independent field model file.

[0063] This embodiment quantifies and classifies the relationships between fields by automatically calculating the information entropy gain rate, thereby constructing a structured data model file and driving data generation. This achieves full automation from data relationship mining to simulation data generation, significantly improving the efficiency and quality of constructing massive amounts of business data. The generated data closely matches the statistical distribution and complex business rules of the original data, while greatly reducing the reliance on human experience and professional programming skills.

[0064] In one embodiment, step S1: Based on the seed data to be processed, for each pairwise ordered combination of fields in the specified target database table, calculate the information entropy gain ratio of the splitting feature field to the target field in each field combination, specifically including the following steps:

[0065] Step S11: Obtain the original business data (i.e., seed data) and its table structure definition for modeling from the specified target database table;

[0066] Specifically, establish a connection with the target database and specify the data table to be modeled; obtain the DDL (Data Definition Language) statements of the data table to extract the list of field names; at the same time, read existing business data from the table as seed data for building the model;

[0067] Step S12: Calculate the probability distribution of each field's value and its information entropy;

[0068] Specifically, seed data is divided according to the field list to obtain an independent data sequence for each field; the frequency of occurrence of each unique value in each field is counted and used as an empirical probability. Substituting the probability distribution into the following information entropy formula, the information entropy of each field is calculated. :

[0069] Equation (1)

[0070] in, The seed data of field X is the first The probability of each value occurring; n is the number of unique values ​​of field X in the seed data;

[0071] Step S13: Construct all pairwise ordered field combinations, and calculate the conditional entropy and information entropy gain of the target field of the split feature field combination; further, this includes the following steps:

[0072] Perform Cartesian product operations on the field list to generate all pairwise ordered combinations of fields. And fix the first field The second field is labeled as the split feature field A. Marked as target field D, forming an "A -> D" analysis pair; this rule ensures that any field in the list can be analyzed comprehensively, whether it is a splitting feature field or a target field in different combinations with other fields.

[0073] For example, for fields A, B, and C, field A is used as a splitting feature field in combinations (A, B) and (A, C), and as a target field in combinations (B, A) and (C, A);

[0074] For each analysis pair, the conditional entropy of the target field D is calculated based on the seed data, given the known values ​​of A. Then, the information entropy gain is obtained by using the information entropy H(D) of D itself. The calculation formula is as follows:

[0075] Equation (2)

[0076] Equation (3)

[0077] In the formula, and These represent the number of unique values ​​retrieved from fields A and D, respectively. To represent that field D takes the value dⱼ and field A also takes the value dⱼ. The probability of; Let be the i-th possible value of condition field A; Let j be the j-th possible value of the target field D; The larger the value, the stronger the influence of field A on the deterministic value of field D;

[0078] Step S14: Divide the information entropy gain by the information entropy of the split feature field itself to obtain the normalized information entropy gain rate, which serves as a fair and objective core indicator of association strength.

[0079] To avoid the influence of the number of values ​​in the split feature field on the information entropy gain value, normalization is performed according to the following formula to calculate the final information entropy gain ratio. The calculation formula is as follows:

[0080] Equation (4)

[0081] The information entropy gain rate objectively quantifies the decisive influence of the splitting feature field A on the target field D, and serves as the direct basis for subsequent relationship discrimination and classification.

[0082] Step S15: Based on the comparison results of information entropy gain rate and preset threshold, pre-screen independent fields that have no obvious statistical correlation with other fields;

[0083] The rule used to determine whether a field in a target database table is an independent field is as follows:

[0084] For a specific field in a target database table, if it is used as the target field, what is the information entropy gain ratio calculated between it and all other fields in the table as splitting feature fields (i.e., all...)? If all values ​​are less than a preset threshold, the field is classified as an independent field, indicating that the field does not show significant constraints or influences from other fields at the data level.

[0085] In one embodiment, step S2: obtaining the classification result of the association relationship between fields based on the comparison result of the information entropy gain rate and the preset threshold, specifically includes the following steps:

[0086] Step S21: Based on the comparison results of information entropy gain rate and preset threshold, the relationship between fields is initially classified into independent relationship or association relationship;

[0087] Specifically, for each field D in the specified target database table, examine its information entropy gain ratio calculated when it is used as the target field, compared to all other fields in the table as splitting feature fields. ;

[0088] If all If all values ​​are less than the preset gain rate threshold, then field D is determined to be an independent field; an independent field means that its value does not show any statistical constraints from any other field in the table at the current data level, and can be generated independently based on its own historical distribution.

[0089] If there exists a value greater than or equal to the preset gain rate threshold If so, it is determined that there is a relationship between field D and the corresponding field A, and then proceed to step S22 for further subdivision;

[0090] Step S22: For the field combinations that are determined to have a relationship, by analyzing the specific characteristics of their conditional probability distribution, the relationship is subdivided into three types with clear data generation semantics: complete relationship, uniform relationship, and strong relationship.

[0091] Specifically, for field combinations (A -> D) that satisfy the association condition, it is necessary to conduct in-depth analysis of each specific value taken in the splitting feature field A. When, the conditional probability distribution of target field D The conditional probability is calculated from the joint probability and the marginal probability, using the following formula:

[0092] Equation (5)

[0093] Based on the characteristics of this distribution, the associations can be precisely divided into the following three categories:

[0094] When the split feature field A takes a specific value At that time, the value of the target field D is uniquely determined; that is, there exists a certain value. , making the conditional probability ;

[0095] When the split feature field A takes a specific value When all possible values ​​of the target field D have equal conditional probabilities, the conditional probability of each value is equal.

[0096] When the split feature field A takes a specific value At that time, the value of the target field D follows a specific conditional probability distribution that is neither completely deterministic nor uniformly distributed and is statistically derived from historical seed data;

[0097] Among them, all possible values ​​refer to all non-repeating numerical values ​​or categories that have actually appeared in a certain field in a given seed dataset, which is the basis for calculating statistical indicators such as information entropy and conditional probability.

[0098] To illustrate the above classification more intuitively, the following example is provided: Assume the seed data contains two fields: "lottery method" and "probability," and a partial data example is shown in the table below:

[0099] Table 1. Seed data for the probability distribution of the four lottery methods.

[0100]

[0101] Based on the data in the table above, calculate the conditional probability of "lottery method" with respect to "probability":

[0102] When "lottery method = guaranteed win" is true, the conditional probability of "probability = 1" is 1, indicating a perfect correlation.

[0103] When "Lottery method = specified", the conditional probability of the "probability" values ​​{1 / 6, 1 / 3, 1 / 2} is 1 / 3, which is a uniform correlation.

[0104] When "lottery method = spin the wheel", the conditional probabilities of the "probability" values ​​{1 / 6, 2 / 3} are approximately 2 / 3 and 1 / 3 respectively. This is a non-uniform specific distribution, and therefore a strong correlation.

[0105] The above classification provides a direct basis for adopting different probability generation strategies (direct assignment, uniform random, weighted random) for different types of associations in the subsequent data generation module.

[0106] In cases where there are related fields, it is assumed that field B determines field A; when generating data for field A, a conditional probability model is used for data generation, as shown in equation (6);

[0107] Equation (6)

[0108] in, P(A) represents the joint probability of field B and A taking a specific value combination, and P(A) represents the marginal probability of field A taking that specific value. This formula is used to statistically obtain the various conditional probability values ​​required for step S22 from the seed data.

[0109] By calculating the probability of field A occurring given that field B has occurred, the probability distribution of field A data is obtained, generating data that meets actual business needs.

[0110] In one embodiment, step S3: Based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between various field combinations, a data model file is constructed and stored, specifically including the following steps:

[0111] Step S31: Based on the calculated information entropy gain ratio of all field combinations and the field list, generate a gain ratio model file by constructing and filling a two-dimensional matrix with the field names as row and column indices; this file is used for persistent storage in matrix form and can globally display the correlation strength between fields.

[0112] Specifically, the core of this document is a two-dimensional matrix; the row and column indices of the matrix are field names obtained from the table in step S11; the element located in the i-th row and j-th column of the matrix has the value of the information entropy gain rate calculated in step S14 when the splitting feature field is the j-th column field and the target field is the i-th row field. ;

[0113] The gain ratio model is used to record the information entropy gain ratio of the target field under the influence of the splitting feature field among various field combinations obtained in the gain ratio calculation stage; its storage structure is shown in the figure below; the model uses a two-dimensional matrix to store the information entropy gain ratio between fields, and its row and column names are fields in the table. The information entropy gain ratio between two fields is defined by the row index and column index. For example, GR(A,B) represents the information entropy gain ratio of column colA when column colB is used as the splitting feature. The value of GR(A,B) represents the degree of correlation between the two columns.

[0114] Table 2 Comparison of Column Relationship Calculation Results

[0115]

[0116] Step S32: Based on the identified list of independent fields and their corresponding seed data, the independent field model file is constructed by statistically analyzing the value distribution and random probability of each independent field and organizing them into a nested matrix structure.

[0117] Specifically, the file is stored in a nested matrix format; the row index of the outer matrix is ​​the name of each field categorized as an independent field; the inner matrix stores all deduplicated values ​​of the corresponding independent field in the seed data and their random probability P(A); its structure is shown in the following example:

[0118] The independent field model is used to record field information that is not constrained by the values ​​of other fields. Its storage structure is shown in the figure below. The structure of the independent field model is described by nested matrices. The column index of the outermost matrix is ​​fixed as colp. In its row index, colA, colB, etc. are the column names of independent fields. The outer column name corresponds to the basic data and probability data of the inner layer. Taking colA in the figure as an example, the data column in its inner matrix describes the basic data of the column, and column P represents the discrete probability distribution of the data in the corresponding data column in the row matrix.

[0119] Table 3. Values, data, and probability distributions of variables A and B.

[0120]

[0121] For each independent field, its corresponding inner matrix stores all deduplicated values ​​of that field extracted from the seed data, and the random probability P(A) of each value; the formula for calculating the random probability is as follows:

[0122] Equation (7)

[0123] Where m represents the total number of rows (sample size) of the seed data, and n represents the number of rows where field A takes a specific value;

[0124] This model ensures that the distribution of the generated independent field data remains consistent with the original business data;

[0125] Step S33: Based on the classified combinations of related fields and their detailed information, construct records containing co-occurrence values, conditional probabilities, and association types for each pair of fields, and organize them into a nested matrix to form a related field model file;

[0126] Specifically, the file uses a nested matrix structure to store information; each row of the outer matrix corresponds to a combination of related fields, and its key (or index) is in the form of "parent field name → child field name", which clearly indicates the direction of association; for each pair of related fields, the corresponding inner matrix stores the specific co-occurrence value combination extracted from the seed data, the calculated conditional probability (condP), and the association type identifier (kind) determined by step S22, row by row.

[0127] The kind value is an association type identifier, which means: kind=1 corresponds to a scenario with a conditional probability of 1, i.e., complete association; kind=2 corresponds to a uniform probability value, i.e., uniform association; kind=3 corresponds to a strong association; this identifier is used to drive different probability strategies when generating data.

[0128] This model file precisely encapsulates the specific dependency rules between fields, providing a direct basis for generating differentiated data based on different association types in step S4. An example of its structure is shown below:

[0129] Table 4 Input / Output Data Mapping and Conditional Probability Table

[0130]

[0131] Step S34: The three model data structures—gain rate, independent fields, and related fields—that have been constructed are converted into persistent model files through serialization and disk write operations, thereby realizing the solidification and reuse of the model;

[0132] Furthermore, it includes the following steps:

[0133] Receive the complete data structures corresponding to the gain rate model, independent field model and related field model constructed in steps S31, S32 and S33 respectively, and serialize these data structures in memory into a specific binary or structured file format (such as JSON, Protocol Buffers, etc.);

[0134] Write the serialized data stream to the specified path on the disk to generate an independent, persistent model file;

[0135] This process materializes the results of all the aforementioned analysis and mining into independent knowledge files that do not rely on database connections and original seed data, providing a foundation for the "model once, generate multiple times" data generation model.

[0136] In one embodiment, step S4: determine the associated parent field for the target field to be generated based on the gain ratio model file, and generate the data of the target field based on the seed data by combining the found field combination, association type and conditional probability; or, if the target field is determined to have no parent field based on the gain ratio model file, then generate its data based on the independent field model file; its core depends on the model file constructed in step S3, and uses random probability (equation (7)) and conditional probability (equation (6)) rules to guide data generation; its data is a column of data values ​​corresponding to the target field that is currently determined to have no parent field (i.e., independent field).

[0137] like Figure 2 As shown, the core of the model-driven data generation module is to automatically parse and apply the data model constructed in the aforementioned steps to drive data generation; its execution logic follows a layered architecture of "model parsing, field association determination, and conditional data generation," and the specific process is as follows:

[0138] After the process starts, the gain rate model, independent field model and related field model files are first read from persistent storage and loaded into memory to obtain the fields and related information of the data table to be generated. Then, preprocessing can be performed on special fields according to requirements. For example, fixed values ​​can be uniformly filled into specific fields of all tables to be generated, or fixed values ​​can be filled into specific fields of a specified table only.

[0139] The system iterates through the row indices (i.e., all field names) of the gain ratio model matrix and adds fields that have not yet generated data to a "queue of fields to be generated". Then, the system processes this queue in a loop: it retrieves the field at the head of the queue as the current target field; if the field has already generated data, it skips it and processes the next one; if it has not generated data, it determines the relationship based on the gain ratio model. Specifically:

[0140] Iterate through all information entropy gain rates that should be generated for this field combination in the gain rate model;

[0141] If all values ​​are less than the preset threshold, the field is identified as an independent field and the process is switched to the independent field data processing flow.

[0142] If there is data that is greater than or equal to the threshold, the column field corresponding to the maximum value will be determined as its parent field.

[0143] For a target field that has a parent field, check whether its parent field data has been generated.

[0144] If not generated, the parent field is added to the head of the queue to be generated and processed first to ensure the dependency order;

[0145] If the parent field data is ready, query the related field model using "parent field name -> target field name" as the key to obtain the relationship type (kind) and conditional probability (condP), and generate the data accordingly:

[0146] If kind=1 (fully correlated), it means that the value of the child field is uniquely determined by the value of the parent field. In this case, the unique child field value corresponding to this parent field value in the correlated field model is directly assigned to the current target field.

[0147] If kind=2 (uniform association), it means that under the current value of the parent field, all possible values ​​of the child field have equal probability of appearing. Then, one child field value is randomly selected uniformly from all possible child field values ​​corresponding to the current value of the parent field and assigned a value.

[0148] If kind=3 (strong association), it means that under the current value of the parent field, the value of the child field follows a specific historical probability distribution. Then, the possible child field values ​​are assigned weighted random sampling values ​​using the condP recorded in the association model as weights.

[0149] For a target field determined to be an independent field, its data generation does not depend on other fields. The system will make a judgment based on the table structure definition (DDL): if the field is defined as an enumeration type in the DDL (for example, defined by CHECK constraint or ENUM type), and its historical value list in the independent field model is empty, then uniform random assignment will be performed according to the enumeration key value table defined in the DDL; otherwise, the historical values ​​will be weighted and randomly sampled and assigned using the random probability recorded in the independent field model as the weight.

[0150] After each field is successfully generated, it is marked as complete and removed from the queue. This iterative process continues until the queue of fields to be generated is cleared, marking the completion of the entire table's data generation. Since the data generation processes between different database tables are logically completely independent, this method supports multi-task parallel scheduling at the table level, and can start concurrent data generation processes for multiple tables at the same time, thereby achieving efficient construction of massive amounts of data.

[0151] Furthermore, it includes the following steps:

[0152] Step S41: Load the gain ratio model, independent field model, and associated field model files into memory and initialize the list of fields to be processed according to the gain ratio model structure;

[0153] Specifically, when the data generation process starts, the persistent gain ratio model, independent field model, and associated field model files are loaded from disk into memory and restored into an operable data structure. Then, the row indices of the gain ratio model matrix (i.e., the names of all fields in the table) are traversed, the data generation status of each field is checked, and fields whose data has not yet been generated are added to a "fields to be generated queue" in turn. This queue serves as the core task list driving the entire generation process, and its initialization logic ensures that the system can manage the generation dependencies between fields in an orderly manner.

[0154] Step S42: For the target field to be generated, determine its associated parent field by querying the gain ratio model, and drive the corresponding strategy to generate data values ​​that conform to the association constraints for the field based on the rule type and condition probability recorded in the associated field model.

[0155] Specifically, retrieve the target field to be processed from the queue to be generated; query the row data corresponding to that field combination in the gain ratio model:

[0156] If all gain values ​​in the row are less than the preset threshold, the target field is determined to be an independent field, and the process jumps to step S43.

[0157] If there is a gain ratio greater than or equal to the threshold, then the field corresponding to the maximum value is selected as the parent field of the current target field;

[0158] Then, check whether the data for its parent field has been generated;

[0159] If not generated, the parent field is added to the head of the queue to be generated and processed first to ensure the dependency relationship; if the parent field data is ready, the related field model is queried using "parent field name → child field name" as the key to find the record that matches the current parent field value, obtain its association type kind and conditional probability condP, and generate the child field data accordingly.

[0160] If kind=1 (fully correlated), it means that the subfield value is uniquely determined by the parent field value, and the unique subfield value of the record in the correlation model is directly assigned to the current target field;

[0161] If kind=2 (uniform association), it means that under the current value of the parent field, all possible values ​​of the child field have equal probability of appearing. From all possible child field values ​​corresponding to the current value of the parent field, one is randomly selected uniformly for assignment.

[0162] If kind=3 (strong association), it means that under the current value of the parent field, the value of the child field follows a specific historical probability distribution. The possible values ​​of the child field are weighted by the condP (conditional probability) recorded in the association model, and the selected values ​​are assigned to the current target field.

[0163] Step S43: For an independent target field determined to have no parent field, generate a data value that conforms to its own statistical characteristics by querying its historical value distribution or combining the enumeration constraints in the table structure and adopting a corresponding random strategy.

[0164] Specifically, for a target field determined to be an independent field, its data generation does not depend on other fields. The system first queries the independent field model to obtain all possible values and their corresponding random probabilities of this field in the seed data. At the same time, it combines the constraint information in the table structure definition (DDL) for judgment: If this field is defined as an enumeration type field in the DDL (for example, defined through a CHECK constraint), and the historical value list in its independent field model is empty, then it turns to perform uniform random assignment based on the enumeration code value table defined in the DDL. The judgment of the enumeration type field is an existing technology in the database field. For example, in the Oracle database, the enumeration value can be defined through the CHECK constraint in the DDL statement (such as CHECK(gender IN ('male', 'female', 'unknown'))), and this constraint information is stored in the system table of the database (such as the USER_CONSTRAINTS table) and can be obtained through query.

[0165] If the above conditions are not met, directly perform weighted random sampling assignment on its historical values with the random probability P(A) recorded in the independent field model as the weight.

[0166] Step S44: Driven by the dynamic state of the field queue to be generated and the generation results of the previous steps, loop through all fields in the queue and schedule parallel generation tasks for multiple logically independent data tables, and output a complete new data table that conforms to the original data relationship and distribution characteristics.

[0167] Specifically, the system loops through Step S42 and Step S43 to continuously process the fields in the queue to be generated. After successfully generating data for a field, it marks it as completed and removes it from the queue. This iterative process continues until the queue of fields to be generated is emptied, at which point it indicates that the data of the entire target data table has been completely generated. At the same time, since the data generation processes between different database tables are completely independent logically, this method supports multi-task parallel scheduling at the table granularity. It can start concurrent data generation processes for multiple tables and independently complete the above iterative steps to achieve efficient construction of massive data.

[0168] In summary, this invention, through the aforementioned steps, achieves automated mining and high-fidelity data generation of complex business relationships within massive amounts of structured data. The entire process eliminates the need for manually writing complex data generation rules or SQL statements, significantly reducing the technical barrier and time cost of data construction. It also effectively solves the core problems of traditional manual or semi-automatic data construction methods, such as low efficiency, poor consistency, high professional requirements, and insufficient performance for large-scale generation.

[0169] Secondly, this application provides a data generation system for massive data relationship mining, including:

[0170] The gain ratio calculation module is used to calculate the information entropy gain ratio of the split feature field to the target field in each pairwise ordered combination of fields in the specified target database table, based on the seed data to be processed.

[0171] The relationship classification module is communicatively connected to the gain rate calculation module and is used to obtain the classification result of the association relationship between fields based on the comparison result of the information entropy gain rate and the preset threshold.

[0172] The model building and storage module is communicatively connected to the relationship classification module. It is used to build and store a data model file based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between each field combination. The data model file includes at least a gain ratio model file that records the information entropy gain ratio between each field combination, an independent field model file that records the fields classified as independent relations and their data probability distribution, and an association field model file that records the field combinations classified as association relations and their conditional probabilities and association types.

[0173] The model-driven data generation module is communicatively connected to the model building and storage module. It is used to determine the associated parent field for the target field to be generated based on the gain ratio model file, and generate the data of the target field based on seed data by combining the found field combinations, association types and conditional probabilities; or, if it is determined from the gain ratio model file that the target field has no parent field, then its data is generated based on the independent field model file.

[0174] In one embodiment, the data model building module includes:

[0175] The gain ratio calculation unit is used to calculate the information entropy gain ratio of the split feature field to the target field in each pairwise ordered combination of fields in the specified target database table, based on the seed data to be processed.

[0176] The relationship classification unit is communicatively connected to the gain rate calculation unit and is used to classify the relationship between fields into independent relationship, complete relationship, uniform relationship or strong relationship based on the comparison result of the information entropy gain rate and the preset threshold.

[0177] The model file generation unit is communicatively connected to the relationship classification unit and is used to construct and store the gain rate model file, the independent field model file, and the associated field model file based on the classification results, seed data and their probability distribution, and the calculated information entropy gain rate between the various field combinations.

[0178] In one embodiment, the model-driven data generation module includes:

[0179] The model parsing unit is used to read and parse the data model file;

[0180] The association decision unit is communicatively connected to the model parsing unit and is used to determine the associated parent field and association relationship type for the target field based on the gain ratio model file;

[0181] The data generation unit is communicatively connected to the model parsing unit and the association decision unit. It is used to generate data for the target field based on seed data by calling the corresponding data generation strategy according to the association relationship type, combining the found field combinations, association types and conditional probabilities.

[0182] In a specific software implementation architecture, a data generation system for mining relationships in massive datasets can be instantiated as an integrated platform comprising six core components.

[0183] like Figure 3 As shown, the top layer of the system consists of business function modules, providing users with a visual operation interface. The core functions are concentrated in environment management, model management, and data generation task triggering. Among them, environment management is responsible for the full lifecycle maintenance and specification of seed data sources and generated data storage targets; model management is responsible for retrieving historical data models from disk and providing an entry point for building new models; the data generation function is the final operation interface that drives the entire process.

[0184] Below the business functionality layer, the business configuration module and the business database module provide rules and data support for the system. The business configuration module allows users to set global fixed values ​​or to perform fine-grained configurations for specific fields in specific data tables. The business database module is dedicated to the unified management and maintenance of the source database connection containing seed data, ensuring stable access to the data source.

[0185] The front-end service module is responsible for preprocessing and analyzing data before it is generated. This module first cleans, transforms, and supplements the rules on the raw seed data obtained from the business database, outputting more complete and higher-quality seed data. Simultaneously, it performs quality assessments on the seed data, calculating key indicators such as non-empty table coverage and non-empty field coverage, and generating a quality report. Furthermore, this module is also responsible for efficiently transmitting the processed seed data and business configuration rules to the core data center module.

[0186] The data center module is the algorithmic core and execution engine of the entire system, specifically composed of a model center, service management, model-driven data generation, and a management center. The model center sequentially performs information entropy calculation, applies user-configured specific value filling rules, calculates the information entropy gain ratio between fields, and finally constructs a gain ratio model, an independent field model, and a related field model based on preset thresholds, serializing and storing them on disk. Service management provides multi-threaded parallel management and progress monitoring for the model building process. During the data generation phase, the model-driven data generation submodule parses the stored model files, determines the specific data generation strategy based on the association type in the related field model (e.g., direct mapping for fully correlated fields, equal-probability randomization for uniformly correlated fields, and weighted randomization for strongly correlated fields), and completes batch data synthesis based on the gain ratio model and the independent field model, while applying the configured specific value filling rules. The management center provides multi-threaded scheduling, database environment isolation, real-time progress monitoring, and coverage calculation and quality assessment of the generated data itself.

[0187] The global module provides fundamental support for the entire system, ensuring its robustness, observability, and maintainability. This module primarily includes: disk storage for persistent model file storage; exception monitoring responsible for capturing and recording runtime anomalies; data writing (or data output) to the target database for the final generated data; and log management for collecting and displaying the entire process's operation records. Through the collaborative work of these modules, the system fully realizes the entire process from automated data relationship mining to high-quality simulation data generation, and provides comprehensive configuration, management, and monitoring functions.

[0188] Thirdly, embodiments of this application provide a data generation device for mining relationships in massive datasets. This data generation device can be a personal computer (PC), a laptop computer, a server, or other devices with data processing capabilities.

[0189] In this embodiment of the application, the data generation device for mining relationships in massive data may include a processor, a memory, a communication interface, and a communication bus.

[0190] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0191] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of data generation devices for massive data relationship mining, as well as interfaces used for interconnecting these devices with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0192] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0193] The processor can be a general-purpose processor, which can call a data generation program for massive data relationship mining stored in memory and execute the data generation method for massive data relationship mining provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the data generation program for massive data relationship mining is called can refer to the various embodiments of the data generation method for massive data relationship mining in this application, and will not be repeated here.

[0194] Fourthly, embodiments of this application also provide a readable storage medium.

[0195] The present application stores a data generation program for massive data relationship mining on a readable storage medium, wherein when the data generation program for massive data relationship mining is executed by a processor, it implements the steps of the data generation method for massive data relationship mining as described above.

[0196] The method implemented when the data generation program for massive data relationship mining is executed can be referred to in various embodiments of the data generation method for massive data relationship mining in this application, and will not be repeated here.

[0197] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data generation method for mining relationships in massive datasets, characterized in that, Includes the following steps: Based on the seed data to be processed, for each pairwise ordered combination of fields in the specified target database table, calculate the information entropy gain ratio of the split feature field to the target field in each field combination; Based on the comparison result between the information entropy gain rate and the preset threshold, the classification result of the association relationship between fields is obtained; Based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between each field combination, a data model file is constructed and stored. The data model file includes at least a gain ratio model file that records the information entropy gain ratio between each field combination, an independent field model file that records the fields classified as independent relations and their data probability distributions, and an association field model file that records the field combinations classified as association relations and their conditional probabilities and association types. Based on the gain ratio model file, determine the associated parent field for the target field to be generated, and combine the found field combinations, association types and conditional probabilities to generate the data of the target field based on the seed data; Alternatively, if the target field is determined to have no parent field based on the gain ratio model file, its data can be generated based on the independent field model file.

2. The data generation method for massive data relationship mining according to claim 1, characterized in that, Based on the seed data to be processed, for each pairwise ordered combination of fields in the specified target database table, the information entropy gain ratio of the splitting feature field to the target field in each field combination is calculated, including: The seed data is divided into multiple independent groups according to the field list, and the information entropy of each field combination is calculated. Perform Cartesian operations on the field list to generate all pairwise ordered field combinations, and label the target field and split feature field in each combination; For each field combination, calculate the information entropy gain of the target field in the combination of the split feature fields; Based on the information entropy gain and the information entropy of the split feature field itself, the information entropy gain rate of the target field combined with the split feature field is calculated.

3. The data generation method for massive data relationship mining according to claim 1, characterized in that, The step of obtaining the classification result of the association between fields based on the comparison result of the information entropy gain rate and the preset threshold includes: If a certain field is taken as the target field, and the information entropy gain ratio of its combination with all other fields is less than the preset threshold, then the field is classified as an independent field. If the information entropy gain ratio of a combination with a certain field as the target field is greater than or equal to the preset threshold, and the conditional probability of the target field taking a value is 1 when the split feature field takes a specific value, then the classification result of the combination is recorded as fully associated. If the information entropy gain ratio is greater than or equal to the preset threshold, and when the split feature field takes a specific value, the conditional probability of each value of the target field is equal, then the relationship corresponding to this specific value under this combination is classified as uniform association. If the information entropy gain rate is greater than or equal to the preset threshold, and the correlation is neither complete nor uniform, then it is classified as a strong correlation.

4. The data generation method for massive data relationship mining according to claim 1, characterized in that, The association field model file is stored in matrix form. Its outer structure defines the combination of association fields, and the inner matrix records the seed data values ​​of the association field combinations, the conditional probabilities corresponding to the seed data values, and the association type identifiers representing complete association, uniform association, or strong association.

5. The data generation method for massive data relationship mining according to claim 1, characterized in that, The step of determining the associated parent field for the target field to be generated based on the gain ratio model file includes: Iterate through all information entropy gain rates in the gain rate model file that correspond to the target field combination; If all information entropy gain ratios are less than the preset threshold, then the target field is determined to be an independent field without a parent field. If there is an information entropy gain rate greater than or equal to the preset threshold, then the splitting feature field corresponding to the maximum value is determined to be the parent field of the target field.

6. The data generation method for massive data relationship mining according to claim 5, characterized in that, The process of generating data for the target field based on seed data, by combining the found field combinations, association types, and conditional probabilities, includes: If the association type is a full association, then the current value of the mapped parent field is the value of the target field; If the association type is uniform association, then the target field will be randomly selected and assigned a value evenly from all possible values ​​corresponding to the current value of the parent field. If the association type is strong association, then a weighted random selection is made from all possible values ​​of the target field based on the conditional probability weight corresponding to the current value of the parent field.

7. A data generation system for mining relationships in massive datasets, characterized in that, The system is used to implement the data generation method for massive data relationship mining as described in any one of claims 1-6; the system includes: The gain ratio calculation module is used to calculate the information entropy gain ratio of the split feature field to the target field in each pairwise ordered combination of fields in the specified target database table, based on the seed data to be processed. The relationship classification module is communicatively connected to the gain rate calculation module and is used to obtain the classification result of the association relationship between fields based on the comparison result of the information entropy gain rate and the preset threshold. The model building and storage module is communicatively connected to the relationship classification module. It is used to build and store a data model file based on the classification results, seed data and their probability distribution, and the calculated information entropy gain ratio between each field combination. The data model file includes at least a gain ratio model file that records the information entropy gain ratio between each field combination, an independent field model file that records the fields classified as independent relations and their data probability distribution, and an association field model file that records the field combinations classified as association relations and their conditional probabilities and association types. The model-driven data generation module is communicatively connected to the model building and storage module. It is used to determine the associated parent field for the target field to be generated based on the gain ratio model file, and generate the data of the target field based on seed data by combining the found field combinations, association types and conditional probabilities; or, if it is determined from the gain ratio model file that the target field has no parent field, then its data is generated based on the independent field model file.

8. The data generation system for massive data relationship mining according to claim 7, characterized in that, The data model construction module includes: The gain ratio calculation unit is used to calculate the information entropy gain ratio of the split feature field to the target field in each pairwise ordered combination of fields in the specified target database table, based on the seed data to be processed. The relationship classification unit is communicatively connected to the gain rate calculation unit and is used to classify the relationship between fields into independent relationship, complete relationship, uniform relationship or strong relationship based on the comparison result of the information entropy gain rate and the preset threshold. The model file generation unit is communicatively connected to the relationship classification unit and is used to construct and store the gain rate model file, the independent field model file, and the associated field model file based on the classification results, seed data and their probability distribution, and the calculated information entropy gain rate between the various field combinations.

9. The data generation system for massive data relationship mining according to claim 7, characterized in that, The model-driven data generation module includes: The model parsing unit is used to read and parse the data model file; The association decision unit is communicatively connected to the model parsing unit and is used to determine the associated parent field and association relationship type for the target field based on the gain ratio model file; The data generation unit is communicatively connected to the model parsing unit and the association decision unit. It is used to generate data for the target field based on seed data by calling the corresponding data generation strategy according to the association relationship type, combining the found field combinations, association types and conditional probabilities.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data generation program for massive data relationship mining, wherein when the data generation program for massive data relationship mining is executed by a processor, it implements the steps of the data generation method for massive data relationship mining as described in any one of claims 1 to 7.

Images