A wide table generation method, device, equipment and storage medium

Abstract

The embodiment of the present specification provides a wide table generation method, device and equipment and storage medium. The method comprises: acquiring a plurality of data tables and data transmission relationships between the plurality of data tables; generating a read-write relationship graph based on the plurality of data tables and the data transmission relationships between the plurality of data tables; wherein the read-write relationship graph comprises a graph node representing a data table and edge data representing a data transmission relationship between the data tables; performing clustering operation on the graph node of the read-write relationship graph to obtain a plurality of clustering clusters; each clustering cluster comprises a base table and a derived table generated based on the base table; the base fields of the base table in each clustering cluster form a base field set, and the derived fields of the derived table in each clustering cluster form a derived field set; and the base fields in the base field set of each clustering cluster and the derived fields in the derived field set are spliced into a wide table corresponding to each clustering cluster, which can improve the data management efficiency of the data table.

Description

Technical Field

[0001] The embodiments described in this specification relate to the field of data processing technology, specifically to a method, apparatus, device, and storage medium for generating wide tables. Background Technology

[0002] Data governance encompasses a comprehensive set of management practices related to data usage, aiming to enhance the value of data. Currently, data governance primarily involves manually extracting valuable fields from original tables to construct wide-field tables.

[0003] However, when there are many original tables to be processed, manually constructing wide tables is slow, inefficient, and time-consuming. Therefore, existing technologies suffer from the technical problem of low data governance efficiency due to the long construction time of wide tables. Summary of the Invention

[0004] In view of this, various embodiments of this specification aim to provide a method, apparatus, device, and storage medium for generating wide tables, so as to reduce the workload of data sorting to a certain extent.

[0005] This specification provides a method for generating a wide table through multiple embodiments. The method includes: acquiring multiple data tables and the data transmission relationships between the multiple data tables; generating a read-write relationship graph based on the multiple data tables and the data transmission relationships between the multiple data tables; wherein the read-write relationship graph includes graph nodes representing data tables and edge data representing data transmission relationships between data tables; performing clustering operations on the graph nodes of the read-write relationship graph to obtain multiple clusters; wherein each cluster includes a portion of the data tables in the multiple data tables, and the data tables included in each cluster are divided into a base table and a derived table generated based on the base table; the base table has multiple base fields, and the derived table has at least one derived field; wherein the base fields of the base table in each cluster form a base field set, and the derived fields of the derived table in each cluster form a derived field set; and concatenating the base fields in the base field set of each cluster with the derived fields in the derived field set to form a wide table corresponding to each cluster.

[0006] One embodiment of this specification provides a wide table generation apparatus, the apparatus comprising an acquisition module, a graph construction module, a clustering module, and a processing module; the acquisition module is used to acquire multiple data tables and the data transmission relationships between the multiple data tables; the graph construction module is used to generate a read-write relationship graph based on the multiple data tables and the data transmission relationships between the multiple data tables; wherein the read-write relationship graph includes graph nodes representing data tables and edge data representing data transmission relationships between data tables; the clustering module is used to perform clustering operations on the graph nodes of the read-write relationship graph to obtain multiple clusters; wherein each cluster includes a portion of the data tables in the multiple data tables, and the data tables included in each cluster are divided into a base table and a derived table generated based on the base table; the base table has multiple base fields, and the derived table has at least one derived field; wherein the base fields of the base table in each cluster form a base field set, and the derived fields of the derived table in each cluster form a derived field set; the processing module is used to combine the base fields in the base field set of each cluster with the derived fields in the derived field set to form a wide table corresponding to each cluster.

[0007] This specification provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any of the above embodiments.

[0008] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0009] The various implementation methods provided in this specification involve acquiring multiple data tables and the data transmission relationships between them, constructing a read-write relationship graph of the multiple data tables including graph nodes and edge data, clustering the graph nodes of the read-write relationship graph to obtain multiple clusters, and dividing the data tables in the same cluster into basic tables and derived tables. Based on the basic tables and derived tables, the basic field set and derived field set of each cluster are determined. The basic fields in the basic field set and the derived fields in the derived field set of each cluster are combined to form a wide table corresponding to each cluster, thereby improving the data governance efficiency of the data tables. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of a wide table generation system provided for one embodiment of this specification.

[0011] Figure 2 This is a schematic diagram of the process for generating a wide table according to one embodiment of this specification.

[0012] Figure 3 This is a diagram illustrating the read / write relationships of a data table provided for one embodiment of this specification.

[0013] Figure 4 This is a schematic diagram of the clustering results of a data table provided for one embodiment of this specification.

[0014] Figure 5 This is a schematic diagram of a data table classification method provided for one embodiment of this specification.

[0015] Figure 6 A schematic diagram of a wide table generation apparatus provided for one embodiment of this specification.

[0016] Figure 7 A schematic diagram of a computer device provided for one embodiment of this specification. Detailed Implementation

[0017] To enable those skilled in the art to better understand the solutions described in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0018] In related technologies, data governance typically involves extracting and processing valuable fields from raw data tables to build wide tables based on specific business needs, thereby enhancing the value of the data. However, when there are many raw data tables to be processed, existing methods for building wide tables suffer from low data governance efficiency due to the time-consuming process.

[0019] In addition, in the actual risk control business, there are situations in the field processing stage where fields are referenced across layers, and derived fields are repeatedly processed and stored, which leads to the problem of unordered derivation and duplicate storage of wide fields.

[0020] Therefore, it is necessary to provide a method for generating wide tables. This method involves acquiring multiple data tables and the data transmission relationships between them, constructing a read-write relationship graph of the multiple data tables including graph nodes and edge data, clustering the graph nodes of the read-write relationship graph to obtain multiple clusters, and dividing the data tables in the same cluster into base tables and derived tables. Based on the base tables and derived tables, the base field set and derived field set of each cluster are determined. The base fields in the base field set and the derived fields in the derived field set of each cluster are then combined to form a wide table corresponding to each cluster, thereby improving the data governance efficiency of the data tables.

[0021] Please see Figure 1 This specification provides a system for generating wide tables. The system may include a client. The client can obtain a data table and construct a wide table from the obtained data table. The client can be an electronic device with network access capabilities. Specifically, for example, the client can be a desktop computer, tablet computer, laptop computer, smartphone, digital assistant, smart wearable device, shopping guide terminal, television, smart speaker, microphone, etc. Smart wearable devices include, but are not limited to, smart bracelets, smartwatches, smart glasses, smart helmets, smart necklaces, etc. Alternatively, the client can also be software capable of running on the electronic device.

[0022] In some implementations, the wide table generation system may include a server. The server can acquire data tables and construct wide tables from the acquired data tables. The server can be an electronic device with certain computing capabilities. It may have a network communication module, a processor, and memory, etc. Of course, the server can also refer to software running on the electronic device. The server can also be a distributed server, which can be a system with multiple processors, memory, network communication modules, etc., operating collaboratively. Alternatively, the server can be a server cluster formed by several servers. Or, with the development of science and technology, the server can also be a new technical means capable of realizing the corresponding functions of the embodiments described in the specification. For example, it could be a new form of "server" based on quantum computing.

[0023] In some implementations, the system for generating wide tables may include a client and a server. The server can construct wide tables from data tables obtained by the client.

[0024] Please see Figure 2 This specification provides a method for generating a wide table, according to one embodiment. This method can be applied to a wide table generation system. The method may include the following steps.

[0025] Step S110: Obtain multiple data tables and the data transfer relationships between them.

[0026] In some cases, to improve the efficiency of wide table construction, the efficiency of clustering operations on data tables can be improved based on the data transfer relationships between multiple data tables, thereby improving the efficiency of wide table construction.

[0027] In this embodiment, multiple data tables and the data transmission relationships between them can be obtained. Specifically, when obtaining multiple data tables, the metadata of each data table can be obtained. The metadata of a data table may include information such as its lineage data and fields. The lineage data indicates the relationship between the data tables; for example, it may include upstream and downstream information. The lineage relationship can be determined through this upstream and downstream information. After obtaining the lineage data, the data transmission relationships between the multiple data tables can be determined based on the lineage data of each data table.

[0028] In this context, a data table can refer to a data storage container within a database, where data can be stored. The data transfer relationship between data tables refers to their input and output relationships. For example, when outputting a field from table1 to table2, or processing a field from table1 and then outputting it to table2, the data transfer relationship between table1 and table2 is that table1 points to table2; that is, table1 is the input table to table2, and table2 is the output table to table1. In other words, table1 is the read table, and table2 is the write table.

[0029] For example, the data tables can be obtained from a data warehouse that includes a source layer, a data warehouse layer, and an application layer. The data tables stored in the source layer can be data tables directly from the source system without any processing. The data warehouse layer and the application layer can store data tables generated by processing the data tables in the source layer.

[0030] Specifically, the field set of a data table can be determined based on multiple fields in the table's metadata. Here, the field set refers to the collection of multiple fields in the table's metadata. For example, when determining the field set of a data table, all fields can be converted to lowercase, or no word segmentation can be performed on them, resulting in a field set composed of the corresponding multiple fields. This allows the fields in the field set to retain their original field information.

[0031] For example, please refer to Table 1. Based on the data transfer relationship between data tables, data tables can be divided into input tables and output tables. Input tables represent the data tables that are read, and output tables represent the data tables that are written after reading the input tables.

[0032] Table 1

[0033] Input table Fields in the input table Output table Fields of the output table table1 a1, a2, a3, a4, a5, a6 table2 a1, a2, a3, a24, a25 table1 a1, a2, a3, a4, a5, a6 table9 a2, a3, a9 table2 a1, a2, a3, a24, a25 table3 a2, a3, a24, a35, a36 table2 a1, a2, a3, a24, a25 table4 a41,a2,a3 table3 a2, a3, a24, a35, a36 table6 a2, a3, a61 table4 a41,a2,a3 table5 a2, a3, a53, a54, a55 table5 a2, a3, a53, a54, a55 table9 a2, a3, a9 table7 a2, a3, a71 table5 a2, a3, a53, a54, a55 table7 a2, a3, a71 table8 a2, a3, a82, a83, a84 table8 a2, a3, a82, a83, a84 table6 a2, a3, a61 table6 a2, a3, a61 table9 a2, a3, a9 table5 a2, a3, a53, a54, a55 table8 a2, a3, a82, a83, a84

[0034] In some cases, to further improve the efficiency of clustering operations on data tables, the data transfer relationships between the acquired data tables can be screened and filtered.

[0035] Therefore, in some implementations, after determining the field set of each of the multiple data tables, the similarity between the multiple data tables can be determined based on the data transmission relationship between the multiple data tables and the field set of each of the multiple data tables. Based on a preset similarity threshold and the similarity between the multiple data tables, the data transmission relationship between the multiple data tables can be filtered to obtain the filtered data transmission relationship between the multiple data tables.

[0036] Specifically, the similarity between data tables can refer to overlap similarity. Overlap similarity can be determined by the data transfer relationships between the data tables and their field sets. Overlap similarity describes the degree of overlap between two field sets.

[0037] For example, the overlap similarity between data tables can be calculated using Formula 1.

[0038]

[0039] Where sim represents the overlap similarity between the data tables, |A∩B| represents the number of identical fields in field set A and field set B, and min{|A|,|B|} represents the minimum number of fields in field set A and field set B. For example, if table1 has field set A = {a1,a2,a3,a4,a5,a6} and table2 has field set B = {a1,a2,a3,a24,a25}, then |A∩B| is 3, min{|A|,|B|} is 5, and the overlap similarity sim is 0.6.

[0040] Specifically, if the number of fields in a field set is too small, or the number of fields in the min{|A|,|B|} field is too small, it can lead to extremely large or small values ​​when calculating the overlap similarity between data tables. However, such overlap similarity calculations contain significant errors. Therefore, a minimum field number threshold can be set to filter the data based on actual business needs. This allows calculations to be performed only on field sets with a field number not less than the minimum threshold. For example, a minimum field number threshold of 3 can be set, meaning the field set must contain at least 3 fields.

[0041] Specifically, after determining the overlap similarity between data tables, data transmission relationships between data tables with an overlap similarity less than the preset similarity threshold can be eliminated based on a preset similarity threshold and overlap similarity. This allows for the filtering and selection of data transmission relationships between multiple data tables, resulting in a filtered list of data transmission relationships. See Table 2 for an example.

[0042] Table 2

[0043] Input table Fields in the input table Output table Fields of the output table Overlapping similarity Remark table1 a1, a2, a3, a4, a5, a6 table2 a1, a2, a3, a24, a25 equal to θ reserve table1 a1, a2, a3, a4, a5, a6 table9 a2, a3, a9 Greater than θ reserve table2 a1, a2, a3, a24, a25 table3 a2, a3, a24, a35, a36 equal to θ reserve table2 a1, a2, a3, a24, a25 table4 a41,a2,a3 Greater than θ reserve table3 a2, a3, a24, a35, a36 table6 a2, a3, a61 Greater than θ reserve table4 a41,a2,a3 table5 a2, a3, a53, a54, a55 Greater than θ reserve table5 a2, a3, a53, a54, a55 table9 a2, a3, a9 Greater than θ reserve table7 a2, a3, a71 table5 a2, a3, a53, a54, a55 Greater than θ reserve table7 a2, a3, a71 table8 a2, a3, a82, a83, a84 Greater than θ reserve table8 a2, a3, a82, a83, a84 table6 a2, a3, a61 Greater than θ reserve table6 a2, a3, a61 table9 a2, a3, a9 Greater than θ reserve table5 a2, a3, a53, a54, a55 table8 a2, a3, a82, a83, a84 Less than θ Eliminate

[0044] In Table 2, θ represents a preset similarity threshold. For example, the preset similarity threshold θ can be set to 0.6. This would allow data transmission relationships between data tables with an overlap similarity less than 0.6 to be eliminated, while retaining data transmission relationships between data tables with an overlap similarity greater than or equal to 0.6.

[0045] Furthermore, data transfer relationships between data tables may include self-dependent data transfer relationships. A self-dependent data transfer relationship refers to a data transfer relationship that relates to exactly one data table, meaning both the input and output tables are the same table. When filtering and selecting data transfer relationships between the acquired data tables, self-dependent data transfer relationships can be eliminated, thereby further improving the efficiency of subsequent clustering operations.

[0046] Furthermore, there may be two data transfer relationships between data tables that are in opposite states. When filtering and selecting data transfer relationships between the acquired data tables, two data transfer relationships in opposite states can be eliminated, thereby further improving the efficiency of subsequent clustering operations. For example, if the data transfer relationship between tables table1 and table3 is that table1 points to table3 and table3 points to table1 (i.e., table1 is both an input table and an output table for table3), then the data transfer relationship between tables table1 and table3 can be eliminated. It should be noted that it is the data transfer relationship between tables table1 and table3 that is eliminated, not the relationship between tables table1 and table3 themselves.

[0047] Step S120: Generate a read-write relationship graph based on multiple data tables and the data transmission relationships between them; wherein, the read-write relationship graph includes graph nodes representing data tables and edge data representing the data transmission relationships between data tables.

[0048] In some cases, after obtaining multiple data tables and the data transfer relationships between them, a read-write relationship graph can be constructed based on these data tables and the data transfer relationships between them, which can then be used for subsequent clustering and wide table construction.

[0049] In this implementation, a read / write relationship graph can be generated based on multiple data tables and the data transfer relationships between them. For details, please refer to [link to documentation]. Figure 3 , Figure 3 The read / write relationship graph of the data tables provided in this embodiment can be represented by the data tables as graph nodes, and the data transmission relationships between the data tables as edge data. The direction of the edge data points from the input table to the output table within the data table.

[0050] Step S130: Perform clustering operations on the graph nodes of the read-write relationship graph to obtain multiple clusters. Each cluster includes a portion of the data tables from multiple data tables. Each cluster includes a base table and a derived table generated from the base table. The base table has multiple base fields, and the derived table has at least one derived field. The base fields of the base table in each cluster form a base field set, and the derived fields of the derived table in each cluster form a derived field set.

[0051] In this embodiment, please refer to Figure 4 , Figure 4This diagram illustrates the clustering results of the data tables provided in this embodiment. The Louvain community detection algorithm can be used to perform clustering operations on the graph nodes of the read-write relationship graph, resulting in multiple clusters. Specifically, the edge data of the read-write relationship graph, i.e., the data transmission relationships between data tables, can be utilized, and the Louvain community detection algorithm can be used to perform clustering operations on the graph nodes (data tables) in the read-write relationship graph, resulting in multiple clusters. This allows data tables to be divided into different clusters based on the data transmission relationships between them, thus achieving grouping of data tables based on their lineage. For example, each cluster can have a corresponding cluster label or cluster number used as a distinguishing identifier, ensuring that data tables within the same cluster have the same cluster number. The cluster number corresponding to a data table can be used to determine the cluster to which the data table belongs.

[0052] In this embodiment, the total number of edge data in the read-write relationship graph of the data table can exceed 5 million, and the distribution of graph nodes can be uneven. By dividing the data table fields into different groups based on the data transmission relationship between data tables and the lineage relationship of data tables, and mining the field characteristics of the groups, it is possible to effectively extract the information of effective graph nodes from the massive number of graph nodes representing the data table, thereby effectively optimizing the field processing process.

[0053] During the process of clustering graph nodes in the read-write relationship graph based on the Louvain community discovery algorithm, the graph node or edge data in the read-write relationship graph can change dynamically. For example, the addition or deletion of data tables or graph nodes can cause the clusters to become larger or smaller, and the cluster to which the data table belongs will also change accordingly.

[0054] The Louvain community detection algorithm can cluster graph nodes representing a data table based on modularity. Modularity can be used to determine the quality of cluster partitioning. Therefore, when performing clustering based on modularity, the Louvain community detection algorithm can aim to maximize modularity, resulting in higher similarity among graph nodes within the same cluster and lower similarity among graph nodes outside the same cluster.

[0055] For example, the modularity Q can be determined using Formula 2.

[0056]

[0057] Among them, A ij k represents the weight of the edge between graph node i and graph node j. i It is the sum of the weights of all edges connected to graph node i, c i σ(c) is the cluster number of graph node i. i ,cj The function returns 1 if graph node i and graph node j are in the same cluster, and 0 otherwise.

[0058] The process of clustering graph nodes in a read-write relationship graph based on the Louvain community discovery algorithm can include a first stage and a second stage, and the first and second stages can be iterated.

[0059] The first stage can include continuously traversing the graph nodes in the read-write relationship graph. Specifically, based on each graph node, define one cluster with N nodes, initialize N clusters, then move the graph node to the cluster of its neighboring graph nodes, and determine which neighbor community the graph node should merge into by maximizing the modularity gain, until all graph nodes no longer change.

[0060] For example, the modularity gain ΔQ can be determined using Formula 3.

[0061]

[0062] Where, k i,in It is the sum of the weights of all edges between graph node i and all graph nodes i to be moved into another cluster, ∑ tot It is the sum of the weights of the edges connecting graph node i to all graph nodes in the read-write relationship graph.

[0063] The second stage may include: after completing the first round of iterations, assigning each graph node to a specific cluster, and then using these specific clusters as large graph nodes, connecting the large graph nodes to form a new relational graph. If the modularity Q does not change after one round of iterations, the iteration can be stopped, the clustering can be completed, and multiple clusters can be obtained.

[0064] Specifically, to avoid the clusters becoming too large, i.e., the number of graph nodes representing data tables contained in the clusters being too many, which could lead to failure to converge in a timely manner, a threshold for the number of clustering layers can be set during clustering.

[0065] In this embodiment, the set of basic fields for each cluster can be determined for the base table in each cluster, and the set of derived fields for each cluster can be determined for the derived table in each cluster.

[0066] In some cases, please refer to Figure 5 , Figure 5This diagram illustrates the data table classification method provided in this embodiment. After performing clustering operations on the graph nodes representing the data tables to obtain multiple clusters, the data tables in each cluster can be classified. Specifically, based on the lineage data in the metadata of the data tables, the data tables in each cluster can be divided into base tables and derived tables. The base tables can refer to data tables from the source layer, and the derived tables can refer to data tables generated based on the base tables.

[0067] Specifically, whether a table is a base table can be determined based on its table name. For example, if table1 is named ods.ods_abc_def_df and table2 is named rcdm_rcf.rcf_abc_def_df, then table1 can be designated as the base table and table2 as the derived table.

[0068] Specifically, the determination of whether a table is a base table can also be based on the data transfer relationships between data tables. For example, a data table that is only used as an input table can be identified as a base table; that is, a data table that is only read in the read-write relationship of data tables can be identified as a base table. Otherwise, it can be identified as a derived table.

[0069] In some cases, the field sets of a data table can be partitioned after the data table's field set has been determined and the data table has been partitioned. In this embodiment, the field sets of the base tables in each cluster can be merged to obtain the base field set of each cluster. The field sets of the derived tables in each cluster can be merged to obtain the derived field set of each cluster.

[0070] Specifically, fields in the field set of the base table can be designated as base fields, and fields in the field set of the derived table can be designated as derived fields. This allows us to distinguish the source of fields in the data table, making it easier to filter and merge the derived field set later. This can solve the problems of duplicate processing of derived fields, disordered derivation leading to duplicate data storage, and waste of storage resources.

[0071] For example, please refer to Figure 5 Based on Table 3, the data tables in the cluster can be divided into base tables (tables 1 and 7) and derived tables (tables 2, 3, 4, 5, 6, 8, and 9). Then, the field sets in the base table (table 1) and table 7 can be merged to obtain the base field set of the cluster. Similarly, the field sets in tables 2, 3, 4, 5, 6, 8, and 9 can be merged to obtain the derived field sets of the cluster.

[0072] Table 3

[0073]

[0074]

[0075] Step S140: Combine the basic fields in the basic field set of each cluster with the derived fields in the derived field set to form a wide table corresponding to each cluster.

[0076] In some cases, the number of derived fields in the derived field set may be large. Based on the actual business situation, the derived field set can be filtered to obtain a pre-processed field set, and a wide table can be generated based on the base field set and the pre-processed field set.

[0077] In this embodiment, the derived field set of each cluster in multiple clusters can be filtered according to preset filtering rules to obtain a pre-processed field set. The basic fields in the basic field set of each cluster and the pre-processed fields in the pre-processed field set are then combined to form a wide table corresponding to each cluster. The preset filtering rules may include at least one of the following: if the out-degree of a derived field in the derived field set is greater than a preset out-degree threshold, the derived field is determined as a pre-processed field; if the occurrence frequency of a derived field in the derived field set is greater than a preset repetition frequency threshold, the derived field is determined as a pre-processed field; if the downstream application model of a derived field in the derived field set is a predictive model, the derived field is determined as a pre-processed field.

[0078] Specifically, the in-degree and out-degree of the data tables in each cluster can be calculated, where in-degree and out-degree can be combined. For example, in a read-write graph, the edge data representing the data transfer relationship between data tables is directional, pointing from the input table to the output table. The out-degree of a data table can refer to the number of times it serves as the input table to other data tables, and the in-degree can refer to the number of times it serves as the output table to other data tables. In other words, the out-degree of a data table can be the number of edges pointing from a graph node representing that data table to other graph nodes, and the in-degree can be the number of edges pointing from other graph nodes to graph nodes representing that data table.

[0079] Based on the in-degree and out-degree of the data table, the frequency of occurrence of derived fields, and whether the downstream is a prediction model, derived fields from data tables whose out-degree exceeds the out-degree threshold, derived fields in the same derived field name appearing more than the repetition threshold in the same cluster, or derived fields from data tables whose downstream is a model prediction are used as pre-processed fields to obtain the pre-processed field set for the wide table. Then, wide table fields are added or the wide table is reconstructed based on the existing wide table.

[0080] As an example, please refer to Table 4. Here, it is assumed that the out-degree threshold is 2, the threshold for the number of times a derived field is repeated is 2, and there is no data table downstream of the data table that can be predicted by the model. In this case, the field set can be pre-processed.

[0081] Table 4

[0082] Basic fields Derived fields Pre-processed fields a1 a35 a24 a2 a36 a25 a3 a41 a4 a53 a5 a54 a6 a55 a71 a61 a82 a83 a84 a9

[0083] Specifically, after determining the set of basic fields for a cluster, the dimensional categories of that cluster can be obtained based on the basic fields in that set. In other words, the basic fields can be used as dimensional fields. A wide table is then constructed based on the dimensional fields and the pre-processed fields from the set of pre-processed fields. Specifically, the dimensional categories determined by the basic fields of the cluster can be combined with the pre-processed fields from the set of pre-processed fields in clusters with the same dimensional categories to form a wide table. Constructing a wide table can involve adding fields to an existing wide table or reconstructing the wide table.

[0084] As an example, please continue to refer to Table 4 and Figure 5 The most frequently occurring basic fields are a2 and a3. If a2 and a3 represent applyno and contra_no, then we can construct a wide table with dimensions of applyno and contra_no, or add a24 and a25 to the existing wide table with dimensions of applyno and contra_no.

[0085] This specification describes several implementation methods that utilize the Louvain community discovery algorithm to automatically partition data tables based on modularity. Furthermore, based on the lineage data contained in the data table metadata, it further divides the tables into base tables and derived tables within the same cluster. Then, using the field metadata information of the tables within the cluster, it can automatically extract base fields and derived fields. The derived fields of the clustered data tables are then filtered based on actual business needs to obtain pre-processed fields for the wide tables. The base fields serve as dimension fields. By combining existing dimension wide tables to add fields or reconstruct wide tables, it achieves the goal of merging original duplicate derived field wide tables and constructing new field wide tables, reducing unordered generation and redundant storage of field wide tables. Simultaneously, it improves the efficiency of wide table construction.

[0086] Please see Figure 6 This specification provides an apparatus for generating wide tables. The apparatus may include an acquisition module, a graphing module, a clustering module, and a processing module.

[0087] The acquisition module is used to acquire multiple data tables and the data transmission relationships between the multiple data tables.

[0088] A graph construction module is used to generate a read-write relationship graph based on the multiple data tables and the data transmission relationships between the multiple data tables; wherein, the read-write relationship graph includes graph nodes representing data tables and edge data representing data transmission relationships between data tables;

[0089] The clustering module is used to perform clustering operations on the graph nodes of the read-write relationship graph to obtain multiple clusters. Each cluster includes a portion of the data tables in the multiple data tables. The data tables in each cluster are divided into a base table and a derived table generated based on the base table. The base table has multiple base fields, and the derived table has at least one derived field. The base fields of the base table in each cluster form a base field set, and the derived fields of the derived table in each cluster form a derived field set.

[0090] The processing module is used to combine the basic fields in the basic field set of each cluster with the derived fields in the derived field set to form a wide table corresponding to each cluster.

[0091] The specific functions and effects of the wide table generation device can be explained by referring to other embodiments in this specification, and will not be repeated here. Each module in the wide table generation device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0092] Please see Figure 7 In some embodiments, a computer device may be provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the wide table generation method of the embodiments described above.

[0093] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, causes the computer to perform the wide table generation method in any of the above embodiments.

[0094] This specification also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the wide table generation method in any of the above embodiments.

[0095] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments described herein, and are not intended to limit the scope of the invention.

[0096] It is understood that in the various embodiments described in this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments described in this specification.

[0097] It is understood that the various implementation methods described in this specification can be implemented individually or in combination, and the implementation methods in this specification are not limited in this respect.

[0098] Unless otherwise stated, all technical and scientific terms used in the embodiments of this specification have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this specification. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0099] It is understood that the processor in the embodiments of this specification can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this specification. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this specification can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0100] It is understood that the memory in the embodiments of this specification may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0101] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0103] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0105] In addition, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0106] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this specification, in essence, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The above description is merely a specific embodiment of this specification, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this specification should be included within the scope of protection of this specification. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for generating a wide table, characterized in that, The method includes: Obtain multiple data tables and the data transmission relationships between the multiple data tables; A read-write relationship graph is generated based on the multiple data tables and the data transmission relationships between them; wherein, the read-write relationship graph includes graph nodes representing data tables and edge data representing data transmission relationships between data tables; Clustering operations are performed on the graph nodes of the read-write relationship graph to obtain multiple clusters; wherein, each cluster includes a portion of the data tables in the multiple data tables, and the data tables included in each cluster are divided into a base table and a derived table generated based on the base table; the base table has multiple base fields, and the derived table has at least one derived field; wherein, the base fields of the base table in each cluster form a base field set, and the derived fields of the derived table in each cluster form a derived field set; The basic fields in the basic field set of each cluster and the derived fields in the derived field set are combined to form a wide table corresponding to each cluster.

2. The method according to claim 1, characterized in that, Before generating the read-write relationship graph based on the plurality of data tables and the data transfer relationships between the plurality of data tables, the method further includes: Determine the set of fields for each of the plurality of data tables; Based on the data transmission relationship between the multiple data tables and the field set of each data table, the similarity between the multiple data tables is determined; Based on a preset similarity threshold and the similarity between the multiple data tables, the data transmission relationships between the multiple data tables are filtered to obtain the filtered data transmission relationships between the multiple data tables.

3. The method according to claim 2, characterized in that, The step of filtering the data transmission relationships between the multiple data tables based on a preset similarity threshold and the similarity values ​​between the multiple data tables includes: If the similarity value between any two data tables is less than the preset similarity threshold, the data transmission relationship between the two data tables corresponding to the similarity value is eliminated.

4. The method according to claim 1, characterized in that, Before generating the read-write relationship graph based on the plurality of data tables and the data transfer relationships between the plurality of data tables, the method further includes: If a self-dependent data transmission relationship exists among the data transmission relationships between the multiple data tables, the self-dependent data transmission relationship is eliminated; wherein, the self-dependent data transmission relationship is a data transmission relationship that only relates to one of the multiple data tables.

5. The method according to claim 1, characterized in that, The step of acquiring multiple data tables and the data transmission relationships between the multiple data tables includes: Obtain the plurality of data tables and the lineage data of each data table; wherein the lineage data is used to represent the lineage relationship between the data tables; Based on the lineage data representing the lineage relationships in each data table, the data transmission relationships between the multiple data tables are determined.

6. The method according to claim 1, characterized in that, For each cluster's base table and derived table, determine the base field set and derived field set for each cluster, including: The basic fields corresponding to the basic tables in each cluster are merged to obtain the basic field set for each cluster; The derived fields corresponding to the derived tables in each cluster are merged to obtain the set of derived fields for each cluster.

7. The method according to claim 6, characterized in that, The generation of a wide table corresponding to each cluster based on the base field set and the derived field set for each cluster includes: Based on preset filtering rules, the derived field set of each cluster is filtered to obtain a pre-processed field set; Combine the basic fields in the basic field set of each cluster with the pre-processed fields in the pre-processed field set to form a wide table corresponding to each cluster; The preset filtering rules include at least one of the following filtering rules: If the out-degree of a derived field in the set of derived fields is greater than a preset out-degree threshold, the derived field is determined as a pre-processing field. If the number of occurrences of a derived field in the set of derived fields exceeds a preset repetition threshold, the derived field is identified as a pre-processed field. When the downstream application model of a derived field in the set of derived fields is a predictive model, the derived field is identified as a pre-processed field.

8. A wide table generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire multiple data tables and the data transmission relationships between the multiple data tables. A graph construction module is used to generate a read-write relationship graph based on the multiple data tables and the data transmission relationships between the multiple data tables; wherein, the read-write relationship graph includes graph nodes representing data tables and edge data representing data transmission relationships between data tables; The clustering module is used to perform clustering operations on the graph nodes of the read-write relationship graph to obtain multiple clusters. Each cluster includes a portion of the data tables in the multiple data tables. The data tables in each cluster are divided into a base table and a derived table generated based on the base table. The base table has multiple base fields, and the derived table has at least one derived field. The base fields of the base table in each cluster form a base field set, and the derived fields of the derived table in each cluster form a derived field set. The processing module is used to combine the basic fields in the basic field set of each cluster with the derived fields in the derived field set to form a wide table corresponding to each cluster.

9. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images