Data warehouse e-r diagram subgraph partitioning method and related devices

Abstract

The application discloses a data warehouse E-R graph subgraph division method and related equipment, and relates to the technical field of data processing, comprising: obtaining the association relationship between each single table in the data warehouse, using a preset first large language model to pre-divide the E-R graph based on the association relationship to obtain a first division result, and using a Louvain clustering algorithm to cluster and divide the E-R graph to obtain a second division result, comparing the first division result and the second division result using a preset second large language model to obtain a comparison result, and generating a target division result using the preset second large language model based on the comparison result. The application fuses the semantic understanding ability of the large language model and the topological analysis ability of the Louvain clustering algorithm, and uses the large model to compare and fuse the two division results, taking into account the business semantics and structural association characteristics, effectively solving the problems of division deviation from the business and fuzzy edge attribution, and realizing more accurate subgraph division.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for partitioning ER diagrams in a data warehouse and related equipment. Background Technology

[0002] In the field of data warehousing, the number of data tables typically ranges from hundreds to thousands, resulting in a massive and complex ER diagram that is difficult to access and has poor usability. Subdividing the ER diagram into subgraphs and identifying the core tables is an effective way to improve its usability. Currently, there are two main approaches: one relies on manual subgraph partitioning, business scope labeling, and core table identification, which is costly to maintain and requires strict management processes and personnel experience; the other uses graph clustering algorithms (such as the Louvain algorithm) to treat the ER diagram as an undirected graph for clustering. However, existing methods have significant drawbacks: clustering is based solely on statistical information without considering the actual business content of the data tables, leading to subgraph partitioning results that deviate from the actual business; edge nodes between subgraphs are inaccurately partitioned due to their proximity to adjacent subgraphs, resulting in blurred subgraph boundaries; furthermore, adding or deleting a single table requires re-partitioning the entire ER diagram, making updates and maintenance difficult and compromising the consistency of the partitioning results. Summary of the Invention

[0003] The main purpose of this application is to provide a method and related equipment for subgraph partitioning of ER diagrams in a data warehouse, aiming to solve the technical problem of how to achieve accurate subgraph partitioning.

[0004] To achieve the above objectives, this application proposes a method for partitioning ER diagrams in a data warehouse, the method comprising: Obtain the relationships between individual tables in the data warehouse; Based on the aforementioned relationship, the ER graph is pre-divided using a preset first large language model to obtain a first partitioning result, and the ER graph is clustered using the Louvain clustering algorithm to obtain a second partitioning result; The first segmentation result and the second segmentation result are compared using a preset second largest language model to obtain a comparison result. Based on the comparison result, the target segmentation result is generated using the preset second largest language model.

[0005] In one embodiment, the step of pre-segmenting the ER graph using a preset first large language model based on the association relationship to obtain a first segmentation result further includes: Perform a deduplication operation on the aforementioned relationships to remove redundant records that are mutually inverse; Based on the deduplicated relationships, the ER graph is pre-partitioned using the first pre-defined language model to obtain the first partitioning result.

[0006] In one embodiment, the step of pre-segmenting the ER graph using a preset first large language model based on the deduplicated association relationships to obtain a first segmentation result further includes: Obtain the Chinese and English names of each individual table, and concatenate the Chinese and English names of the tables and the deduplicated relationships into semantic prompt text according to a preset format; The semantic prompt text is semantically parsed using a preset first major language model to obtain the business semantic category corresponding to each single table; Single tables belonging to the same business semantic category are divided into the same subgraph to form multiple first subgraphs, and the association weight of each single table in each first subgraph is determined. Set the single table with the highest association weight in each first subgraph as the first core table of that first subgraph; Based on multiple first subgraphs and the first core table corresponding to each first subgraph, the first partitioning result is obtained.

[0007] In one embodiment, the step of using the Louvain clustering algorithm to cluster the ER graph and obtain a second partitioning result further includes: Based on the deduplicated relationships, each individual table is used as a node to construct an undirected weighted network; The Louvain clustering algorithm is used to cluster the undirected weighted network, and each single table is aggregated into different communities based on the density of node connections in the undirected weighted network. Each community is treated as a second subgraph, resulting in multiple second subgraphs. The weighting degree of each single table in each second subgraph is then determined. The single table with the highest weight in each second subgraph is designated as the second core table of that second subgraph. The second partitioning result is obtained based on multiple second subgraphs and the second core table corresponding to each map.

[0008] In one embodiment, the step of comparing the first segmentation result and the second segmentation result using a preset second language model to obtain a comparison result further includes: Based on the preset second language model, each first subgraph in the first partitioning result is mapped and compared with each second subgraph in the second partitioning result to determine whether each single table is partitioned into a subgraph with the same subgraph affiliation in the first partitioning result and the second partitioning result. For a single table whose subgraphs belong to the same subgraph in the first partitioning result and the second partitioning result, mark the single table as a node with consistent subgraph affiliation and mark the corresponding subgraph as a consistent subgraph; For a single table whose subgraph affiliation is inconsistent in the first partitioning result and the second partitioning result, the single table is marked as a boundary affiliation conflict node. The comparison results are obtained based on the nodes with consistent affiliation and the nodes with conflicting affiliation at the boundary.

[0009] In one embodiment, the step of generating target segmentation results using a preset second-largest language model based on the comparison results further includes: Obtain the first candidate subgraph attribution information of the boundary attribution conflict node in the first partitioning result and its corresponding first business scope description, obtain the second candidate subgraph attribution information of the boundary attribution conflict node in the second partitioning result and its corresponding second business scope description, and obtain the table name information, Chinese table name and English table name of the boundary attribution conflict node; The Chinese and English names of the tables for each boundary-attributed conflicting node, the attribution information of the first candidate subgraph, the description of the first business scope, the attribution information of the second candidate subgraph, and the description of the second business scope are concatenated into a conflict judgment prompt text. The contradiction judgment prompt text is processed using a preset second major language model to determine the first degree of matching between the business semantics of each boundary-attributed contradiction node and the first business scope description, and the second degree of matching between the business semantics of the node and the second business scope description. Based on the first matching degree and the second matching degree, the candidate subgraph with the highest matching degree is set as the target subgraph; The target subgraph and the consistent subgraph are merged to obtain the target partitioning result.

[0010] In one embodiment, after the step of generating the target segmentation result using a preset second-largest language model based on the comparison result, the method further includes any one of the following: In response to a request to add a single table, the system obtains the table name and field definitions of the new table. By parsing the SQL script in the data warehouse, it determines the association between the new table and the existing tables. If the new table is associated with only one subgraph in the target partitioning result, the new table is added to the associated subgraph. If the new table is associated with multiple subgraphs in the target partitioning result, a target subgraph is selected for the new table based on the target partitioning result, association status, table name information, and field definitions, using a preset third semantic model. In response to a single table deletion request, determine whether the single table to be deleted involves core business data. If not, delete the single table directly from the subgraph corresponding to the single table to be deleted. If it does, update the data warehouse based on the single table to be deleted and update the association relationships between the single tables in the data warehouse. Based on the updated data warehouse and the updated association relationships, return to the step of obtaining the association relationships between the single tables in the data warehouse and regenerate a new target partitioning result. In response to the repartitioning instruction, the relationships between individual tables in the data warehouse are updated. Based on the updated relationships, the step of obtaining the relationships between individual tables in the data warehouse is returned, and a new target partitioning result is regenerated. The repartitioning instruction is triggered periodically based on a preset time threshold.

[0011] Furthermore, to achieve the above objectives, this application also proposes a data warehouse ER diagram subgraph partitioning device, which includes: The acquisition module is used to acquire the relationships between various single tables in the data warehouse; The partitioning module is used to pre-partition the ER graph based on the association relationship using a preset first large language model to obtain a first partitioning result, and then use the Louvain clustering algorithm to cluster the ER graph to obtain a second partitioning result; The generation module is used to compare the first segmentation result and the second segmentation result using a preset second language model to obtain a comparison result, and based on the comparison result, to generate a target segmentation result using the preset second language model.

[0012] In addition, to achieve the above objectives, this application also proposes a data warehouse ER diagram subgraph partitioning device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data warehouse ER diagram subgraph partitioning method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data warehouse ER diagram subgraph partitioning method as described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data warehouse ER diagram subgraph partitioning method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application discloses a method and related equipment for subgraph partitioning of an ER diagram in a data warehouse, relating to the field of data processing technology. In contrast to related technologies, where clustering partitioning is based solely on statistical information without considering the actual business content of the data tables, resulting in subgraph partitioning results deviating from actual business operations, inaccurate partitioning of edge nodes between subgraphs due to their proximity to adjacent subgraphs, and blurred subgraph boundaries, the entire ER diagram must be repartitioned when a single table is added or deleted, making updates and maintenance difficult and compromising the consistency of partitioning results. In this application, firstly, the relationships between individual tables in the data warehouse are obtained. Then, based on these relationships, a preset first language model is used to prepartition the ER diagram, obtaining a first partitioning result. Next, the Louvain clustering algorithm is used to cluster the ER diagram, obtaining a second partitioning result. Finally, a preset second language model is used to compare the first and second partitioning results, obtaining a comparison result. Based on the comparison result, a target partitioning result is generated using the preset second language model.

[0016] Understandably, this application integrates the semantic understanding capabilities of a large language model with the topological analysis capabilities of the Louvain clustering algorithm, and uses the large model to compare and fuse the two partitioning results. This approach takes into account both business semantics and structural association features, effectively solving the problems of partitioning deviating from business and ambiguous edge affiliation, and achieving more accurate subgraph partitioning. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the data warehouse ER diagram subgraph partitioning method in this application. Figure 2 This is a flowchart illustrating the second embodiment of the data warehouse ER diagram subgraph partitioning method in this application. Figure 3 This is a flowchart illustrating the third embodiment of the data warehouse ER diagram subgraph partitioning method in this application. Figure 4 This is a schematic diagram of the module structure of the data warehouse ER diagram subgraph partitioning device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the data warehouse ER diagram subgraph partitioning method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution in this application embodiment is: In this embodiment, for ease of description, the following description will focus on the data warehouse ER diagram subgraph partitioning device as the execution subject.

[0024] Based on this, embodiments of this application provide a method for partitioning subgraphs of a data warehouse ER diagram, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data warehouse ER diagram subgraph partitioning method of this application.

[0025] In this embodiment, the data warehouse ER diagram subgraph partitioning method includes steps S100~S300: Step S100: Obtain the relationships between the individual tables in the data warehouse; It should be noted that a data warehouse refers to a subject-oriented, integrated, relatively stable collection of data that reflects historical changes and is used to support enterprise decision analysis.

[0026] It should be noted that a single table refers to a two-dimensional table structure in a data warehouse that stores business data. It consists of rows and columns, and each table corresponds to a specific data entity or business object.

[0027] Relationships refer to the connections between individual tables in a data warehouse based on foreign keys, business logic, or data lineage, and are used to characterize the data dependencies and business relationships between tables.

[0028] In this embodiment, metadata information for all individual tables, including table names, field definitions, primary keys, and foreign key constraints, is extracted from the data warehouse's metadata management system or data dictionary. By parsing the metadata information, the primary and foreign key dependencies between individual tables are identified. Simultaneously, lineage analysis during data flow is combined to supplement relationships based on business logic, ultimately constructing a complete set of relationships reflecting all table relationships.

[0029] Understandably, this step provides a data foundation for subsequent subgraph partitioning, ensuring that the correlation information on which the partitioning is based is comprehensive and accurate, and that the partitioning results have a reasonable structural basis.

[0030] Step S200: Based on the association relationship, the ER graph is pre-divided using a preset first large language model to obtain a first partitioning result, and the ER graph is clustered using the Louvain clustering algorithm to obtain a second partitioning result; It should be noted that the first pre-trained language model refers to a large-scale language model that has natural language understanding and semantic reasoning capabilities, and can parse the business semantic information contained in the table name, field name, and business comments of the data table.

[0031] An ER diagram is an entity-relationship diagram used to describe the relationships between individual tables in a data warehouse. In this application, the ER diagram is composed of all individual tables as nodes and the relationships between tables as edges.

[0032] Pre-partitioning refers to using the ability of a large language model to understand business semantics, and performing preliminary sub-partitioning of the ER diagram based on the semantic information of table names and the relationships between tables without human intervention.

[0033] The first partitioning result refers to the partitioning result output after pre-partitioning by the preset first major language model, which includes the composition of each subgraph, the business scope description of each subgraph, and the core table information of each subgraph.

[0034] Louvain clustering algorithm is a community discovery algorithm based on modularity optimization. It iteratively optimizes the aggregation of nodes in the network into communities, making the connections within communities as close as possible and the connections between communities as sparse as possible.

[0035] Clustering partitioning refers to the unsupervised topological clustering of ER graphs using the Louvain clustering algorithm, grouping structurally closely related single tables into the same subgraph.

[0036] The second partitioning result refers to the partitioning result output after clustering by the Louvain clustering algorithm, which includes the composition of each subgraph and the core table information of each subgraph.

[0037] In this embodiment, on the one hand, the Chinese and English names of each table and the relationships between each table are concatenated into semantic prompt text according to a preset format, and input into a preset first language model. The model performs semantic parsing on the business meaning of each table, identifies a set of tables with similar business semantics, and performs constraint verification on the set according to the relationship. Tables belonging to the same business semantic category are divided into the same subgraph to form the first division result. At the same time, the business scope description of each subgraph is output, and the core table of each subgraph is identified according to the association weight.

[0038] On the other hand, the relationships between the extracted individual tables are cleaned and deduplicated, redundant bidirectional and reciprocal relationship records are removed, and only one valid relationship record is retained. An undirected weighted network is constructed based on the cleaned relationship, with each individual table as a node, the valid relationship record as an edge, and the relationship frequency or strength as a weight. The undirected weighted network is input into the Louvain clustering algorithm for community detection. By maximizing the modularity, each individual table is aggregated into different communities, and each community is a subgraph. The weight degree of each node in each subgraph is calculated, and the node with the highest weight degree is taken as the core table of the subgraph. The second partitioning result is output.

[0039] Understandably, this step utilizes two different partitioning methods—the large language model and the Louvain clustering algorithm—in parallel processing to obtain semantic-based business-oriented partitioning results and topology-based structure-oriented partitioning results, providing multi-dimensional references for subsequent result fusion.

[0040] The step of pre-segmenting the ER graph using a preset first large language model based on the association relationship to obtain the first segmentation result further includes steps S210~S220: Step S210: Perform a deduplication operation on the association relationship to remove redundant association records that are mutually inverse. It should be noted that the deduplication operation refers to screening and cleaning the records of relationships between the extracted individual tables, removing duplicate and redundant records, and retaining the effective relationships that are of practical value for subgraph partitioning.

[0041] Bidirectional reciprocity means that there are two records with opposite directions between two single tables. That is, there are records in table A that are related to B, and there are records in table B that are related to A. In terms of business meaning, they actually point to the same related fact.

[0042] Redundant association records refer to the phenomenon in multi-table relationship descriptions where the connection between the same table is recorded multiple times due to data modeling methods or metadata extraction rules. Bidirectional reversal is a typical example. Such records introduce unnecessary connection redundancy in the topology of the ER graph, interfering with the accuracy of subsequent algorithms in analyzing the graph structure.

[0043] In this embodiment, all inter-table relationship records extracted in step S100 are obtained. Each relationship record is checked to see if it has a record with the opposite direction. That is, it is determined whether there is a relationship record from table A to table B, and at the same time, there is a relationship record from table B to table A. For the identified bidirectional reversible relationship record pairs, one of them is retained as a valid relationship record, and its reverse duplicate record is deleted, so that only a unique unidirectional relationship is retained between the same pair of single tables. In terms of retention strategy, the direction that is more reasonable in business semantics can be retained, or the record with a higher relationship weight value can be retained. After deduplication, a cleaned relationship set is output, which no longer contains redundant bidirectional reversible records.

[0044] Understandably, this step eliminates topological redundancy caused by bidirectional associations in the ER graph, making the graph structure simpler and clearer, avoiding ambiguity or bias in the determination of node affiliation by bidirectional edges during subsequent pre-partitioning, and improving the data quality foundation of pre-partitioning.

[0045] Step S220: Based on the deduplicated associations, the ER graph is pre-partitioned using a preset first large language model to obtain the first partitioning result.

[0046] It should be noted that the deduplicated association relationship refers to the unique set of valid association records retained after the cleaning process in step S210. The connection relationship between each single table is presented in a unidirectional and non-repeating form.

[0047] In this embodiment, the deduplicated association relationships output in step S210 are used as input. The Chinese and English names of each table, along with the deduplicated valid association relationships, are concatenated into semantic prompt text according to a preset format and input into a preset first language model. This model performs semantic parsing on the business meaning of each table, identifies a set of tables with similar business semantics, and performs association constraint verification on this set based on the deduplicated association relationships to ensure that there is indeed a direct association relationship between tables assigned to the same subgraph or a reachable association path through an intermediate table. After semantic parsing and constraint verification, based on the semantic parsing results output by the model, tables belonging to the same business semantic category are assigned to the same subgraph, forming the first division result. At the same time, a business scope description of each subgraph is output, and the first core table of each subgraph is identified based on the association weight of each table within its respective subgraph.

[0048] Understandably, this step is based on the effective relationships after removing redundant associations. It utilizes the semantic understanding capabilities of the large language model to complete the pre-partitioning of the ER diagram. While ensuring the accuracy of the association data, it fully leverages the advantages of the large language model in parsing business semantics, so that the first partitioning result is both consistent with business logic and unaffected by redundant associations.

[0049] Specifically, the step of pre-segmenting the ER graph using a preset first large language model based on the deduplicated association relationships to obtain the first segmentation result further includes steps S221~S225: Step S221: Obtain the Chinese and English names of each single table, and concatenate the Chinese and English names of the tables and the deduplicated relationships into semantic prompt text according to a preset format; It should be noted that the Chinese name of the table refers to the business meaning name recorded in the data warehouse, which is used to describe the business entity type corresponding to the table.

[0050] The English name of a table refers to the English name given to the data table in the physical storage of the data warehouse.

[0051] Preset format refers to a predefined text organization structure used to combine the Chinese names of tables, the English names of tables, and the relationships into structured input text according to a specific order and delimiters, so that the large language model can accurately identify the type and boundaries of each part of the information.

[0052] Semantic prompt text refers to the text content generated after being concatenated according to a preset format. It serves as the input to the preset first language model and contains a complete description of the business semantic information and structural association information of each table.

[0053] In this embodiment, all individual tables in the data warehouse are traversed, and the Chinese and English names of each table are read sequentially. Then, the deduplicated set of relationships output in step S210 is obtained. This information is then concatenated according to a preset format: for each individual table, a semantic description of the table is provided using a combination of "Chinese table name" and "English table name," and after the descriptions of all tables, the deduplicated relationships are appended in a list format. Each relationship record is described according to the format "source table, target table, relationship type." After concatenation, a complete semantic prompt text is generated.

[0054] Understandably, this step unifies structured metadata information and unstructured business semantic information into natural language text, providing standardized input for subsequent semantic parsing of the large language model, and ensuring that the model can accurately understand the business meaning of each single table and the relationships between tables.

[0055] Step S222: Use the preset first language model to perform semantic parsing on the semantic prompt text to obtain the business semantic category corresponding to each single table; It should be noted that semantic parsing refers to the process by which a large language model deeply understands and analyzes the input semantic prompt text. This includes identifying business keywords in the names of each table, understanding the business scenario meaning of the relationships between tables, and classifying and judging the business domain corresponding to each individual table based on this.

[0056] Business semantic category refers to the category label obtained by classifying the business functions carried by each table in the data warehouse. Tables belonging to the same business semantic category indicate that they serve the same or similar business areas.

[0057] In this embodiment, the semantic prompt text generated in step S221 is used as input and passed to a preset first large language model. This model performs word-by-word parsing and contextual understanding of the semantic prompt text, identifying the business terms and their corresponding business scenarios contained in the Chinese and English names of each table. Simultaneously, it combines the relationship information described in the prompt text to infer the role of each table within the overall business system. After internal reasoning, the model outputs a business semantic category label for each table.

[0058] Understandably, this step leverages the natural language understanding capabilities of the large language model to classify and label each single table from a business semantic perspective. This ensures that the subsequent subgraph division is not limited to the structural relationships between tables but incorporates deeper business meanings, laying the foundation for business-oriented subgraph division.

[0059] Step S223: Divide single tables belonging to the same business semantic category into the same subgraph to form multiple first subgraphs, and determine the association weight of each single table in each first subgraph; It should be noted that the first subgraph refers to the subgraph structure formed in the pre-partitioning stage by the preset first major language model after aggregating multiple single tables according to business semantic categories. Each first subgraph corresponds to a business semantic category and contains all single tables belonging to that category.

[0060] Association weight is a numerical indicator used to quantify the degree of association between each single table and other single tables in its subgraph. The higher the association weight, the denser the connection between the single table and other tables in the subgraph, and the more important its structural position in the subgraph.

[0061] In this embodiment, the business semantic category corresponding to each single table output in step S222 is obtained. All single tables with the same business semantic category are grouped into the same group to form multiple first subgraphs. Each first subgraph is assigned a business scope description corresponding to its business semantic category. For each first subgraph, the number and strength of association relationships between each single table within it and other single tables within the same subgraph are counted. Combined with the deduplicated association relationship records, the association weight value of each single table within the subgraph is calculated. The association weight is calculated as follows: the number of valid association records between the single table and other single tables within the subgraph is counted, and the results are weighted and summed using the strength coefficient of each association relationship to obtain the final association weight value.

[0062] Understandably, this step completes the initial subgraph division based on business semantics, ensuring that each table in the first subgraph has a unified business domain affiliation, and at the same time, it provides a quantitative basis for the identification of subsequent core tables through the calculation of association weights.

[0063] Step S224: Set the single table with the highest association weight in each first subgraph as the first core table of that first subgraph; It should be noted that the first core table refers to the single table with the highest association weight in each first subgraph during the pre-division stage. It represents the core data table that is most closely related to other tables in that subgraph and occupies a central position in the business chain.

[0064] In this embodiment, each first subgraph formed in step S223 is traversed, and the association weight values ​​calculated in step S223 for each table within that subgraph are read. The association weights of each table are sorted and compared, and the table with the largest association weight value is identified. This table is set as the first core table of the first subgraph, and the table name information of the core table is associated with the corresponding subgraph. The above operation is performed for each first subgraph to ensure that each subgraph has a unique first core table.

[0065] Understandably, this step identifies the core table that occupies a central position in the structure from the subgraph of business semantics, providing a semantic reference for determining the core table in the final target partitioning result.

[0066] Step S225: Based on multiple first subgraphs and the first core table corresponding to each first subgraph, the first partitioning result is obtained.

[0067] It should be noted that the first partitioning result refers to the comprehensive partitioning result output after the complete pre-partitioning process from steps S221 to S224, which includes the composition information of multiple first subgraphs, the business scope description of each first subgraph, and the first core table information corresponding to each first subgraph.

[0068] In this embodiment, the multiple first sub-graphs formed in step S223, along with their respective single-table member information and business scope descriptions, are integrated with the first core table information corresponding to each first sub-graph determined in step S224. The above information is organized according to a preset data structure to form complete pre-division result data, which includes at least the sub-graph identifier, a list of single tables contained in the sub-graph, a sub-graph business scope description, and the first core table identifier of the sub-graph. The integrated data is output as the first division result for use in subsequent comparison steps.

[0069] Understandably, this step systematically integrates the outputs of each pre-divided stage to form a first division result with a complete structure and comprehensive information, providing standardized data input for subsequent comparison and fusion with the second division result.

[0070] Specifically, the step of using the Louvain clustering algorithm to cluster the ER graph and obtain the second partitioning result further includes steps S230~S270: Step S230: Based on the deduplicated association relationships, construct an undirected weighted network using each single table as a node; It should be noted that in an undirected weighted network, a node represents a single data table in the data warehouse and is the basic element that constitutes the network. Each node corresponds to a single table.

[0071] An undirected weighted network is a graph structure consisting of nodes and edges, where an edge connects two nodes, indicating a relationship between two single tables; "undirected" means that the edges have no directionality, that is, the relationship is bidirectional; "weighted" means that each edge is assigned a numerical weight to characterize the tightness or strength of the relationship.

[0072] In this embodiment, based on the deduplicated association relationships output in step S210, each single table in the data warehouse is treated as a node. Each valid association record after deduplication is parsed to identify the two single tables associated with each record, and an edge is established between the corresponding two nodes. Each edge is assigned a weight value based on the association frequency or strength; the higher the association frequency or the stronger the business dependency, the larger the weight value. After traversing all valid association records, the construction of all nodes and edges is completed, forming a complete undirected weighted network.

[0073] Understandably, this step transforms the deduplicated inter-table relationships into a graph data structure, providing network data that meets the input requirements for the subsequent Louvain clustering algorithm, enabling the clustering algorithm to perform community detection based on the connection relationships and weights between nodes.

[0074] Step S240: Use the Louvain clustering algorithm to cluster the undirected weighted network, so as to aggregate each single table into different communities based on the density of node connections in the undirected weighted network. It should be noted that clustering in this step specifically refers to the process of using the Louvain clustering algorithm to discover communities in an undirected weighted network, and iteratively optimizing the nodes in the network to be assigned to different communities.

[0075] It should be noted that in graph theory, a community refers to a set of nodes that are relatively tightly connected within a network and relatively sparsely connected to other parts of the network. In this scheme, a community corresponds to a subgraph, which is a set of single tables that are structurally closely related.

[0076] In this embodiment, the undirected weighted network constructed in step S230 is input into the Louvain clustering algorithm. The algorithm first initializes each node in the undirected weighted network as an independent community. Then, it iteratively traverses all nodes, calculating the change in network modularity after moving each node to the community of its neighboring nodes. If the modularity increases, the move operation is performed. After completing all node move attempts, the algorithm uses the communities formed in the first stage as new nodes to construct a new weighted network, repeating the above iterative process. Through continuous optimization, the iteration terminates when the network modularity no longer increases, ultimately forming multiple communities, each containing several nodes. Nodes within a community are tightly connected, while connections between communities are relatively sparse.

[0077] Understandably, by leveraging the topology analysis capabilities of the Louvain clustering algorithm, a single table can be clustered from the perspective of graph structure to obtain partitioning results based on the tightness of connections, providing a structure-oriented reference for subsequent fusion.

[0078] Step S250: Treat each community as a second subgraph to obtain multiple second subgraphs, and determine the weighting degree of each single table in each second subgraph; It should be noted that the second subgraph refers to the subgraph structure corresponding to each community output by the algorithm during the Louvain clustering partitioning stage. Each second subgraph contains several closely related single tables and is the basic building block of the second partitioning result.

[0079] It should be noted that in graph theory, the weighted degree of a node refers to the sum of the weights of all the edges connecting that node. In this scheme, the weighted degree of a single table is equal to the sum of the weights of the relationships between that table and all other single tables in the undirected weighted network, reflecting the structural importance of that table in the entire network.

[0080] In this embodiment, the communities output by the Louvain clustering algorithm in step S240 are obtained, and each community is defined as a second subgraph. Each second subgraph contains single tables corresponding to all nodes within the community, and a unique subgraph identifier is assigned to each second subgraph. Subsequently, for each single table node in each second subgraph, all connection edges of the node in the undirected weighted network are traversed, the weight value of each edge is read, and the weight values ​​of all edges are summed to obtain the weighted degree of the node in the second subgraph.

[0081] Understandably, this step formally defines the community structure output by the clustering algorithm as a subgraph and provides a quantitative indicator for the subsequent identification of the core table of each subgraph from a structural dimension by calculating the weighted degree.

[0082] Step S260: Take the single table with the highest weight in each second subgraph as the second core table of that second subgraph; It should be noted that the second core table refers to the single table with the highest weight in each second subgraph during the Louvain clustering stage. It represents the data table with the strongest connection to other tables in the same subgraph and occupies a central hub position in the subgraph structure.

[0083] In this embodiment, each second subgraph formed in step S250 is traversed to obtain the weighted degree value of all individual tables within that subgraph. The weighted degrees of each individual table are sorted, and the individual table node with the highest weighted degree value is identified. If multiple individual tables have the same highest weighted degree, the sparsity of the node's connections to other subgraphs can be used as an auxiliary criterion, prioritizing the selection of nodes with fewer connections to external subgraphs as the second core table. The finally determined individual table is set as the second core table of the second subgraph, and the association relationship between the core table information and the corresponding subgraph is recorded. The above operation is performed on all second subgraphs.

[0084] Understandably, this step identifies the structural center table from each cluster subgraph based on topological structure indicators, providing a structural dimension reference for determining the core table in the final target partitioning result.

[0085] Step S270: Based on multiple second sub-graphs and the second core table corresponding to each map, the second partitioning result is obtained.

[0086] It should be noted that the second partitioning result refers to the comprehensive partitioning result output after the complete clustering partitioning process from steps S230 to S260, which includes the composition information of multiple second subgraphs and the second core table information corresponding to each second subgraph.

[0087] In this embodiment, the multiple second subgraphs formed in step S250 and their respective single-table member information are integrated with the second core table information corresponding to each second subgraph determined in step S260. The above information is organized according to a preset data structure to form complete clustering result data, which includes at least the subgraph identifier, the list of single tables contained in the subgraph, and the second core table identifier of the subgraph. The integrated data is output as the second partitioning result for use in subsequent comparison steps.

[0088] Understandably, this step systematically integrates the outputs of each stage of clustering to form a second partitioning result with a complete structure and comprehensive information, providing standardized data input for subsequent comparison and fusion with the first partitioning result.

[0089] Step S300: Compare the first segmentation result and the second segmentation result using a preset second language model to obtain a comparison result. Based on the comparison result, generate the target segmentation result using the preset second language model.

[0090] It should be noted that the preset second large language model refers to a large language model that has been pre-trained and is used to compare, analyze and make a comprehensive judgment on the two segmentation results. It can be the same model as the preset first large language model, or it can be a different model.

[0091] The comparison refers to the process of comparing and analyzing the differences between the first and second division results at the level of subgraph composition and single table classification.

[0092] The comparison result refers to the difference analysis result output after comparing the first and second division results item by item. It includes at least single tables with consistent classification and boundary node information of conflicting classifications.

[0093] The target partitioning result refers to the final subgraph partitioning and core table identification result determined after comparison, analysis and comprehensive judgment, taking into account both business semantic consistency and topological structure rationality.

[0094] In this embodiment, each first subgraph in the first partitioning result is mapped and compared one-to-one with each second subgraph in the second partitioning result to determine whether each single table is partitioned into the same subgraph in both partitioning results. For single tables with consistent affiliation, they are marked as consistent nodes, and the corresponding subgraphs are marked as consistent subgraphs; for single tables with inconsistent affiliation, they are marked as boundary affiliation conflict nodes.

[0095] Subsequently, the candidate subgraph attribution information and corresponding business scope descriptions of the boundary attribution conflicting nodes in the two partitioning results are obtained. This information is concatenated into a conflict judgment prompt text, which is then input into a preset second language model. This model scores the matching degree between the business semantics of the boundary attribution conflicting nodes and the business scope descriptions of each candidate subgraph. The candidate subgraph attribution with the higher matching degree score is selected as the final subgraph attribution for that node. The boundary attribution conflicting node attribution subgraphs determined by comprehensive judgment are merged with the consistent subgraphs to generate the target partitioning results. The final core table of each target subgraph is determined by combining the core table information from the first and second partitioning results.

[0096] Understandably, this step integrates and corrects the two partitioning results using a large language model, retaining the consistent parts of the two partitioning results, and making accurate decisions on contradictory boundary nodes in combination with business semantics, so that the final partitioning result has both structural rationality and business semantic accuracy, achieving accurate subgraph partitioning and core table identification.

[0097] Furthermore, it should be noted that after the step of generating the target segmentation result using a preset second-largest language model based on the comparison result, the following one is also included: In response to a request to add a single table, the system obtains the table name and field definitions of the new table. By parsing the SQL script in the data warehouse, it determines the association between the new table and the existing tables. If the new table is associated with only one subgraph in the target partitioning result, the new table is added to the associated subgraph. If the new table is associated with multiple subgraphs in the target partitioning result, a target subgraph is selected for the new table based on the target partitioning result, association status, table name information, and field definitions, using a preset third semantic model. It should be noted that a single table add request refers to an operation instruction issued by a user or system to add a new data table in the data warehouse, which includes the table name information, field definitions and initial relationship information between the new single table and the existing single table.

[0098] "Association" refers to a direct association between the newly added table and existing tables within a subgraph of the target partitioning result. In other words, the newly added table and these existing tables have a connection dependency in terms of business logic or data flow.

[0099] Updating a data warehouse refers to writing the table structure information, metadata information, and relationship information of a newly added table into the data warehouse's metadata management system, so that the physical or logical model of the data warehouse includes the newly added table.

[0100] Updating relationships refers to rescanning and extracting the relationships between all tables in the data warehouse after a new table is added, and incorporating the new related records brought about by the new table into the relationship set to form the updated complete relationship.

[0101] In this embodiment, in response to a single table addition request, the table name and field definitions of the new single table are obtained. Subsequently, by parsing the various SQL scripts stored in the data warehouse, the relationships between the new single table and existing single tables are extracted and determined from the JOIN and association conditions in the SQL statements. Based on the parsed relationships, existing single tables that are related to the new single table are searched in the target partitioning results. It is then determined which subgraphs these existing single tables belong to, and a set of associated subgraphs is obtained. The number of subgraphs in the associated subgraph set is counted. If there is only one subgraph, it means that the business association scope of the newly added single table is simple. The newly added single table is directly added to the member list of the subgraph to complete the lightweight addition operation.

[0102] If the set of associated subgraphs contains multiple subgraphs, it indicates that the newly added table has relationships across multiple business domains. In this case, a preset third-largest language model is invoked for subgraph selection. The table name information, field definitions, parsed relationships, and existing complete target partitioning results of the newly added table are concatenated into a subgraph selection prompt text according to a preset format and input into the preset third-largest language model. This model comprehensively judges the matching degree between the newly added table and each candidate subgraph based on the business semantics of the newly added table, the data content reflected in the field definitions, and the degree of its association with existing tables in each subgraph, and finally outputs the most suitable subgraph assignment for the newly added table. The newly added table is then added to the target subgraph, completing the addition operation without triggering a global re-partition.

[0103] In response to a single table deletion request, determine whether the single table to be deleted involves core business data. If not, delete the single table directly from the subgraph corresponding to the single table to be deleted. If it does, update the data warehouse based on the single table to be deleted and update the association relationships between the single tables in the data warehouse. Based on the updated data warehouse and the updated association relationships, return to the step of obtaining the association relationships between the single tables in the data warehouse and regenerate a new target partitioning result. It should be noted that a single table deletion request refers to an operation command issued by a user or system to remove an existing data table from the data warehouse, which includes the table name information of the single table to be deleted.

[0104] Core business data refers to the business data carried by the core tables identified in the target partitioning results in this solution. That is, if the single table to be deleted is a target core table, it is considered to involve core business data; if it is a non-core table, it is considered not to involve core business data.

[0105] In this embodiment, when a single table deletion request is received, the system first searches for the single table to be deleted in the target partitioning results and determines whether the single table is the target core table of its subgraph. If the single table to be deleted is a non-core table, it means that deleting the table has little impact on the overall subgraph structure, and the single table is directly removed from the subgraph member list to which the table belongs, completing the lightweight deletion operation. If the single table to be deleted is a target core table, it means that deleting the table will destroy the original central structure of the subgraph, and at this time, a repartitioning process is executed: the single table and its associated records are removed from the data warehouse, the metadata information of the data warehouse is updated, and the association relationships between all remaining single tables in the data warehouse are re-extracted. The updated association relationships are used as input, and the system returns to step S100 to re-execute the complete subgraph partitioning process and generate a new target partitioning result.

[0106] Understandably, for single-table deletion scenarios, core tables and non-core tables are treated differently. Non-core tables are removed directly using a low-cost method, while core tables are deleted by triggering a global repartition to prevent subgraph structure disorder, thus achieving a balance between operational efficiency and structural stability.

[0107] In response to the repartitioning instruction, the relationships between individual tables in the data warehouse are updated. Based on the updated relationships, the step of obtaining the relationships between individual tables in the data warehouse is returned, and a new target partitioning result is regenerated. The repartitioning instruction is triggered periodically based on a preset time threshold.

[0108] It should be noted that the re-partitioning instruction is a control signal that triggers the re-execution of the sub-graph partitioning process for the entire data warehouse ER diagram. It can be automatically generated by the system based on a preset time threshold, or it can be manually initiated by the administrator.

[0109] The preset time threshold refers to a pre-set time interval value N, which is used to monitor the interval from the last time the global repartition was completed to the current time. When the interval exceeds the threshold, the system automatically generates a repartition command.

[0110] In this embodiment, the system continuously monitors the interval duration calculated from the last global repartition completion time and compares this interval duration with a preset time threshold N. If the interval duration exceeds the preset time threshold N, the system automatically generates a repartition instruction. In response to this repartition instruction, the system first re-extracts the latest relationships between all current single tables in the data warehouse to ensure that the relationships reflect the latest state of the data warehouse. Subsequently, using the updated relationships as input, the system returns to step S100 to re-execute the complete subgraph partitioning process and generate a new target partitioning result. After the new partitioning result is generated, the global repartitioning time record is reset, and the current time is used as the new last global repartition completion time for monitoring the interval duration in the next round.

[0111] Understandably, by building a timed repartitioning fallback mechanism through preset time thresholds, periodic global optimization is automatically triggered when no single table addition or deletion operations occur for a long period of time, resulting in the failure of the subgraph structure to be updated in a timely manner, so as to continuously ensure the timeliness of ER subgraph partitioning and its adaptability to business development.

[0112] This application discloses a method and related equipment for subgraph partitioning of an ER diagram in a data warehouse, relating to the field of data processing technology. In contrast to related technologies, where clustering partitioning is based solely on statistical information without considering the actual business content of the data tables, resulting in subgraph partitioning results deviating from actual business operations, inaccurate partitioning of edge nodes between subgraphs due to their proximity to adjacent subgraphs, and blurred subgraph boundaries, the entire ER diagram must be repartitioned when a single table is added or deleted, making updates and maintenance difficult and compromising the consistency of partitioning results. In this application, firstly, the relationships between individual tables in the data warehouse are obtained. Then, based on these relationships, a preset first language model is used to prepartition the ER diagram, obtaining a first partitioning result. Next, the Louvain clustering algorithm is used to cluster the ER diagram, obtaining a second partitioning result. Finally, a preset second language model is used to compare the first and second partitioning results, obtaining a comparison result. Based on the comparison result, a target partitioning result is generated using the preset second language model.

[0113] Understandably, this application integrates the semantic understanding capabilities of a large language model with the topological analysis capabilities of the Louvain clustering algorithm, and uses the large model to compare and fuse the two partitioning results. This approach takes into account both business semantics and structural association features, effectively solving the problems of partitioning deviating from business and ambiguous edge affiliation, and achieving more accurate subgraph partitioning.

[0114] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The step of comparing the first segmentation result and the second segmentation result using a preset second language model to obtain the comparison result further includes steps A10 to A40: Step A10: Based on the preset second large language model, each first subgraph in the first partitioning result is mapped and compared with each second subgraph in the second partitioning result to determine whether each single table is partitioned into a subgraph with the same subgraph affiliation in the first partitioning result and the second partitioning result. It should be noted that one-to-one mapping comparison refers to the process of matching and comparing each first subgraph in the first partitioning result with each second subgraph in the second partitioning result. Specifically, it involves checking which subgraph the same single table is assigned to in the two partitioning results, and then determining whether the two subgraph assignments of the single table are consistent.

[0115] Subgraph affiliation refers to the affiliation of a single table to a specific subgraph in a particular partitioning result, including subgraph identification information and the business or structural scope covered by the subgraph.

[0116] In this embodiment, the first partitioning result output in step S225 and the second partitioning result output in step S270 are obtained. Using a single table as the basic comparison unit, each single table in the data warehouse is traversed, and the first subgraph identifier to which the single table belongs in the first partitioning result and the second subgraph identifier to which it belongs in the second partitioning result are found. The subgraph identifiers in the two partitioning results are compared to determine whether they point to the same subgraph. For semantic equivalence judgments required during the judgment process—that is, cases where the first and second subgraphs have different names but actually contain the same set of single tables—a preset second language model is called to perform subgraph equivalence semantic judgment. After completing the attribution comparison of all single tables, it is recorded whether the subgraph attribution of each single table is consistent in the two partitioning results.

[0117] Understandably, this step establishes the correspondence between the first and second partitioning results, accurately locating the differences in subgraph attribution between the two partitioning schemes on a single table basis, thus providing a data foundation for subsequent classification processing.

[0118] Step A20: For a single table whose subgraphs belong to the same subgraph in the first partitioning result and the second partitioning result, mark the single table as a node with consistent subgraph ownership, and mark the corresponding subgraph as a consistent subgraph. It should be noted that a node with consistent attribution refers to a single table that is assigned to the same subgraph in both partitioning results. That is, the single table belongs to the same subgraph in both the first and second partitioning results, indicating that the two partitioning schemes have reached a consensus on the attribution of the single table.

[0119] A consistent subgraph is one in which all single tables within a subgraph are assigned to the same subgraph in the same way by the other partitioning result in two different partitioning results. In other words, the composition of the members of the subgraph is exactly the same in both partitioning results, and there are no nodes with conflicting affiliations.

[0120] In this embodiment, the attribution comparison results of each single table recorded in step A10 are read, and single tables whose subgraphs have the same attribution in both partitioning results are selected. "Attribution Consistent Node" markers are added to these single tables, and the subgraph identifiers to which they belong are recorded. Subsequently, the results are summarized on a subgraph basis, identifying those subgraphs where all member single tables are attribution consistent nodes, and these subgraphs are marked as consistent subgraphs. A consistent subgraph indicates that the overall partitioning of the subgraph is completely consistent under both partitioning schemes, requiring no further processing.

[0121] Understandably, this step extracts the stable parts that do not have disagreements from the two partitioning results, and uses these consistent nodes and consistent subgraphs as the basic retention modules of the final target partitioning result, reducing the scope of subsequent processing that requires the intervention of a large language model to resolve conflicts.

[0122] Step A30: For a single table whose subgraph affiliation is inconsistent in the first partitioning result and the second partitioning result, mark the single table as a boundary affiliation conflict node; It should be noted that a node with conflicting boundary attribution refers to a single table that is assigned to different subgraphs in the two partitioning results. That is, the single table belongs to a certain first subgraph in the first partitioning result, but belongs to a different second subgraph in the second partitioning result. The two partitioning schemes have a disagreement in determining the attribution of the single table.

[0123] In this embodiment, the attribution comparison results of each single table recorded in step A10 are read, and single tables whose subgraph attributions are inconsistent in the two partitioning results are filtered out. "Boundary attribution conflict node" marker information is added to these single tables, and the attribution information of the first candidate subgraph in the first partitioning result and the second candidate subgraph in the second partitioning result are recorded respectively, including the candidate subgraph identifier and its corresponding business scope description or structural feature information. Boundary attribution conflict nodes are usually located in the boundary area of ​​two or more subgraphs, and their topological connection characteristics show that they have a relatively close association strength with adjacent subgraphs.

[0124] Understandably, this step precisely locates the single tables where the two partitioning schemes differ, identifying them as edge nodes that require further comprehensive judgment, thus providing a clear target set for subsequent attribution determination using a large language model.

[0125] Step A40: Based on the consistent nodes and the conflicting boundary nodes, obtain the comparison results.

[0126] It should be noted that the comparison result refers to the comprehensive difference analysis results output after comparing the first and second partitioning results item by item. It includes the information of nodes with consistent ownership and their corresponding consistent subgraphs, as well as the information of nodes with conflicting boundary ownership and their candidate subgraph ownership in the two partitioning results.

[0127] In this embodiment, the consistent node and consistent subgraph information marked in step A20 are summarized and integrated with the boundary conflict nodes and their candidate subgraph attribution information marked in step A30. The above information is organized according to a preset data structure to form a structured comparison result, which includes at least three types of data fields: first, a list of consistent nodes and their common subgraph identifiers; second, a list of consistent subgraphs and their member tables; and third, a list of boundary conflict nodes and their corresponding candidate subgraph attribution information in the first and second partitioning results, respectively. The integrated data is output as the comparison result for use in subsequent steps to generate the target partitioning result.

[0128] Understandably, this step categorizes, collects, and structures all the outputs from the comparison analysis phase to form complete comparison result data, providing clear and standardized input information for the next step of generating the final target segmentation result based on the comparison result using a pre-set second language model.

[0129] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The step of generating the target segmentation result using a preset second language model based on the comparison result further includes steps S310 to S350: Step S310: Obtain the first candidate subgraph attribution information of the boundary attribution conflict node in the first partitioning result and its corresponding first business scope description, obtain the second candidate subgraph attribution information of the boundary attribution conflict node in the second partitioning result and its corresponding second business scope description, and obtain the table name information, Chinese table name and English table name of the boundary attribution conflict node. It should be noted that the first candidate subgraph attribution information refers to the identification information of the specific first subgraph to which the boundary attribution conflict node is assigned in the first partitioning result, as well as the relevant description of the subgraph, including the subgraph name, subgraph number, and the list of single-table members covered by the subgraph.

[0130] The second candidate subgraph attribution information refers to the identification information of the specific second subgraph to which the boundary attribution conflict node is assigned in the second partitioning result, as well as the relevant description of the subgraph, including the subgraph name, subgraph number, and the list of single-table members covered by the subgraph.

[0131] The first business scope description refers to the business domain description information corresponding to each first subgraph in the first division result. It is generated by the preset first major language model in the pre-division stage and is used to summarize the business function scope that all single tables in the subgraph jointly serve. For example, "This subgraph contains customer basic information, customer contact information and customer account related tables, serving the customer management business domain."

[0132] The second business scope description refers to the business domain description information corresponding to each second subgraph in the second partitioning result. Since the second partitioning result comes from the Louvain clustering algorithm, which itself does not output business semantic descriptions, the second business scope description is generated by inputting the single-table member list of the second subgraph into a preset first or second major language model. The model summarizes the business function scope corresponding to the second subgraph based on the Chinese and English names of these single tables.

[0133] Table name information refers to the unique identifier of a data table in a data warehouse. It is usually the English name of the table or the table code, and is used to uniquely identify a data table in the system.

[0134] In this embodiment, a list of all boundary attribution conflicting nodes is extracted from the comparison results. For each boundary attribution conflicting node in the list, the following information acquisition operations are performed: The first subgraph to which the node belongs is located in the first partitioning result; the subgraph identifier and corresponding first business scope description of the first subgraph are read as the first candidate subgraph attribution information and the first business scope description; the second subgraph to which the node belongs is located in the second partitioning result; the subgraph identifier of the second subgraph is read; the single-table member list of the second subgraph is input into the large language model to generate the corresponding second business scope description; the subgraph identifier and the generated business scope description are used as the second candidate subgraph attribution information and the second business scope description; the table name information, Chinese table name, and English table name of the single table are read from the metadata of the data warehouse. After extracting information from all boundary attribution conflicting nodes, all acquired information is organized and summarized.

[0135] Understandably, this is to prepare complete data materials for the construction of subsequent contradiction judgment prompt text, ensuring that the candidate attribution information of each boundary attribution contradiction node in the two division results and its own business semantic information are fully collected, so as to provide sufficient basis for the model to make accurate decisions.

[0136] Step S320: Combine the Chinese name of the table, the English name of the table, the first candidate subgraph attribution information, the first business scope description, the second candidate subgraph attribution information, and the second business scope description of each boundary attribution conflict node into a conflict judgment prompt text; It should be noted that the contradiction judgment prompt text refers to the natural language text generated by splicing together the candidate attribution information of the boundary attribution contradiction node in the two partitioning results and the business semantic information of the node itself according to a preset format. This text is used as the input of the preset second language model to guide the model to judge the final attribution of the boundary node.

[0137] In this embodiment, for each boundary attribution conflict node, the Chinese and English names of the table, the attribution information of the first candidate subgraph and its corresponding first business scope description, and the attribution information of the second candidate subgraph and its corresponding second business scope description, extracted in step S310, are obtained. This information is then concatenated according to a preset prompt text format. Specifically, the format is as follows: first, the Chinese and English names of the table for the boundary attribution conflict node are described to clarify its business identity; then, the candidate subgraph attribution and its corresponding business scope description in the first partitioning result and the candidate subgraph attribution and its corresponding business scope description in the second partitioning result are listed in the form of "Candidate Scheme 1" and "Candidate Scheme 2," respectively; finally, a guiding statement is added at the end of the text, requiring the model to make an attribution selection based on the matching degree between the node's business semantics and the business scope of each candidate subgraph. A corresponding conflict judgment prompt text is generated for each boundary attribution conflict node.

[0138] Understandably, transforming multi-source information about boundary attribution conflict nodes into structured natural language prompts enables the pre-defined second language model to clearly understand the attribution conflict faced by the node and the possible attribution schemes, providing a standardized input form for the model's accurate reasoning and judgment.

[0139] Step S330: The contradiction judgment prompt text is processed using a preset second language model to determine the first matching degree between the business semantics of each boundary-attributed contradiction node and the first business scope description, and the second matching degree between the business semantics of the node and the second business scope description. It should be noted that the first matching degree refers to the degree of fit between the business semantics of the boundary-attribution conflict node and the business scope description of the first candidate subgraph. It is expressed in numerical or grade form. The higher the value, the more the business function of the node matches the business domain served by the first candidate subgraph.

[0140] The second matching degree refers to the degree of fit between the business semantics of the boundary-attribution conflict node and the business scope description of the second candidate subgraph. It is expressed in numerical or grade form. The higher the value, the more the business function of the node matches the business domain served by the second candidate subgraph.

[0141] In this embodiment, the contradiction judgment prompt text generated in step S320 is input into a preset second language model. This model performs reading comprehension on the prompt text. First, it parses the business semantics contained in the Chinese and English names of the boundary-attribution contradiction nodes to clarify the core business function served by the node. Then, it parses the business scope descriptions of the first and second candidate subgraphs respectively to understand the business domain categories corresponding to each candidate subgraph. Finally, it calculates the semantic similarity and evaluates the matching degree between the node's business semantics and the business scope descriptions of the two candidate subgraphs. After internal reasoning, the model outputs two matching degree scores: the first matching degree between the node's business semantics and the first business scope description, and the second matching degree between the node's business semantics and the second business scope description.

[0142] Understandably, this step utilizes the semantic understanding capabilities of large language models to quantitatively evaluate the business fit between conflicting boundary attribution nodes and the two candidate subgraphs, transforming fuzzy semantic matching relationships into comparable matching values, thus providing an objective quantitative basis for subsequent attribution decisions.

[0143] Step S340: Based on the first matching degree and the second matching degree, the candidate subgraph with the highest matching degree is set as the target subgraph; It should be noted that the target subgraph refers to the final subgraph selected after comparing the matching degree for each boundary ownership conflict node. This subgraph can be the first candidate subgraph or the second candidate subgraph, and the selection principle is the candidate scheme with the higher matching degree.

[0144] In this embodiment, the first and second matching degree values ​​output by the preset second language model in step S330 are obtained. The two matching degree values ​​are compared to determine which value is higher. If the first matching degree is higher than the second matching degree, the first candidate subgraph is selected as the target subgraph for the boundary attribution conflict node; if the second matching degree is higher than the first matching degree, the second candidate subgraph is selected as the target subgraph for the boundary attribution conflict node; if the two matching degree values ​​are equal, the association weight or connection strength of the node in the two candidate subgraphs is compared as an auxiliary judgment criterion, and the candidate subgraph with the higher association weight or connection strength is selected as the target subgraph. The above comparison and selection operation is performed for each boundary attribution conflict node to ensure that each node obtains a unique target subgraph attribution.

[0145] Understandably, this step makes a final decision on the attribution of each boundary node with conflicting attribution based on the matching degree comparison results. It uses the business semantic matching degree as the core basis for the decision, ensuring that the boundary node is assigned to the subgraph that best matches its business function, thereby achieving accurate determination of edge attribution.

[0146] Step S350: Merge the target subgraph and the consistent subgraph to obtain the target partitioning result.

[0147] It should be noted that the target partitioning result refers to the final subgraph partitioning result generated by merging the target subgraph and the consistent subgraph after the second pre-set language model has made a comprehensive decision on the boundary ownership conflict nodes. It includes the final subgraph ownership information of all single tables and the final core table information of each subgraph.

[0148] In this embodiment, all consistent subgraphs marked in step A20 and their contained single-table member information are obtained, and the target subgraph attribution information determined for each boundary attribution conflict node in step S340 is also obtained. A merging operation is performed on a subgraph-by-subgraph basis: for a target subgraph selected by a boundary attribution conflict node, if the subgraph was originally a consistent subgraph, the boundary attribution conflict node is directly added to the member list of the consistent subgraph; if the subgraph is a non-consistent subgraph in the first or second candidate subgraph, a new subgraph is constructed using it as a framework, and all boundary attribution conflict nodes that selected this subgraph as the target attribution are included in it. After merging, a target partitioning result containing all single tables is obtained. Subsequently, for each target subgraph, the first core table information in the first partitioning result and the second core table information in the second partitioning result are comprehensively determined to form the final core table. If both point to the same single table, it is directly determined as the core table; if they are inconsistent, the single table with higher association weight in the target subgraph is selected as the final core table. After integrating the subgraph composition information and the core table information, a complete target partitioning result is formed and output.

[0149] Understandably, this step merges and combines the consistent parts of the two partitioning results with the contradictory parts after model adjudication, forming a final partitioning result that takes into account both business semantic consistency and topological rationality, thus achieving accurate subgraph partitioning and core table identification.

[0150] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data warehouse ER diagram subgraph partitioning method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0151] This application also provides a data warehouse ER diagram subgraph partitioning device, please refer to... Figure 4 The data warehouse ER diagram subgraph partitioning device includes: Acquisition module 10, the acquisition module is used to acquire the relationship between various single tables in the data warehouse; The partitioning module 20 is used to pre-partition the ER graph based on the association relationship using a preset first large language model to obtain a first partitioning result, and then use the Louvain clustering algorithm to cluster the ER graph to obtain a second partitioning result; The generation module 30 is used to compare the first segmentation result and the second segmentation result using a preset second language model to obtain a comparison result, and based on the comparison result, to generate a target segmentation result using the preset second language model.

[0152] The data warehouse ER diagram subgraph partitioning apparatus provided in this application employs the data warehouse ER diagram subgraph partitioning method described in the above embodiments, and can solve the technical problem of data warehouse ER diagram subgraph partitioning. Compared with related technologies, the beneficial effects of the data warehouse ER diagram subgraph partitioning apparatus provided in this application are the same as those of the data warehouse ER diagram subgraph partitioning method described in the above embodiments, and other technical features in the data warehouse ER diagram subgraph partitioning apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0153] This application provides a data warehouse ER diagram subgraph partitioning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data warehouse ER diagram subgraph partitioning method in the first embodiment described above.

[0154] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of a data warehouse ER diagram subgraph partitioning device suitable for implementing embodiments of this application. The data warehouse ER diagram subgraph partitioning device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The data warehouse ER diagram subgraph partitioning device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0155] like Figure 5As shown, the data warehouse ER diagram subplot partitioning device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the data warehouse ER diagram subplot partitioning device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the data warehouse ER diagram subplot partitioning device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a data warehouse ER diagram subplot partitioning device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0156] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0157] The data warehouse ER diagram subgraph partitioning device provided in this application, employing the data warehouse ER diagram subgraph partitioning method described in the above embodiments, can solve the technical problem of data warehouse ER diagram subgraph partitioning. Compared with related technologies, the beneficial effects of the data warehouse ER diagram subgraph partitioning device provided in this application are the same as those of the data warehouse ER diagram subgraph partitioning method described in the above embodiments, and other technical features in this data warehouse ER diagram subgraph partitioning device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0158] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0159] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0160] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the data warehouse ER diagram subgraph partitioning method in the above embodiments.

[0161] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0162] The aforementioned computer-readable storage medium may be included in the data warehouse ER diagram subplot partitioning device; or it may exist independently and not be assembled into the data warehouse ER diagram subplot partitioning device.

[0163] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the data warehouse ER diagram subgraph partitioning device, cause the data warehouse ER diagram subgraph partitioning device to: Obtain the relationships between individual tables in the data warehouse; Based on the aforementioned relationship, the ER graph is pre-divided using a preset first large language model to obtain a first partitioning result, and the ER graph is clustered using the Louvain clustering algorithm to obtain a second partitioning result; The first segmentation result and the second segmentation result are compared using a preset second largest language model to obtain a comparison result. Based on the comparison result, the target segmentation result is generated using the preset second largest language model.

[0164] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0166] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0167] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data warehouse ER diagram subgraph partitioning method, thereby solving the technical problem of data warehouse ER diagram subgraph partitioning. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data warehouse ER diagram subgraph partitioning method provided in the above embodiments, and will not be repeated here.

[0168] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data warehouse ER diagram subgraph partitioning method described above.

[0169] The computer program product provided in this application can solve the technical problem of subgraph partitioning of ER diagrams in data warehouses. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the data warehouse ER diagram subgraph partitioning method provided in the above embodiments, and will not be repeated here.

[0170] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for partitioning ER diagram subgraphs in a data warehouse, characterized in that, The data warehouse ER diagram sub-diagram partitioning method includes: Obtain the relationships between individual tables in the data warehouse; Based on the aforementioned relationship, the ER graph is pre-divided using a preset first large language model to obtain a first partitioning result, and the ER graph is clustered using the Louvain clustering algorithm to obtain a second partitioning result; The first segmentation result and the second segmentation result are compared using a preset second largest language model to obtain a comparison result. Based on the comparison result, the target segmentation result is generated using the preset second largest language model.

2. The data warehouse ER diagram subgraph partitioning method as described in claim 1, characterized in that, The step of pre-segmenting the ER graph based on the association relationship using a preset first large language model to obtain a first segmentation result further includes: Perform a deduplication operation on the aforementioned relationships to remove redundant records that are mutually inverse; Based on the deduplicated relationships, the ER graph is pre-partitioned using the first pre-defined language model to obtain the first partitioning result.

3. The data warehouse ER diagram subgraph partitioning method as described in claim 2, characterized in that, The step of pre-segmenting the ER graph using a preset first large language model based on the deduplicated association relationships to obtain the first segmentation result further includes: Obtain the Chinese and English names of each individual table, and concatenate the Chinese and English names of the tables and the deduplicated relationships into semantic prompt text according to a preset format; The semantic prompt text is semantically parsed using a preset first major language model to obtain the business semantic category corresponding to each single table; Single tables belonging to the same business semantic category are divided into the same subgraph to form multiple first subgraphs, and the association weight of each single table in each first subgraph is determined. Set the single table with the highest association weight in each first subgraph as the first core table of that first subgraph; Based on multiple first subgraphs and the first core table corresponding to each first subgraph, the first partitioning result is obtained.

4. The data warehouse ER diagram subgraph partitioning method as described in claim 1, characterized in that, The step of using the Louvain clustering algorithm to cluster the ER graph and obtain the second partitioning result further includes: Based on the deduplicated relationships, each individual table is used as a node to construct an undirected weighted network; The Louvain clustering algorithm is used to cluster the undirected weighted network, and each single table is aggregated into different communities based on the density of node connections in the undirected weighted network. Each community is treated as a second subgraph, resulting in multiple second subgraphs. The weighting degree of each single table in each second subgraph is then determined. The single table with the highest weight in each second subgraph is designated as the second core table of that second subgraph. The second partitioning result is obtained based on multiple second subgraphs and the second core table corresponding to each map.

5. The data warehouse ER diagram subgraph partitioning method as described in claim 1, characterized in that, The step of comparing the first segmentation result and the second segmentation result using a preset second language model to obtain the comparison result further includes: Based on the preset second language model, each first subgraph in the first partitioning result is mapped and compared with each second subgraph in the second partitioning result to determine whether each single table is partitioned into a subgraph with the same subgraph affiliation in the first partitioning result and the second partitioning result. For a single table whose subgraphs belong to the same subgraph in the first partitioning result and the second partitioning result, mark the single table as a node with consistent subgraph affiliation and mark the corresponding subgraph as a consistent subgraph; For a single table whose subgraph affiliation is inconsistent in the first partitioning result and the second partitioning result, the single table is marked as a boundary affiliation conflict node. The comparison results are obtained based on the nodes with consistent affiliation and the nodes with conflicting affiliation at the boundary.

6. The data warehouse ER diagram subgraph partitioning method as described in claim 5, characterized in that, The step of generating target segmentation results using a preset second-largest language model based on the comparison results further includes: Obtain the first candidate subgraph attribution information of the boundary attribution conflict node in the first partitioning result and its corresponding first business scope description, obtain the second candidate subgraph attribution information of the boundary attribution conflict node in the second partitioning result and its corresponding second business scope description, and obtain the table name information, Chinese table name and English table name of the boundary attribution conflict node; The Chinese and English names of the tables for each boundary-attributed conflicting node, the attribution information of the first candidate subgraph, the description of the first business scope, the attribution information of the second candidate subgraph, and the description of the second business scope are concatenated into a conflict judgment prompt text. The contradiction judgment prompt text is processed using a preset second major language model to determine the first degree of matching between the business semantics of each boundary-attributed contradiction node and the first business scope description, and the second degree of matching between the business semantics of the node and the second business scope description. Based on the first matching degree and the second matching degree, the candidate subgraph with the highest matching degree is set as the target subgraph; The target subgraph and the consistent subgraph are merged to obtain the target partitioning result.

7. The data warehouse ER diagram subgraph partitioning method as described in claim 1, characterized in that, After the step of generating the target segmentation result using a preset second-largest language model based on the comparison result, it further includes any one of the following: In response to a single table addition request, the system obtains the table name and field definitions of the new table. It then parses the SQL script in the data warehouse to determine the relationships between the new table and existing tables. Based on these relationships, it assesses the association between the new table and each subgraph in the target partitioning result. If the new table is associated with only one subgraph in the target partitioning result, it is added to that subgraph. If the new table is associated with multiple subgraphs in the target partitioning result, a preset third semantic model is used to select a target subgraph for the new table based on the target partitioning result, the association relationships, the table name, and the field definitions. In response to a single table deletion request, determine whether the single table to be deleted involves core business data. If not, delete the single table directly from the subgraph corresponding to the single table to be deleted. If applicable, update the data warehouse based on the single table to be deleted, and update the relationships between the single tables in the data warehouse. Based on the updated data warehouse and the updated relationships, return to the step of obtaining the relationships between the single tables in the data warehouse, and regenerate the new target partitioning result. In response to the repartitioning instruction, the relationships between individual tables in the data warehouse are updated. Based on the updated relationships, the step of obtaining the relationships between individual tables in the data warehouse is returned, and a new target partitioning result is regenerated. The repartitioning instruction is triggered periodically based on a preset time threshold.

8. A data warehouse ER diagram subgraph partitioning device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data warehouse ER graph subgraph partitioning method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data warehouse ER diagram subgraph partitioning method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data warehouse ER graph subgraph partitioning method as described in any one of claims 1 to 7.

Images