Integrated metadata management method and system based on data source extensibility

Abstract

The present invention provides an integrated metadata management method and system based on data source expandability, which relates to the technical field of data source, including performing real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collecting connection information and establishing a dynamic connection pool; obtaining table structure, field attributes and association relationship list, and assembling them into JSON format data; parsing and standardizing the data, building a correlation calculation matrix; generating a standardized data structure description file; establishing a three-layer index directory, detecting structural changes and updating; calculating data table reference dependencies, and generating a DAG data flow path. The present invention realizes efficient integration and management of heterogeneous data sources and improves metadata processing efficiency.

Description

Technical Field

[0001] The present invention relates to data source technology, and in particular to an integrated metadata management method and system based on an expandable data source. Background Art

[0002] With the advent of the big data era, enterprises and organizations are faced with the challenge of managing and integrating metadata from multiple heterogeneous data sources. Metadata management is essential for data governance, data quality control, and data analysis. Traditional metadata management methods are usually designed for specific types of data sources and are difficult to adapt to the increasingly complex and diverse data environment.

[0003] In the current technical environment, metadata management faces several major problems. First, the metadata formats and structures of different data sources vary greatly, making integration and unified management difficult. Second, with the increase in the number of data sources and the frequent changes in data structures, manual updating and maintenance of metadata becomes extremely time-consuming and error-prone. Finally, existing metadata management systems often lack flexibility and are difficult to quickly adapt to new data source types or changed data structures.

[0004] These challenges highlight the need to develop a more flexible, automated, and scalable metadata management approach. An ideal solution should be able to automatically detect and collect metadata from multiple heterogeneous data sources, monitor changes in data structure in real time, and provide a unified metadata access and management interface. At the same time, the approach should also have efficient data association analysis capabilities to support complex data integration and analysis requirements. Summary of the invention

[0005] The embodiments of the present invention provide an integrated metadata management method and system that are scalable based on data sources, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provides an integrated metadata management method that is scalable based on data sources, including:

[0008] Perform real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collect the connection addresses, access credentials and protocol types of the multiple heterogeneous data sources, establish a dynamic connection pool according to the connection addresses, and set the maximum number of connections to N, the timeout period to T seconds, and the number of retries to M times in the dynamic connection pool; use the connection instances in the dynamic connection pool to respectively execute the SELECT statement to obtain the table structure list, execute the SHOW COLUMNS statement to obtain the field attribute list, and execute the SHOW KEYS statement to obtain the association relationship list, and assemble the obtained table structure list, field attribute list and association relationship list into unified JSON format data;

[0009] Parse and standardize the JSON format data, extract field names, field types, primary key information and foreign key information, and construct an N×N dimensional association calculation matrix; in the association calculation matrix, use a field naming similarity algorithm to calculate the similarity score of the field name, use a data type compatibility rule to calculate the matching score of the field type, calculate the inter-table association score based on the foreign key constraint relationship, and use the weighted average of the similarity score, the matching score and the inter-table association score as the final field mapping association; generate a standardized data structure description file according to the field mapping association;

[0010] Based on the data structure description file, a three-layer index directory is established using a B+ tree index structure, wherein the first layer index records the data source type and its physical storage location, the second layer index records the business attributes and its access path, and the third layer index records the usage scenario and its version information; the three-layer index directory is used to poll the structure changes of the multiple heterogeneous data sources at a frequency of F seconds / times, and when a change in the table structure, field attribute or association relationship is detected, the data structure description file is updated again; based on the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationship between data tables, and a data flow path in the form of a DAG directed acyclic graph is generated.

[0011] In the association calculation matrix, the field name similarity score is calculated using the field naming similarity algorithm, the field type matching score is calculated using the data type compatibility rule, and the inter-table association score is calculated based on the foreign key constraint relationship. The weighted average of the similarity score, the matching score and the inter-table association score is used as the final field mapping association degree, including:

[0012] The association degree calculation matrix includes a to-be-mapped field identifier and a score storage unit, and the score storage unit is used to store the calculated scores generated during the field mapping process;

[0013] Obtaining a field name corresponding to the to-be-mapped field identifier, performing word segmentation preprocessing on the field name to obtain a root list, calculating edit distance values ​​between roots in the root list using a field naming similarity algorithm, normalizing the edit distance values ​​to obtain a similarity score, and writing the similarity score into the score storage unit based on the to-be-mapped field identifier;

[0014] Acquire field data type information according to the to-be-mapped field identifier, extract compatibility rules and conversion rules matching the field data type information from a preset type mapping rule library, calculate a direct compatibility score based on the compatibility rule, calculate a conversion cost score based on the conversion rule, use a weighted combination value of the direct compatibility score and the conversion cost score as a matching score, and write the matching score into the score storage unit based on the to-be-mapped field identifier;

[0015] Using the to-be-mapped field identifier to query the foreign key constraint information of the data table to which the field belongs, parsing the primary key field information and the reference constraint information from the foreign key constraint information, constructing an associated field index based on the primary key field information, calculating an associated weight using the reference constraint information, taking the product of the associated field index and the associated weight as an inter-table association score, and writing the inter-table association score into the score storage unit based on the to-be-mapped field identifier;

[0016] The similarity score, matching score and inter-table association score of the current field to be mapped are read from the score storage unit using the field identifier to be mapped, and the read similarity score, matching score and inter-table association score are weighted averaged to obtain the field mapping association degree.

[0017] Detecting structural changes of the plurality of heterogeneous data sources by polling the three-layer index directory at a frequency of F seconds / times, and when detecting changes in table structure, field attributes or association relationships, re-updating the data structure description file includes:

[0018] Detecting structural changes of the plurality of heterogeneous data sources by polling the three-layer index directory at a frequency of F seconds / times, the three-layer index directory comprising a data source-level directory, a table-level directory and a field-level directory, establishing a first association relationship between the table-level directory and the data source-level directory, and establishing a second association relationship between the field-level directory and the table-level directory;

[0019] Reading a data source version mark stored in the data source level directory, and comparing the data source version mark with the real-time version marks of the multiple heterogeneous data sources, wherein the data source version mark is used to identify the structural status of the multiple heterogeneous data sources;

[0020] When the data source version mark is inconsistent with the real-time version mark, obtaining a changed heterogeneous data source identifier, and obtaining a table structure identifier and a table structure version mark from the table-level directory according to the first association relationship, wherein the table structure version mark is used to identify the state of the table structure;

[0021] The table structure version mark is compared with the real-time table structure version mark in the changed heterogeneous data source, and when a table structure change is detected, a field attribute identifier and a field attribute version mark are obtained from the field-level directory according to the second association relationship, wherein the field attribute version mark is used to identify the state of the field attribute;

[0022] The field attribute version tag is compared with the real-time field attribute version tag in the changed table structure. When a change in field attribute or association relationship is detected, the changed field attribute information and association relationship information are obtained; and the data structure description file is re-updated based on the changed field attribute information and the association relationship information.

[0023] According to the updated data structure description file, the reference dependency relationship between data tables is calculated using a depth-first search algorithm to generate a data flow path in the form of a DAG directed acyclic graph, including:

[0024] Parsing the updated data structure description file, extracting inter-table association information in the data structure description file, wherein the inter-table association information includes primary and foreign key relationship information, field reference relationship information, and business association relationship information;

[0025] Constructing an adjacency list structure based on the association information between tables, wherein each node in the adjacency list structure includes a table identifier, table attribute information, reference field information and association type information, and the association type information is used to identify the dependency strength between nodes;

[0026] Initialize a depth-first search environment, create an access stack structure for recording access paths, create an access mark bitmap for avoiding repeated accesses, and set a maximum recursive depth threshold for controlling the search depth. The access stack structure, the access mark bitmap, and the maximum recursive depth threshold are used to assist in the execution of the depth-first search;

[0027] Using a depth-first search algorithm to traverse the adjacency list structure, pushing the current access node into the access stack structure and marking the accessed state in the access mark bitmap, obtaining a set of adjacent nodes of the current access node;

[0028] For each to-be-visited node in the adjacent node set, detecting whether the to-be-visited node already exists in the access stack structure, and when the detection result is yes, extracting the node information in the access stack structure to generate a ring dependency path;

[0029] Analyze the association type information between the nodes in the ring dependency path, and when there is a strong dependency relationship between the nodes, add a strong dependency mark to the corresponding edge in the adjacency list structure; when there is a non-strong dependency relationship between the nodes, delete the edge forming the loop from the adjacency list structure to obtain an optimized adjacency list structure;

[0030] Converting the optimized adjacency list structure into a directed acyclic graph, layering the directed acyclic graph based on the dependency depth of the nodes, assigning nodes with the same dependency depth to the same level, and obtaining a hierarchical directed acyclic graph;

[0031] A transitive closure analysis is performed on the hierarchical directed acyclic graph to identify and remove redundant dependency edges to obtain a simplified directed acyclic graph; based on the simplified directed acyclic graph, a topological sorting algorithm is used to generate a data flow path.

[0032] Creating an access mark bitmap to avoid repeated access, setting a maximum recursive depth threshold to control the search depth, the access stack structure, the access mark bitmap and the maximum recursive depth threshold to assist the execution of depth-first search include:

[0033] Based on the number of nodes to be recorded in the access stack structure, an access mark bitmap is created to avoid repeated access, wherein the access mark bitmap uses two binary bits to represent the access status of the node, wherein the first bit represents whether the node is accessed, and the second bit represents whether the node is on the current access path;

[0034] According to the node depth information recorded in the access stack structure and the estimated maximum dependency chain length, a maximum recursive depth threshold is set to control the search depth, and the maximum recursive depth threshold is used to limit the maximum search depth of the depth-first search process;

[0035] When a new node is accessed, the node information is pushed into the access stack structure, and the first bit of the node is marked as a visited state in the access mark bitmap, and the second bit is marked as a state on the current path;

[0036] During the node access process, the access timestamp information in the access stack structure is continuously recorded and updated, and the existing circular dependency is identified by comparing the access timestamps of adjacent nodes;

[0037] When the node access depth reaches the maximum recursive depth threshold, the current access path information is recorded, and the depth critical point is marked in the access mark bitmap, and then backtracking to the previous node in the access stack structure;

[0038] After the node access is completed, the node information is popped from the access stack structure, and the access status of the node is updated in the access mark bitmap, and the second bit is reset to a state of not being on the current path;

[0039] A depth control mechanism provided based on the complete access path recorded by the access stack structure, the access mark bitmap and the maximum recursive depth threshold is used to assist the execution of depth-first search.

[0040] Performing transitive closure analysis on the hierarchical directed acyclic graph, identifying and removing redundant dependency edges, and obtaining a simplified directed acyclic graph; based on the simplified directed acyclic graph, using a topological sorting algorithm to generate a data flow path includes:

[0041] Constructing an adjacency matrix to represent the dependency relationship between nodes in the hierarchical directed acyclic graph, wherein the row and column identifiers of the adjacency matrix correspond to the node identifiers, and the element values ​​of the adjacency matrix represent whether there is a direct dependency between the nodes;

[0042] Performing matrix iteration operation based on the adjacency matrix to obtain a transitive closure matrix, wherein the transitive closure matrix is ​​used to represent the transitive dependency relationship between nodes, and the element value of the transitive closure matrix represents whether there is a transitive path between corresponding nodes;

[0043] Traversing the direct dependency edges in the adjacency matrix, detecting each direct dependency edge based on the transitive closure matrix, and marking the direct dependency edge as a redundant dependency edge when there is a transitive path between node pairs via an intermediate node;

[0044] The redundant dependency edges are removed to obtain a simplified directed acyclic graph, the in-degree value of each node is calculated based on the simplified directed acyclic graph, and the nodes with zero in-degree values ​​are added to a zero-in-degree node queue;

[0045] Taking a node from the zero-in-degree node queue and adding it to the access order list, updating the in-degree values ​​of all successor nodes pointed to by the node, and adding the successor node to the zero-in-degree node queue when the in-degree value of the successor node is updated to zero;

[0046] The operation of taking out nodes from the zero-in-degree node queue is repeatedly performed until the zero-in-degree node queue is empty, thereby obtaining a final access sequence list, which is the data flow path.

[0047] According to a second aspect of the embodiments of the present invention,

[0048] Provides an integrated metadata management system that is scalable based on data sources, including:

[0049] The first unit is used to perform real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collect connection addresses, access credentials and protocol types of the multiple heterogeneous data sources, establish a dynamic connection pool according to the connection addresses, and set the maximum number of connections to N, the timeout period to T seconds, and the number of retries to M times in the dynamic connection pool; use the connection instances in the dynamic connection pool to respectively execute a SELECT statement to obtain a table structure list, execute a SHOW COLUMNS statement to obtain a field attribute list, and execute a SHOW KEYS statement to obtain an association relationship list, and assemble the obtained table structure list, field attribute list and association relationship list into unified JSON format data;

[0050] The second unit is used to parse and standardize the JSON format data, extract field names, field types, primary key information and foreign key information, and construct an N×N dimensional association calculation matrix; in the association calculation matrix, the field naming similarity algorithm is used to calculate the similarity score of the field name, the data type compatibility rule is used to calculate the matching score of the field type, and the inter-table association score is calculated based on the foreign key constraint relationship, and the weighted average of the similarity score, the matching score and the inter-table association score is used as the final field mapping association; a standardized data structure description file is generated according to the field mapping association;

[0051] The third unit is used to establish a three-layer index directory based on the data structure description file using a B+ tree index structure, wherein the first layer index records the data source type and its physical storage location, the second layer index records the business attributes and its access path, and the third layer index records the usage scenario and its version information; the three-layer index directory is used to poll the structure changes of the multiple heterogeneous data sources at a frequency of F seconds / times, and when a change in the table structure, field attribute or association relationship is detected, the data structure description file is updated again; according to the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationship between data tables, and a data flow path in the form of a DAG directed acyclic graph is generated.

[0052] According to a third aspect of the embodiments of the present invention,

[0053] An electronic device is provided, comprising:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] A fourth aspect of the embodiments of the present invention is:

[0058] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0059] The beneficial effects of this application are as follows:

[0060] The PF4J plug-in framework enables real-time detection of multiple data sources and dynamic connection pool management, which can flexibly adapt to different types of data source systems and improve the efficiency and reliability of metadata collection. The parameter configuration of the dynamic connection pool ensures the rational use of system resources and avoids the waste of connection resources. At the same time, the standardized processing of heterogeneous data is achieved through unified JSON format conversion.

[0061] A scientific field mapping mechanism was established by adopting a multi-dimensional correlation calculation method, taking into account field naming similarity, data type compatibility, and inter-table associations. This method not only improves the accuracy of metadata mapping, but also automatically identifies and processes complex data structure relationships, greatly reducing the need for manual intervention.

[0062] The three-layer index structure design based on B+ tree realizes efficient metadata retrieval and real-time update mechanism. Through regular polling detection and depth-first search algorithm, the system can timely discover and respond to changes in data structure, automatically maintain data flow path, ensure the real-time and accuracy of metadata management, and provide reliable support for data governance and analytical decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of an integrated metadata management method that is extensible based on data source according to an embodiment of the present invention;

[0064] Figure 2 This is a metadata collection timing diagram of an embodiment of the present invention;

[0065] Figure 3 This is a diagram of metadata collection combined with plug-in interaction according to an embodiment of the present invention;

[0066] Figure 4 A schematic diagram of a process for dynamically loading / unloading plug-ins according to an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of a flow chart for dynamically updating plug-in connection parameters according to an embodiment of the present invention;

[0068] Figure 6 The diagram is a structural diagram of an integrated metadata management system that is expandable based on data sources according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0071] Figure 1 FIG. 1 is a flow chart of an integrated metadata management method that is scalable based on data sources according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0072] S11. Perform real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collect the connection addresses, access credentials and protocol types of the multiple heterogeneous data sources, establish a dynamic connection pool according to the connection addresses, and set the maximum number of connections to N, the timeout period to T seconds, and the number of retries to M times in the dynamic connection pool; use the connection instances in the dynamic connection pool to respectively execute the SELECT statement to obtain the table structure list, execute the SHOW COLUMNS statement to obtain the field attribute list, and execute the SHOW KEYS statement to obtain the association relationship list, and assemble the obtained table structure list, field attribute list and association relationship list into unified JSON format data;

[0073] S12. Parse and standardize the JSON format data, extract field names, field types, primary key information and foreign key information, and construct an N×N dimensional association calculation matrix; in the association calculation matrix, use the field naming similarity algorithm to calculate the similarity score of the field name, use the data type compatibility rule to calculate the matching score of the field type, calculate the inter-table association score based on the foreign key constraint relationship, and use the weighted average of the similarity score, the matching score and the inter-table association score as the final field mapping association; generate a standardized data structure description file according to the field mapping association;

[0074] S13. Based on the data structure description file, a three-layer index directory is established using a B+ tree index structure, wherein the first-layer index records the data source type and its physical storage location, the second-layer index records the business attributes and its access path, and the third-layer index records the usage scenario and its version information; the three-layer index directory is used to poll the structure changes of the multiple heterogeneous data sources at a frequency of F seconds / times, and when changes in the table structure, field attributes or association relationships are detected, the data structure description file is updated again; based on the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationships between data tables, and a data flow path in the form of a DAG directed acyclic graph is generated.

[0075] In an optional implementation, in the association calculation matrix, the field name similarity score is calculated using a field naming similarity algorithm, the field type matching score is calculated using a data type compatibility rule, and the inter-table association score is calculated based on a foreign key constraint relationship. The weighted average of the similarity score, the matching score, and the inter-table association score is used as the final field mapping association degree, including:

[0076] The association degree calculation matrix includes a to-be-mapped field identifier and a score storage unit, and the score storage unit is used to store the calculated scores generated during the field mapping process;

[0077] Obtaining a field name corresponding to the to-be-mapped field identifier, performing word segmentation preprocessing on the field name to obtain a root list, calculating edit distance values ​​between roots in the root list using a field naming similarity algorithm, normalizing the edit distance values ​​to obtain a similarity score, and writing the similarity score into the score storage unit based on the to-be-mapped field identifier;

[0078] Acquire field data type information according to the to-be-mapped field identifier, extract compatibility rules and conversion rules matching the field data type information from a preset type mapping rule library, calculate a direct compatibility score based on the compatibility rule, calculate a conversion cost score based on the conversion rule, use a weighted combination value of the direct compatibility score and the conversion cost score as a matching score, and write the matching score into the score storage unit based on the to-be-mapped field identifier;

[0079] Using the to-be-mapped field identifier to query the foreign key constraint information of the data table to which the field belongs, parsing the primary key field information and the reference constraint information from the foreign key constraint information, constructing an associated field index based on the primary key field information, calculating an associated weight using the reference constraint information, taking the product of the associated field index and the associated weight as an inter-table association score, and writing the inter-table association score into the score storage unit based on the to-be-mapped field identifier;

[0080] The similarity score, matching score and inter-table association score of the current field to be mapped are read from the score storage unit using the field identifier to be mapped, and the read similarity score, matching score and inter-table association score are weighted averaged to obtain the field mapping association degree.

[0081] In the process of building the correlation calculation matrix, a two-dimensional array structure is first created to store the identifiers and calculation scores of the fields to be mapped. The rows and columns of the matrix correspond to the fields of the source table and the target table respectively, and each cell stores the mapping correlation calculation results between the corresponding fields.

[0082] When calculating the similarity of field names, the field names are preprocessed by word segmentation and root extraction. For example, the field name "user account name" is segmented to obtain ["user", "account", "name"]. The plural form is converted to singular through word form restoration, and the verb conjugation is unified to the original form. Then the edit distance between the roots is calculated, such as the edit distance between "account" and "acct" is 2. The edit distance value is mapped to the range of 0-1 to obtain the similarity score.

[0083] When evaluating data type compatibility, a type mapping rule base is predefined. For example, the VARCHAR type is directly compatible with the CHAR type, with a score of 1.0; the conversion of the INT type to the VARCHAR type requires additional processing, with a score of 0.8. For cases where type conversion is required, a cost score is calculated based on the conversion complexity. The final type matching score is obtained by weighting the direct compatibility score and the conversion cost score in a ratio of 7:3.

[0084] The analysis of associations between tables is based on foreign key constraint information. The primary key fields and reference relationships are obtained by parsing the foreign key definitions. For example, the customer ID field in the order table references the id primary key in the customer table, and an association index is constructed and assigned a higher weight. Multi-level associations are processed recursively, and the weight decreases with the association level.

[0085] The final field mapping correlation is calculated by weighted average. The weights of field name similarity, type matching, and inter-table correlation are 0.4, 0.3, and 0.3, respectively. When the correlation exceeds the preset threshold (such as 0.8), it can be considered that there is a valid mapping relationship between the two fields.

[0086] The solution of this application can:

[0087] Through the field naming similarity algorithm and word segmentation preprocessing, fields expressing the same meaning under different naming specifications can be accurately identified, improving the accuracy and robustness of field mapping. The use of data type compatibility rules and conversion cost evaluation mechanisms not only ensures the feasibility of data migration, but also selects the optimal type mapping solution, reducing conversion costs and data loss risks. The introduction of foreign key constraint analysis and multi-level association processing fully utilizes the structured information in the database model, making the field mapping results more consistent with business semantics and improving the practicality and reliability of the mapping solution.

[0088] In an optional implementation, detecting structural changes of the plurality of heterogeneous data sources by polling the three-layer index directory at a frequency of F seconds / times, and when detecting changes in table structure, field attributes or association relationships, re-updating the data structure description file includes:

[0089] Detecting structural changes of the plurality of heterogeneous data sources by polling the three-layer index directory at a frequency of F seconds / times, the three-layer index directory comprising a data source-level directory, a table-level directory and a field-level directory, establishing a first association relationship between the table-level directory and the data source-level directory, and establishing a second association relationship between the field-level directory and the table-level directory;

[0090] Reading a data source version mark stored in the data source level directory, and comparing the data source version mark with the real-time version marks of the multiple heterogeneous data sources, wherein the data source version mark is used to identify the structural status of the multiple heterogeneous data sources;

[0091] When the data source version mark is inconsistent with the real-time version mark, obtaining a changed heterogeneous data source identifier, and obtaining a table structure identifier and a table structure version mark from the table-level directory according to the first association relationship, wherein the table structure version mark is used to identify the state of the table structure;

[0092] The table structure version mark is compared with the real-time table structure version mark in the changed heterogeneous data source, and when a table structure change is detected, a field attribute identifier and a field attribute version mark are obtained from the field-level directory according to the second association relationship, wherein the field attribute version mark is used to identify the state of the field attribute;

[0093] The field attribute version tag is compared with the real-time field attribute version tag in the changed table structure. When a change in field attribute or association relationship is detected, the changed field attribute information and association relationship information are obtained; and the data structure description file is re-updated based on the changed field attribute information and the association relationship information.

[0094] A heterogeneous data source structure change detection method based on a three-layer index directory achieves efficient monitoring and updating of heterogeneous data sources by establishing a multi-level index structure at the data source level, table level and field level.

[0095] First, a three-layer index directory structure is constructed. The data source level directory stores basic information about the data source, including data source identification, data source type, connection parameters, and version tags; the table level directory stores table structure information, including table name, field list, primary key information, and version tags; the field level directory stores field attribute information, including field name, data type, length limit, default value, and version tag. The three-layer directory is organically connected through association relationships to achieve a complete description of the data structure.

[0096] Perform polling detection at a preset frequency of F seconds. Specifically, it can be set to detect once every 300 seconds. When the system load is low, the frequency can be appropriately increased. When detecting, first read the version tag in the data source level directory. The tag can be in the form of a timestamp, such as "20240315120000" to indicate the last update time. Compare it with the real-time version tag of the heterogeneous data source. If it is inconsistent, it means that a structural change has occurred.

[0097] When a change at the data source level is found, the affected table-level directory is located based on the first association relationship. For example, if a user information table is added to a relational data source, the structure identifier "USER_INFO" and its version tag of the table need to be obtained. Compare the table-level version tag with the real-time version to confirm whether the table structure has changed.

[0098] If a table structure change is detected, the field-level directory is queried based on the second association relationship. For example, if a "telephone number" field is added to the user information table, the attribute identifier and version tag of the field need to be obtained. By comparing the field attribute version tag, the specific change content can be confirmed, such as the data type changed from VARCHAR(20) to VARCHAR(50).

[0099] Finally, based on the detected change information, the data structure description file is updated. The description file is stored in JSON format and contains complete data source, table structure and field attribute information.

[0100] The solution of this application can:

[0101] By establishing a three-layer index directory structure, this solution can achieve accurate positioning and efficient detection of heterogeneous data sources, greatly improving the performance and accuracy of structural change monitoring. The multi-level version marking mechanism can quickly identify the specific location where the change occurred, avoiding the waste of resources caused by full scanning and significantly reducing system overhead. The change transmission mechanism based on association relationships ensures the real-time and consistency of data structure description files, provides a reliable data structure reference for upper-level applications, and improves the reliability of heterogeneous data source integration.

[0102] In an optional implementation, according to the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationship between data tables, and a data flow path in the form of a directed acyclic graph (DAG) is generated, including:

[0103] Parsing the updated data structure description file, extracting inter-table association information in the data structure description file, wherein the inter-table association information includes primary and foreign key relationship information, field reference relationship information, and business association relationship information;

[0104] Constructing an adjacency list structure based on the association information between tables, wherein each node in the adjacency list structure includes a table identifier, table attribute information, reference field information and association type information, and the association type information is used to identify the dependency strength between nodes;

[0105] Initialize a depth-first search environment, create an access stack structure for recording access paths, create an access mark bitmap for avoiding repeated accesses, and set a maximum recursive depth threshold for controlling the search depth. The access stack structure, the access mark bitmap, and the maximum recursive depth threshold are used to assist in the execution of the depth-first search;

[0106] Using a depth-first search algorithm to traverse the adjacency list structure, pushing the current access node into the access stack structure and marking the accessed state in the access mark bitmap, obtaining a set of adjacent nodes of the current access node;

[0107] For each to-be-visited node in the adjacent node set, detecting whether the to-be-visited node already exists in the access stack structure, and when the detection result is yes, extracting the node information in the access stack structure to generate a ring dependency path;

[0108] Analyze the association type information between the nodes in the ring dependency path, and when there is a strong dependency relationship between the nodes, add a strong dependency mark to the corresponding edge in the adjacency list structure; when there is a non-strong dependency relationship between the nodes, delete the edge forming the loop from the adjacency list structure to obtain an optimized adjacency list structure;

[0109] Converting the optimized adjacency list structure into a directed acyclic graph, layering the directed acyclic graph based on the dependency depth of the nodes, assigning nodes with the same dependency depth to the same level, and obtaining a hierarchical directed acyclic graph;

[0110] A transitive closure analysis is performed on the hierarchical directed acyclic graph to identify and remove redundant dependency edges to obtain a simplified directed acyclic graph; based on the simplified directed acyclic graph, a topological sorting algorithm is used to generate a data flow path.

[0111] The present invention provides a method for calculating reference dependencies between data tables based on a depth-first search algorithm, which is used to generate a data flow path in the form of a directed acyclic graph (DAG). The method comprises the following steps:

[0112] First, parse the updated data structure description file. This file contains the structure information of all data tables in the system and the relationship between tables. During the parsing process, focus on extracting the relationship between tables, including primary and foreign key relationships, field reference relationships, and business relationship relationships. For example, for an order management system, there may be a primary and foreign key relationship between the "order table" and the "user table", a field reference relationship between the "order table" and the "product table", and a business relationship between the "order table" and the "logistics table".

[0113] Next, an adjacency list structure is constructed based on the extracted inter-table association information. The adjacency list is a commonly used graph data structure suitable for representing sparse graphs. In this method, each node of the adjacency list represents a data table and contains the following information: table identifier (such as table name), table attribute information (such as field list), reference field information (such as foreign key field), and association type information (used to identify dependency strength). For example, the "order table" node may contain information such as order ID, user ID (foreign key), product ID (reference field), as well as a strong dependency relationship with the "user table" and a weak dependency relationship with the "logistics table".

[0114] After building the adjacency list, initialize the depth-first search (DFS) environment. This includes creating a visit stack structure, a visit mark bitmap, and setting the maximum recursion depth threshold. The visit stack is used to record the current access path, the visit mark bitmap is used to avoid repeated visits to the same node, and the maximum recursion depth threshold is used to prevent excessive recursion from causing stack overflow. For example, you can use an array to implement the visit stack, a Boolean array to implement the visit mark bitmap, and set the maximum recursion depth to 100.

[0115] Then, start the depth-first search algorithm. Select an unvisited node from the adjacency list as the starting point, push it into the visit stack and mark it as visited. Get all the neighboring nodes of this node (that is, other tables associated with this table). For each neighboring node, check if it is already in the visit stack. If so, it means a circular dependency has been found.

[0116] When a circular dependency is found, further analysis is required. Extract the node information that forms the loop in the access stack and generate a circular dependency path. For example, a circular dependency such as "Order Table" -> "Payment Table" -> "Refund Table" -> "Order Table" may be found. Then analyze the association type information between the nodes in the ring. If there is a strong dependency relationship (such as a primary and foreign key relationship), add a strong dependency mark to the corresponding edge in the adjacency table. If they are all non-strong dependencies (such as business associations), delete an edge that forms a loop from the adjacency table, usually choosing to delete the edge with the weakest dependency strength.

[0117] After completing the loop detection and processing, the optimized adjacency list structure is obtained. This structure is converted into a directed acyclic graph (DAG). The DAG is layered based on the dependency depth of the nodes, and nodes with the same dependency depth are assigned to the same level. For example, the bottom layer may contain "user table" and "product table", the second layer may be "order table", and the third layer may be "payment table" and "logistics table".

[0118] Perform transitive closure analysis on the hierarchical DAG to identify and remove redundant dependency edges. For example, if there are dependencies between "order table"->"payment table"->"refund table" and "order table"->"refund table", the redundant edge "order table"->"refund table" can be removed. This step can be implemented using the Floyd-Warshall algorithm with a time complexity of O(n^3), where n is the number of nodes.

[0119] Finally, based on the streamlined DAG, a topological sorting algorithm is used to generate a data flow path. Topological sorting can ensure that the dependent data of each table is ready during the data processing process. For example, the generated data flow path may be: user table -> product table -> order table -> payment table -> logistics table -> refund table.

[0120] Through the above steps, this method can effectively analyze complex data table dependencies, generate optimized data flow paths, and provide important references for data processing and system design.

[0121] The solution of this application can:

[0122] The method provided by the present invention can effectively identify and process complex dependencies between data tables, including circular dependencies, so as to generate an optimized data flow path. This is of great significance for the design and optimization of large-scale data systems, and can improve data processing efficiency and reduce the risk of data inconsistency. By using a depth-first search algorithm and an adjacency list structure, the present method can efficiently process large-scale data table relationship networks. Even when there are a large number of tables and complex relationships, analysis and optimization can be completed within a reasonable time, with good scalability. The present method generates a more streamlined and reasonable data flow path by analyzing dependency strength, removing redundant edges, etc. This not only helps to optimize the data processing process, but also provides valuable insights for database design and system architecture optimization, and improves the performance and maintainability of the entire system.

[0123] In an optional implementation, creating an access mark bitmap to avoid repeated access, setting a maximum recursive depth threshold to control the search depth, and using the access stack structure, the access mark bitmap, and the maximum recursive depth threshold to assist in the execution of a depth-first search include:

[0124] Based on the number of nodes to be recorded in the access stack structure, an access mark bitmap is created to avoid repeated access, wherein the access mark bitmap uses two binary bits to represent the access status of the node, wherein the first bit represents whether the node is accessed, and the second bit represents whether the node is on the current access path;

[0125] According to the node depth information recorded in the access stack structure and the estimated maximum dependency chain length, a maximum recursive depth threshold is set to control the search depth, and the maximum recursive depth threshold is used to limit the maximum search depth of the depth-first search process;

[0126] When a new node is accessed, the node information is pushed into the access stack structure, and the first bit of the node is marked as a visited state in the access mark bitmap, and the second bit is marked as a state on the current path;

[0127] During the node access process, the access timestamp information in the access stack structure is continuously recorded and updated, and the existing circular dependency is identified by comparing the access timestamps of adjacent nodes;

[0128] When the node access depth reaches the maximum recursive depth threshold, the current access path information is recorded, and the depth critical point is marked in the access mark bitmap, and then backtracking to the previous node in the access stack structure;

[0129] After the node access is completed, the node information is popped from the access stack structure, and the access status of the node is updated in the access mark bitmap, and the second bit is reset to a state of not being on the current path;

[0130] A depth control mechanism provided based on the complete access path recorded by the access stack structure, the access mark bitmap and the maximum recursive depth threshold is used to assist the execution of depth-first search.

[0131] The solution of this application can:

[0132] By using a bitmap structure with two binary bits to represent the node access status, memory usage is significantly reduced, while providing efficient status query and update operations, resulting in better performance during large-scale node traversal. The use of an access stack structure combined with a timestamp mechanism can accurately identify and record circular dependencies. By comparing the access timestamps of adjacent nodes, the specific location and formation time of the circular dependency can be quickly located, providing an important basis for subsequent optimization. The depth control mechanism of the maximum recursive depth threshold is introduced to effectively prevent stack overflow problems that may occur in depth-first search, and by recording the depth critical point information, it provides reliable data support for analyzing and optimizing complex dependency chains.

[0133] In an optional implementation, performing transitive closure analysis on the hierarchical directed acyclic graph, identifying and removing redundant dependency edges, and obtaining a simplified directed acyclic graph; based on the simplified directed acyclic graph, using a topological sorting algorithm to generate a data flow path includes:

[0134] Constructing an adjacency matrix to represent the dependency relationship between nodes in the hierarchical directed acyclic graph, wherein the row and column identifiers of the adjacency matrix correspond to the node identifiers, and the element values ​​of the adjacency matrix represent whether there is a direct dependency between the nodes;

[0135] Performing matrix iteration operation based on the adjacency matrix to obtain a transitive closure matrix, wherein the transitive closure matrix is ​​used to represent the transitive dependency relationship between nodes, and the element value of the transitive closure matrix represents whether there is a transitive path between corresponding nodes;

[0136] Traversing the direct dependency edges in the adjacency matrix, detecting each direct dependency edge based on the transitive closure matrix, and marking the direct dependency edge as a redundant dependency edge when there is a transitive path between node pairs via an intermediate node;

[0137] The redundant dependency edges are removed to obtain a simplified directed acyclic graph, the in-degree value of each node is calculated based on the simplified directed acyclic graph, and the nodes with zero in-degree values ​​are added to a zero-in-degree node queue;

[0138] Taking a node from the zero-in-degree node queue and adding it to the access order list, updating the in-degree values ​​of all successor nodes pointed to by the node, and adding the successor node to the zero-in-degree node queue when the in-degree value of the successor node is updated to zero;

[0139] The operation of taking out nodes from the zero-in-degree node queue is repeatedly performed until the zero-in-degree node queue is empty, thereby obtaining a final access sequence list, which is the data flow path.

[0140] First, we construct an adjacency matrix to represent the node dependencies of a hierarchical directed acyclic graph. Take a directed acyclic graph with five nodes as an example, where the nodes are labeled A, B, C, D, and E. By traversing each edge in the graph, we mark the corresponding position in the adjacency matrix with 1 to indicate a direct dependency, and 0 to indicate no dependency. For example, the edge from node A to node B is marked as 1 in row A and column B of the adjacency matrix.

[0141] Then perform matrix iteration calculation to obtain the transitive closure matrix. The specific process is to repeatedly traverse the adjacency matrix. When it is found that node i can reach node j, and node j can reach node k, it means that node i can indirectly reach node k through node j. At this time, the value of the i-th row and k-th column in the transitive closure matrix is ​​marked as 1. After multiple rounds of iterations, the complete transitive closure matrix is ​​finally obtained. The 1 in the matrix indicates that there is a reachable path between the corresponding node pairs.

[0142] Then, redundant dependency edges are detected and removed. For example, if the path A->B->C is found in the transitive closure matrix, it means that the direct dependency edge A->C is redundant and can be removed. The specific method is to traverse the position with a value of 1 in the adjacency matrix and check whether there is another path in the transitive closure matrix that can connect this pair of nodes. If so, this edge is marked as a redundant edge.

[0143] After removing all redundant edges, a simplified directed acyclic graph is obtained. Then calculate the in-degree value of each node in the graph. The in-degree value indicates the number of edges pointing to the node. Add nodes with an in-degree value of 0 to the queue. These nodes are the nodes that can be accessed first. For example, if the in-degree of node A is 0, add A to the queue.

[0144] Take the node out of the queue and add it to the access order list, and update the in-degree values ​​of all successor nodes pointed to by the node. For example, after visiting node A, reduce the in-degree values ​​of all its successor nodes by 1. If the in-degree value of a successor node becomes 0, add the node to the queue. Repeat this process until the queue is empty.

[0145] The access order list finally obtained is the data flow path that conforms to the dependency relationship. For example, the sequence [A, B, D, C, E] is obtained, which means that the data should flow between nodes in this order.

[0146] The solution of this application can:

[0147] By identifying and removing redundant dependency edges through transitive closure analysis, the structure of the directed acyclic graph is simplified, the complexity of the graph is reduced, and the data flow path is clearer and more intuitive. Topological sorting based on the streamlined directed acyclic graph ensures that the generated data flow path strictly follows the dependency relationship between nodes, avoids circular dependencies, and ensures the correctness of data processing. The use of adjacency matrix representation and queue-assisted calculation improves the execution efficiency of the algorithm, so that a reasonable data flow path can be quickly generated when there are a large number of nodes.

[0148] Figure 2 This is a metadata collection timing diagram of an embodiment of the present invention, such as Figure 2 As shown, the method also includes:

[0149] First, the collection caller initializes the collection plug-in by loading the plug-in interface locally. During this process, the system automatically scans the configuration file in the plug-in directory and reads the basic information of the plug-in, including the plug-in ID, version number, supported data source type, etc. The system loads this information into the plug-in registry in memory for quick subsequent calls. For example, the configuration information of a collection plug-in includes: plug-in ID "PLUGIN_001", version "2.0.1", supported MySQL data source, etc.

[0150] Next, the collection caller calls the metadata acquisition method to pass the collection parameters to the collection plug-in. The collection parameters include data source connection information, collection table configuration, field mapping rules, etc. After receiving the request, the collection plug-in first verifies the integrity and legality of the parameters. For example, check whether the required parameters are complete and whether the parameter format meets the requirements.

[0151] The collection plug-in then uses the configured connection information to connect to the data source. The connection process includes: establishing a network connection, identity authentication, selecting a target database, etc. After the connection is successful, the plug-in caches the connection object in the connection pool for subsequent data interaction.

[0152] The data source returns metadata information, including table structure definition, field attributes, etc. For example, a table contains fields: id (integer), name (character), create_time (time), etc. These raw metadata information will be received by the plug-in and initially processed.

[0153] The plug-in performs metadata structure conversion, converting the original metadata into a standard format. The conversion process includes: field type standardization, field attribute supplementation, index information extraction, etc. The converted metadata is convenient for subsequent unified processing.

[0154] The collection plug-in returns the converted metadata to the caller. The returned information adopts a unified data structure, including complete content such as table information, field information, index information, etc.

[0155] After receiving the metadata, the caller performs a structural conversion process. The conversion process mainly converts the standard format returned by the plug-in into the format used within the application system to ensure compatibility with the existing system.

[0156] The converted metadata is persistently stored. The storage process includes: generating a unique identifier, recording version information, establishing index relationships, etc., to facilitate subsequent rapid retrieval and use.

[0157] Finally, the collection caller uninstalls the collection plug-in through the uninstall plug-in interface. The uninstall process includes: closing the data source connection, cleaning up cache resources, and releasing system resources occupied by the plug-in.

[0158] The solution of this application can:

[0159] Technical effect: A plug-in data collection architecture is implemented to support flexible expansion of different types of data sources; the consistency of data structure is ensured through unified metadata standardization processing; the connection pool mechanism is adopted to improve the reuse efficiency of data source connections. Performance effect: The overhead of repeatedly establishing data source connections is reduced, and the system response speed is improved; the system stability is improved through parameter verification and exception handling mechanisms; the resource release process is optimized and the system resource usage is reduced. Application effect: The access process of adding new data sources is simplified, and the scalability of the system is improved; the unified metadata format facilitates subsequent data processing and analysis; the standardized interface design improves the efficiency of system maintenance.

[0160] Figure 3 This is a diagram of metadata collection combined with plug-in interaction in an embodiment of the present invention, such as Figure 3 As shown, the method also includes:

[0161] The metadata collector first receives the collection task request through the standardized collection API. The request contains data source connection information, such as database access address, port number, username and password, target database name, table list to be collected, and other configuration parameters. The collector dynamically identifies and loads the corresponding collection plug-in file from the plug-in library according to different data source types.

[0162] The standardized collection API performs a validity check on the received request parameters. The verification content includes: integrity check of data source connection information, parameter format verification, access permission confirmation, etc. After the verification is passed, the collection API selects the corresponding collection plug-in according to the data source type and passes the request parameters to the plug-in processing module.

[0163] The acquisition plug-in module is responsible for establishing a connection with the data source and extracting metadata information. Taking a relational database as an example, the plug-in module first attempts to establish a connection with the data source. After the connection is successful, it queries the database's system table to obtain the table structure definition. The original metadata information obtained includes the basic information of the table, field attributes, index constraints, etc.

[0164] The format converter receives the original metadata from the acquisition plug-in and performs standardized conversion processing. The conversion process mainly includes: mapping the data types of different data sources to unified standard types, unifying the representation of field attributes, processing special characters and naming conventions, etc. The converted metadata is stored in a unified structured format.

[0165] The persistent storage module writes the standardized metadata into the storage system. The storage process includes: generating a globally unique identifier for the metadata, recording the data source information and collection time, and establishing a metadata index. The complete metadata information stored includes: metadata identifier, data source type, table structure definition, field attribute list, collection timestamp, etc.

[0166] The original metadata information of a data table includes: the table name is user information table, including basic information such as user identification field (integer, length 11, primary key), user name field (character type, length 50, non-empty), creation time field (date and time type), etc.

[0167] After standardization, the metadata records are: table identification number, data source type identification, table name, and the field information list contains field identification, field name, standardized field type, field length, field constraints and other normalized information.

[0168] The solution of this application can:

[0169] Architectural effect: The plug-in architecture design supports flexible expansion of different types of data sources; realizes the standardized process of metadata collection and processing; the responsibilities of each functional module are clear, and the system has strong scalability. Functional effect: Automatically complete the metadata collection of heterogeneous data sources; realize the unified conversion and standardized processing of metadata formats; provide complete metadata storage and retrieval functions. Operation and maintenance effect: Reduce the difficulty of accessing new data sources; improve the standardization and maintainability of metadata management; enhance the stability and reliability of system operation.

[0170] Figure 4 The following is a schematic diagram of the process of dynamically loading / unloading plug-ins according to an embodiment of the present invention. Figure 4 As shown, the method also includes:

[0171] The caller initializes the collection plug-in through the loading method of the plug-in manager. The plug-in loading process first scans the plug-in files in the specified directory and reads the plug-in configuration information, including basic information such as the plug-in name, version number, and supported data source types. For each scanned plug-in file, the manager verifies the file integrity, parses the plug-in description information, loads the plug-in into the memory, and initializes the environment required for the plug-in to run.

[0172] After receiving the loading request, the plug-in management module registers the plug-in through the registration manager. During the registration process, the system will fully verify the plug-in, including checking the plug-in integrity, version compatibility verification, assigning a unique plug-in identifier, and establishing a corresponding index in the registry. The information of the successfully registered plug-in will be persisted and saved for subsequent calls and management.

[0173] When the caller uses a standardized interface to access a data source, the system automatically selects the corresponding database dialect implementation based on the data source type. For example, when accessing a MySQL data source, the system will call the dialect implementation in the MySQL-specific plug-in to automatically handle specific operations such as data type mapping conversion and query syntax optimization. In this way, the system can adaptively handle the special requirements of different data sources.

[0174] The unified interface layer defines a set of standard data source access methods, covering core functions such as initializing connections, obtaining table structures, and executing queries. Each data source plug-in needs to implement these standard interfaces to ensure consistency of operations. In actual execution, the MySQL plug-in will query its system tables to obtain table structure information, while the Oracle plug-in will query its unique data dictionary to achieve the same function.

[0175] After the business processing is completed, the caller closes the plug-in through the plug-in manager's uninstall method. During the uninstall process, the logout manager will perform a series of cleanup tasks, including closing all active data source connections, cleaning cache data generated during the plug-in operation, and releasing system resources occupied by the plug-in. This ensures the timely recovery and effective use of system resources.

[0176] In actual applications, when it is necessary to collect information from a MySQL data source, the plug-in manager will load a MySQL-specific plug-in, establish a database connection, and read the structure definition of the user information table, including field number, name, creation time, etc. When collecting Oracle data sources, the system will load an Oracle-specific plug-in, connect to the specified database instance, and obtain the structural information of the employee table, including field definitions such as employee number, name, and date of employment.

[0177] The solution of this application can:

[0178] Technical effect: Implemented an extensible plug-in data source access architecture that supports unified access to multiple heterogeneous data sources. Standardized interface definitions ensure consistency in access methods for different data sources, and the dialect mechanism effectively solves the syntax differences between different data sources. Management effect: A complete plug-in lifecycle management mechanism ensures the rational use of system resources. The centralized plug-in registry provides a unified management entry, and the standardized error handling mechanism improves the maintainability and stability of the system. Performance effect: The dynamic loading mechanism reduces system resource usage and enables on-demand loading. Unified connection pool management improves data access efficiency, and the intelligent caching strategy optimizes query performance and improves the overall response speed of the system.

[0179] Figure 5 FIG. 1 is a flow chart of dynamically updating plug-in connection parameters according to an embodiment of the present invention. Figure 5 As shown, the method also includes:

[0180] The plugin manager first performs an uninstall operation to clean up existing plugin instances in the system. The uninstall process closes existing connections, releases resources, and cleans up cached data to ensure that the system is in a clean state. This step prepares for the subsequent loading of new plugins to avoid resource conflicts.

[0181] The plugin loading and startup process is then executed. The plugin manager scans the specified directory and identifies available data source plugin files. The loading process verifies the integrity of the plugin and parses the plugin configuration information, including key information such as supported data source types and version numbers. The startup phase initializes the plugin runtime environment and prepares the necessary system resources.

[0182] After loading is complete, the system obtains the connection configuration parameters. These parameters include the data source connection address, port number, authentication information, etc. The configuration information will be standardized to ensure that it meets the connection requirements of the target data source. The system will also set connection pool parameters, such as the maximum number of connections, timeout, retry strategy, etc.

[0183] Next, the system calls the data source dialect interface to obtain a connection instance. The dialect interface selects the corresponding implementation class based on the data source type and handles the specific connection logic of different data sources. For example, the connection parameter format and authentication method of the MySQL data source and the Oracle data source are different, and the dialect interface will perform adaptation conversion.

[0184] Finally, the connection is initialized and the system attempts to establish a connection with the target data source. The initialization process includes: verifying connection parameters, establishing a physical connection, setting session attributes, initializing the connection pool, etc. After the connection is successful, the system will perform a connection test to ensure the connection availability.

[0185] When you need to connect to a MySQL data source, the system loads the mysql-plugin plugin and sets connection parameters such as host address, port 3306, database name, and user credentials. The connection pool is configured with a maximum number of connections of 20 and a connection timeout of 30 seconds. The system establishes a connection through the MySQL dialect interface and performs a connection test.

[0186] For Oracle data source connection, the system loads the oracle-plugin plug-in and configures the service name, port 1521, instance name and other connection information. The minimum number of connections in the connection pool is set to 5 and the maximum number of connections is set to 30. The connection string format is processed through the Oracle-specific dialect to complete the connection creation.

[0187] The solution of this application can:

[0188] Architectural effect: The plug-in design realizes the modularization and standardization of data source connection management, and supports flexible expansion of different types of data sources. The unified connection management interface simplifies the difficulty of system maintenance and improves code reusability. Performance effect: The intelligent connection pool management mechanism optimizes system resource utilization and reduces the overhead of connection establishment and release. The dynamic loading mechanism ensures the on-demand allocation of system resources and improves the overall operating efficiency. Reliability effect: The perfect exception handling and retry mechanism improves the reliability of connection management. Standardized parameter verification and connection testing ensure the stability of data source connections. The automatic resource recovery mechanism prevents resource leakage and enhances system stability.

[0189] Figure 6 FIG. 1 is a schematic diagram of a structure of an integrated metadata management system that is expandable based on a data source according to an embodiment of the present invention. Figure 6 As shown, the system comprises:

[0190] The first unit is used to perform real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collect connection addresses, access credentials and protocol types of the multiple heterogeneous data sources, establish a dynamic connection pool according to the connection addresses, and set the maximum number of connections to N, the timeout period to T seconds, and the number of retries to M times in the dynamic connection pool; use the connection instances in the dynamic connection pool to respectively execute a SELECT statement to obtain a table structure list, execute a SHOW COLUMNS statement to obtain a field attribute list, and execute a SHOW KEYS statement to obtain an association relationship list, and assemble the obtained table structure list, field attribute list and association relationship list into unified JSON format data;

[0191] The second unit is used to parse and standardize the JSON format data, extract field names, field types, primary key information and foreign key information, and construct an N×N dimensional association calculation matrix; in the association calculation matrix, the field naming similarity algorithm is used to calculate the similarity score of the field name, the data type compatibility rule is used to calculate the matching score of the field type, and the inter-table association score is calculated based on the foreign key constraint relationship, and the weighted average of the similarity score, the matching score and the inter-table association score is used as the final field mapping association; a standardized data structure description file is generated according to the field mapping association;

[0192] The third unit is used to establish a three-layer index directory based on the data structure description file using a B+ tree index structure, wherein the first layer index records the data source type and its physical storage location, the second layer index records the business attributes and its access path, and the third layer index records the usage scenario and its version information; the three-layer index directory is used to poll the structure changes of the multiple heterogeneous data sources at a frequency of F seconds / times, and when a change in the table structure, field attribute or association relationship is detected, the data structure description file is updated again; according to the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationship between data tables, and a data flow path in the form of a DAG directed acyclic graph is generated.

[0193] According to a third aspect of the embodiments of the present invention,

[0194] An electronic device is provided, comprising:

[0195] processor;

[0196] a memory for storing processor-executable instructions;

[0197] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0198] A fourth aspect of the embodiments of the present invention is:

[0199] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0200] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An integrated metadata management method based on data source extensibility, characterized in that: include: Perform real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collect the connection addresses, access credentials and protocol types of the multiple heterogeneous data sources, establish a dynamic connection pool according to the connection addresses, and set the maximum number of connections to N, the timeout period to T seconds, and the number of retries to M times in the dynamic connection pool; use the connection instances in the dynamic connection pool to respectively execute the SELECT statement to obtain the table structure list, execute the SHOW COLUMNS statement to obtain the field attribute list, and execute the SHOW KEYS statement to obtain the association relationship list, and assemble the obtained table structure list, field attribute list and association relationship list into unified JSON format data; Parse and standardize the JSON format data, extract field names, field types, primary key information and foreign key information, and construct an N×N dimensional association calculation matrix; In the association calculation matrix, the field name similarity score is calculated using the field naming similarity algorithm, the field type matching score is calculated using the data type compatibility rule, and the inter-table association score is calculated based on the foreign key constraint relationship. The weighted average of the similarity score, the matching score and the inter-table association score is used as the final field mapping association degree; a standardized data structure description file is generated according to the field mapping association degree; Based on the data structure description file, a three-layer index directory is established using a B+ tree index structure, wherein the first layer index records the data source type and its physical storage location, the second layer index records the business attributes and its access path, and the third layer index records the usage scenario and its version information; The structural changes of the multiple heterogeneous data sources are detected by polling the three-layer index directory at a frequency of F seconds / times. When changes in table structure, field attributes or association relationships are detected, the data structure description file is updated again; based on the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationship between data tables, and a data flow path in the form of a DAG directed acyclic graph is generated.

2. The method according to claim 1, characterized in that In the association calculation matrix, the field name similarity score is calculated using the field naming similarity algorithm, the field type matching score is calculated using the data type compatibility rule, and the inter-table association score is calculated based on the foreign key constraint relationship. The weighted average of the similarity score, the matching score and the inter-table association score is used as the final field mapping association degree, including: The association degree calculation matrix includes a to-be-mapped field identifier and a score storage unit, and the score storage unit is used to store the calculated scores generated during the field mapping process; Obtaining a field name corresponding to the to-be-mapped field identifier, performing word segmentation preprocessing on the field name to obtain a root list, calculating edit distance values ​​between roots in the root list using a field naming similarity algorithm, normalizing the edit distance values ​​to obtain a similarity score, and writing the similarity score into the score storage unit based on the to-be-mapped field identifier; Acquire field data type information according to the to-be-mapped field identifier, extract compatibility rules and conversion rules matching the field data type information from a preset type mapping rule library, calculate a direct compatibility score based on the compatibility rule, calculate a conversion cost score based on the conversion rule, use a weighted combination value of the direct compatibility score and the conversion cost score as a matching score, and write the matching score into the score storage unit based on the to-be-mapped field identifier; Using the to-be-mapped field identifier to query the foreign key constraint information of the data table to which the field belongs, parsing the primary key field information and the reference constraint information from the foreign key constraint information, constructing an associated field index based on the primary key field information, calculating an associated weight using the reference constraint information, taking the product of the associated field index and the associated weight as an inter-table association score, and writing the inter-table association score into the score storage unit based on the to-be-mapped field identifier; The similarity score, matching score and inter-table association score of the current field to be mapped are read from the score storage unit using the field identifier to be mapped, and the read similarity score, matching score and inter-table association score are weighted averaged to obtain the field mapping association degree.

3. The method according to claim 1, characterized in that Detecting structural changes of the plurality of heterogeneous data sources by polling the three-layer index directory at a frequency of F seconds / times, and when detecting changes in table structure, field attributes or association relationships, re-updating the data structure description file includes: Detecting structural changes of the plurality of heterogeneous data sources by polling the three-layer index directory at a frequency of F seconds / times, the three-layer index directory comprising a data source-level directory, a table-level directory and a field-level directory, establishing a first association relationship between the table-level directory and the data source-level directory, and establishing a second association relationship between the field-level directory and the table-level directory; Reading a data source version mark stored in the data source level directory, and comparing the data source version mark with the real-time version marks of the multiple heterogeneous data sources, wherein the data source version mark is used to identify the structural status of the multiple heterogeneous data sources; When the data source version mark is inconsistent with the real-time version mark, obtaining a changed heterogeneous data source identifier, and obtaining a table structure identifier and a table structure version mark from the table-level directory according to the first association relationship, wherein the table structure version mark is used to identify the state of the table structure; The table structure version mark is compared with the real-time table structure version mark in the changed heterogeneous data source, and when a table structure change is detected, a field attribute identifier and a field attribute version mark are obtained from the field-level directory according to the second association relationship, wherein the field attribute version mark is used to identify the state of the field attribute; The field attribute version tag is compared with the real-time field attribute version tag in the changed table structure. When a change in field attribute or association relationship is detected, the changed field attribute information and association relationship information are obtained; and the data structure description file is re-updated based on the changed field attribute information and the association relationship information.

4. The method according to claim 1, characterized in that: According to the updated data structure description file, the reference dependency relationship between data tables is calculated using a depth-first search algorithm to generate a data flow path in the form of a DAG directed acyclic graph, including: Parsing the updated data structure description file, extracting inter-table association information in the data structure description file, wherein the inter-table association information includes primary and foreign key relationship information, field reference relationship information, and business association relationship information; Constructing an adjacency list structure based on the association information between tables, wherein each node in the adjacency list structure includes a table identifier, table attribute information, reference field information and association type information, and the association type information is used to identify the dependency strength between nodes; Initialize a depth-first search environment, create an access stack structure for recording access paths, create an access mark bitmap for avoiding repeated accesses, and set a maximum recursive depth threshold for controlling the search depth. The access stack structure, the access mark bitmap, and the maximum recursive depth threshold are used to assist in the execution of the depth-first search; Using a depth-first search algorithm to traverse the adjacency list structure, pushing the current access node into the access stack structure and marking the accessed state in the access mark bitmap, obtaining a set of adjacent nodes of the current access node; For each to-be-visited node in the adjacent node set, detecting whether the to-be-visited node already exists in the access stack structure, and when the detection result is yes, extracting the node information in the access stack structure to generate a ring dependency path; Analyze the association type information between the nodes in the ring dependency path, and when there is a strong dependency relationship between the nodes, add a strong dependency mark to the corresponding edge in the adjacency list structure; when there is a non-strong dependency relationship between the nodes, delete the edge forming the loop from the adjacency list structure to obtain an optimized adjacency list structure; Converting the optimized adjacency list structure into a directed acyclic graph, layering the directed acyclic graph based on the dependency depth of the nodes, assigning nodes with the same dependency depth to the same level, and obtaining a hierarchical directed acyclic graph; A transitive closure analysis is performed on the hierarchical directed acyclic graph to identify and remove redundant dependency edges to obtain a simplified directed acyclic graph; based on the simplified directed acyclic graph, a topological sorting algorithm is used to generate a data flow path.

5. The method according to claim 4, characterized in that Creating an access mark bitmap to avoid repeated access, setting a maximum recursive depth threshold to control the search depth, the access stack structure, the access mark bitmap and the maximum recursive depth threshold to assist the execution of depth-first search include: Based on the number of nodes to be recorded in the access stack structure, an access mark bitmap is created to avoid repeated access, wherein the access mark bitmap uses two binary bits to represent the access status of the node, wherein the first bit represents whether the node is accessed, and the second bit represents whether the node is on the current access path; According to the node depth information recorded in the access stack structure and the estimated maximum dependency chain length, a maximum recursive depth threshold is set to control the search depth, and the maximum recursive depth threshold is used to limit the maximum search depth of the depth-first search process; When a new node is accessed, the node information is pushed into the access stack structure, and the first bit of the node is marked as a visited state in the access mark bitmap, and the second bit is marked as a state on the current path; During the node access process, the access timestamp information in the access stack structure is continuously recorded and updated, and the existing circular dependency is identified by comparing the access timestamps of adjacent nodes; When the node access depth reaches the maximum recursive depth threshold, the current access path information is recorded, and the depth critical point is marked in the access mark bitmap, and then backtracking to the previous node in the access stack structure; After the node access is completed, the node information is popped from the access stack structure, and the access status of the node is updated in the access mark bitmap, and the second bit is reset to a state of not being on the current path; A depth control mechanism provided based on the complete access path recorded by the access stack structure, the access mark bitmap and the maximum recursive depth threshold is used to assist the execution of depth-first search.

6. The method according to claim 4, characterized in that Performing transitive closure analysis on the hierarchical directed acyclic graph to identify and remove redundant dependency edges to obtain a streamlined directed acyclic graph; Based on the simplified directed acyclic graph, a topological sorting algorithm is used to generate a data flow path, including: Constructing an adjacency matrix to represent the dependency relationship between nodes in the hierarchical directed acyclic graph, wherein the row and column identifiers of the adjacency matrix correspond to the node identifiers, and the element values ​​of the adjacency matrix represent whether there is a direct dependency between the nodes; Performing matrix iteration operation based on the adjacency matrix to obtain a transitive closure matrix, wherein the transitive closure matrix is ​​used to represent the transitive dependency relationship between nodes, and the element value of the transitive closure matrix represents whether there is a transitive path between corresponding nodes; Traversing the direct dependency edges in the adjacency matrix, detecting each direct dependency edge based on the transitive closure matrix, and marking the direct dependency edge as a redundant dependency edge when there is a transitive path between node pairs via an intermediate node; The redundant dependency edges are removed to obtain a simplified directed acyclic graph, the in-degree value of each node is calculated based on the simplified directed acyclic graph, and the nodes with zero in-degree values ​​are added to a zero-in-degree node queue; Taking a node from the zero-in-degree node queue and adding it to the access order list, updating the in-degree values ​​of all successor nodes pointed to by the node, and adding the successor node to the zero-in-degree node queue when the in-degree value of the successor node is updated to zero; The operation of taking out nodes from the zero-in-degree node queue is repeatedly performed until the zero-in-degree node queue is empty, thereby obtaining a final access sequence list, which is the data flow path.

7. An integrated metadata management system that is scalable based on data sources, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to perform real-time detection on multiple heterogeneous data sources through the PF4J plug-in framework, collect connection addresses, access credentials and protocol types of the multiple heterogeneous data sources, establish a dynamic connection pool according to the connection addresses, and set the maximum number of connections to N, the timeout period to T seconds, and the number of retries to M times in the dynamic connection pool; use the connection instances in the dynamic connection pool to respectively execute a SELECT statement to obtain a table structure list, execute a SHOW COLUMNS statement to obtain a field attribute list, and execute a SHOW KEYS statement to obtain an association relationship list, and assemble the obtained table structure list, field attribute list and association relationship list into unified JSON format data; The second unit is used to parse and standardize the JSON format data, extract field names, field types, primary key information and foreign key information, and construct an N×N dimensional association calculation matrix; in the association calculation matrix, the field naming similarity algorithm is used to calculate the similarity score of the field name, the data type compatibility rule is used to calculate the matching score of the field type, and the inter-table association score is calculated based on the foreign key constraint relationship, and the weighted average of the similarity score, the matching score and the inter-table association score is used as the final field mapping association; a standardized data structure description file is generated according to the field mapping association; The third unit is used to establish a three-layer index directory based on the data structure description file using a B+ tree index structure, wherein the first layer index records the data source type and its physical storage location, the second layer index records the business attributes and its access path, and the third layer index records the usage scenario and its version information; The structural changes of the multiple heterogeneous data sources are detected by polling the three-layer index directory at a frequency of F seconds / times. When changes in table structure, field attributes or association relationships are detected, the data structure description file is updated again; based on the updated data structure description file, a depth-first search algorithm is used to calculate the reference dependency relationship between data tables, and a data flow path in the form of a DAG directed acyclic graph is generated.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images