Data dependency relationship confirmation method and device, equipment and medium

Abstract

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a data dependency relationship confirmation method, which comprises the following steps of: importing metadata information containing a field name, a data table name and data source information into a data processing module to obtain a target field name, and retrieving a target field in the metadata information by utilizing the data processing module to confirm a data dependency relationship. Determining an upstream data source and a downstream data destination of the target field; generating upstream and downstream data link diagrams of the target field, wherein the upstream and downstream data link diagrams are used for identifying a flow relation of data, a data generator and processing logic; and analyzing a dependency relationship between the target field and the upstream and downstream fields based on the link diagram, and further determining a data table on which the target field depends and owner information of the data table. According to the method, through automatic analysis and link diagram generation, the original complex process needing manual query and processing is simplified, the data dependency relationship is clearer, the communication cost and query time of developers are reduced, and the data use accuracy and processing efficiency are effectively improved.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and financial technology, and in particular to a data dependency confirmation method, device, equipment and storage medium. Background Art

[0002] As financial institutions continue to expand their businesses, banks have accumulated a large amount of data assets. These data assets cover customer information, transaction records, financial data, risk management and other multi-dimensional content. In order to achieve effective data management, banks usually assign different data tables or data sets to different teams or owners for management according to business needs.

[0003] However, data tables within banks often have complex upstream and downstream dependencies. The generation of a certain data field may involve the processing and circulation of multiple upstream data tables. When a field of a certain table needs to be used, it is usually necessary to trace the structure of many upstream tables and understand the data source of each table and its relationship with other tables. This results in developers having to spend a lot of time and energy to understand and sort out these data dependency links during data use.

[0004] Data assets in banks are managed by multiple teams, and data owners are divided according to different business departments. Due to the different management responsibilities of data owners, data modification, maintenance, and updates may lack unified coordination. When developers from a team need to call tables or fields managed by other teams across departments, multi-party communication and coordination are often required. This complex team collaboration process is not only time-consuming, but may also lead to problems such as poor communication and information lag, which in turn affects the efficiency and accuracy of data use.

[0005] In the process of using data assets, especially when it comes to complex data dependencies, developers often need to manually query the structure of multiple data tables to trace the source and destination of the data. Existing technical means usually rely on the experience of developers or manual operations, and lack intelligent tools to automatically generate data dependency link diagrams or perform dependency analysis. Such a cumbersome process not only increases the difficulty and workload of development, but is also prone to human omissions, affecting the accuracy of data query and use. Summary of the invention

[0006] The main purpose of the present invention is to provide a data dependency confirmation method, device, equipment and storage medium, aiming to solve the technical problem that the prior art cannot effectively sort out complex data dependency relationships, resulting in the need to manually query the structure and data source of multiple tables when using data, which is inefficient and prone to data errors.

[0007] To achieve the above object, the present invention provides a method for confirming data dependency, comprising:

[0008] Importing metadata information of data resources into the data processing module, the metadata information includes field names, data table names and data source information;

[0009] Obtaining a target field name, and based on the data processing module, retrieving the target field name in the metadata information, and determining an upstream data source and a downstream data destination of the target field;

[0010] Based on the upstream data source and the downstream data destination, an upstream and downstream data link diagram of the target field is generated, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field;

[0011] Based on the upstream and downstream data link diagram, analyzing and determining the dependency link between the target field and the corresponding upstream and downstream fields;

[0012] Based on the dependency link, the data table and corresponding owner information on which the target field depends are determined.

[0013] Furthermore, to achieve the above object, the present invention provides a data dependency confirmation device, comprising:

[0014] A metadata import module, used to import metadata information of data resources into a data processing module, wherein the metadata information includes field names, data table names and data source information;

[0015] A field search module, used to obtain a target field name, retrieve the target field name in the metadata information based on the data processing module, and determine the upstream data source and downstream data destination of the target field;

[0016] A data link diagram generation module, used to generate an upstream and downstream data link diagram of the target field based on the upstream data source and the downstream data destination, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field;

[0017] A dependency link analysis module, used to analyze and determine the dependency link between the target field and the corresponding upstream and downstream fields based on the upstream and downstream data link diagram;

[0018] The dependency confirmation module is used to determine the data table and corresponding owner information that the target field depends on based on the dependency link.

[0019] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data dependency confirmation program stored in the memory and executable on the processor, and the data dependency confirmation program, when executed by the processor, implements the steps of the data dependency confirmation method as described above.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data dependency confirmation program is stored, and when the data dependency confirmation program is executed by a processor, the steps of the data dependency confirmation method as described above are implemented.

[0021] Beneficial effects: The present invention relates to the fields of big data technology and financial technology, and discloses a method for confirming data dependency relationships, which imports metadata information containing field names, data table names, and data source information into a data processing module, obtains the target field name, and uses the data processing module to retrieve the target field in the metadata information to determine the upstream data source and downstream data destination of the target field; generates an upstream and downstream data link diagram of the target field to identify the data flow relationship, data generator, and processing logic; analyzes the dependency relationship between the target field and the upstream and downstream fields based on the link diagram, and further determines the data table and its owner information on which the target field depends. The present invention simplifies the complex process that originally required manual query and processing through automated analysis and link diagram generation, makes data dependency relationships clearer, reduces communication costs and query time for developers, and effectively improves the accuracy of data use and processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0023] Figure 1 A schematic diagram of an application environment of a method for confirming data dependency in an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for confirming data dependency of the present invention;

[0025] Figure 3 A schematic diagram of functional modules of a preferred embodiment of the data dependency confirmation device of the present invention;

[0026] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0027] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0028] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0029] The data dependency confirmation method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user end communicates with the server end through a network. The server end can import metadata information containing field names, data table names and data source information into a data processing module through the user end, obtain the target field name, use the data processing module to retrieve the target field in the metadata information, and determine the upstream data source and downstream data destination of the target field; generate an upstream and downstream data link diagram of the target field to identify the flow relationship, data generator and processing logic of the data; analyze the dependency relationship between the target field and the upstream and downstream fields based on the link diagram, and further determine the data table and its owner information on which the target field depends. The present invention simplifies the complex process that originally required manual query and processing through automated analysis and link diagram generation, makes the data dependency relationship clearer, reduces the communication cost and query time of developers, and effectively improves the accuracy of data use and processing efficiency. Among them, the user end can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0030] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for confirming data dependency provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0031] like Figure 2 As shown, the data dependency confirmation method proposed by the present invention includes the following steps:

[0032] S10, importing metadata information of data resources into a data processing module, wherein the metadata information includes field names, data table names, and data source information;

[0033] In this embodiment, metadata information refers to a structured description of data, which is used to explain the information of the data itself. Metadata information can come from different data sources, including internal databases, external data suppliers, or third-party APIs. These metadata include but are not limited to:

[0034] Field name: A unique identifier for each data field, usually used to describe the content or meaning of each data unit in a data table.

[0035] Data table name: The name of the data structure that stores multiple fields, usually a logical unit in the database.

[0036] Data source information: The source of the data, such as database type, storage location, data interface, etc., helps track the generation and use path of data.

[0037] Develop a data collection module to batch extract metadata information from multiple data sources through API, SQL query or file system. Use custom scripts or middleware interfaces to collect structured and unstructured metadata information from internal and external systems, ensuring a unified format and adaptability to multiple sources.

[0038] The collected metadata information usually comes from multiple heterogeneous systems, which may use different field naming and data types. Therefore, metadata needs to be standardized to ensure its unified structure and format for subsequent processing and storage.

[0039] Use data conversion tools (such as ETL: Extract, Transform, Load) to format metadata information. ETL tools can automatically convert metadata information from different data sources into a consistent format (such as unified field naming, data type standardization), and convert it into a data format suitable for the data processing module.

[0040] During the data collection and standardization process, duplicate or erroneous data may appear. In order to ensure the integrity and accuracy of metadata information, data cleaning operations are required, including removing duplicate data, filling missing values, or correcting erroneous data.

[0041] Implement data cleansing rules and use automated scripts and tools to detect and deduplicate metadata. For example, use Python scripts or database triggers to detect duplicate field names or data table names and apply deduplication algorithms to filter out redundant data.

[0042] The formatted and cleaned metadata information needs to be imported into the data processing module, which is the core unit for storing and processing metadata information and provides subsequent data dependency analysis and management functions.

[0043] Data import can be achieved through API interfaces or database batch import tools. For example, use the bulk loading function of the database management system (such as MySQL's LOAD DATA command or MongoDB's bulkWrite function) to store the processed metadata in the database. During the import process, a unique identifier needs to be assigned to each metadata record for subsequent query and indexing.

[0044] Example: In the financial sector, banks’ data assets are very complex, and metadata includes information from multiple dimensions, such as customer information, transaction records, and risk assessments. Suppose a bank needs to manage a database containing millions of customer data, where each field has a specific definition, such as “customer ID,” “transaction amount,” and “transaction date.” Banks can automatically collect metadata information from multiple internal and external data sources and perform unified formatting. After data cleaning and deduplication, the data is imported into the bank’s data processing module to generate a metadata set for management and analysis. With these metadata, banks can quickly understand the definition, source, and dependencies of data fields, reduce the time for multi-party coordination and communication, and improve the efficiency and accuracy of data management.

[0045] By importing metadata information of data resources into the data processing module, the consistency and accuracy of data during collection and import can be ensured. By uniformly importing field names, data table names, and data source information, the complexity of data management is simplified and the tedious steps of manual entry are avoided. Systematic data import can reduce errors caused by human factors and improve the standardization of data.

[0046] S20, obtaining a target field name, searching the target field name in the metadata information based on the data processing module, and determining an upstream data source and a downstream data destination of the target field;

[0047] In this embodiment, the target field name is a field specified by a user or system operation. It is used to track the upstream source and downstream destination of data and serves as the starting point for data dependency analysis. The target field name can be obtained by user input, system preset, or external interface transmission.

[0048] Develop a user input interface to receive the target field name input by the user through a graphical user interface (GUI) or API interface. In addition, the system can automatically select the target field through preset conditions or rules, or import the specified field name from an external system.

[0049] The retrieval operation is to locate the target field name in the stored metadata information. The metadata information contains information such as field name, table structure and data source. The retrieval process can be implemented through indexing or full-text search mechanism to ensure fast search for metadata related to the target field.

[0050] Use database indexing techniques, such as B-tree or hash indexing, to quickly retrieve the target field name in the metadata information. Database query statements can be implemented through SQL or NoSQL commands, such as SELECT statements for relational databases or find statements for document databases (such as MongoDB) to achieve efficient field name retrieval.

[0051] Upstream data sources refer to the data tables or other data sources that generate the target fields. By recording metadata information, the system can track the upstream data tables of the target fields and understand which data sources generate the fields.

[0052] Automatically locate the upstream source of the target field through the data dependency record in the data processing module. The dependency link determines its direct upstream table or data source by parsing the historical processing path and data table structure of the field. The implementation method may include using a graphical interface to display the upstream source, or automatically generating a tracking path for the upstream data table.

[0053] Downstream data destination refers to the subsequent data table or application system where the target field is used or processed. Determining the downstream destination helps understand how the field is further processed and referenced in the system.

[0054] Use data dependency tables or query logs to track the usage of target fields and display their reference paths in different data tables or applications. Use automated tools to analyze data flow chains to help users understand the flow of target fields in the entire data system.

[0055] Example: In a banking system, a customer's "transaction amount" field involves the processing and aggregation of multiple transaction tables. By entering the field name, the system can quickly retrieve the source table of the transaction amount and track how the field is processed to the downstream table. Banks can use this function to quickly determine the dependencies of data fields, reduce the time of multi-party communication, and improve the accuracy of data query. For example, when a problem occurs in the "transaction amount" field during data cleaning, the bank can immediately track the upstream table and downstream system of the field and quickly fix the data problem.

[0056] By automatically retrieving the target field name and quickly locating its upstream source and downstream destination, the efficiency of data dependency analysis is improved. Developers do not need to manually query multiple data tables. The system can intelligently generate dependency links for target fields, reducing human errors and improving the transparency and accuracy of data flows.

[0057] S30, based on the upstream data source and the downstream data destination, generating an upstream and downstream data link diagram of the target field, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field;

[0058] In this embodiment, the upstream data source and the downstream data destination respectively represent the input and output paths of the target field in the data flow. By tracing the dependencies between data tables, the upstream data tables from which the target field is obtained are analyzed, and its usage in the downstream data tables is determined.

[0059] Through the dependency link information in the data processing module, the system automatically parses the upstream and downstream data tables related to the target field. Using the database query log or metadata dependency table, the automation tool can analyze the dependency path between database tables and determine the upstream and downstream field associations. The query log can include the source and destination of the target field to build a clear dependency chain.

[0060] The upstream and downstream data link diagram is a visual diagram used to show the dependency links of the target field. This diagram can intuitively show the upstream data source of the target field (such as the input table or field) and the downstream data destination (such as how the target field is referenced or processed in other tables).

[0061] Using graphical modules (such as D3.js, Graphviz, etc.), the system can dynamically generate a link diagram to show the relationship between upstream and downstream data tables and target fields. The nodes in the diagram can represent data tables, and the lines show the flow path of the data. For each node, field information can also be attached, such as the field name, data generator, and data processing steps. The data link diagram can be displayed through a graphical user interface (GUI) and allow users to interactively click to view detailed information.

[0062] The data flow relationship describes how the target field is transferred, converted, and processed between different data tables or systems. It shows the entire life cycle of the target field, from upstream data generation to downstream data consumption or further processing. Relying on the data flow record in the data processing module, all associated tables of the target field are identified, and clear flow paths are marked for the connection lines of the link diagram. The flow relationship can be detected by automated analysis tools, and the key flow nodes are highlighted in the data link diagram to facilitate users to identify key links.

[0063] The data generator refers to the system or team responsible for generating the target field or data table. By identifying the data generator, you can clearly understand the source of the target field and the relevant business department or development team. Extract the owner information of each table or field from the metadata information and associate it with the data table. The system can attach an identifier to each data generator in the upstream and downstream data link diagram to help users clearly understand which team or system generated the data.

[0064] Processing logic refers to the processing process that the target field undergoes in the upstream or downstream data table. It includes operations such as data conversion, aggregation, and calculation. By identifying the processing logic, users can understand the change process of the target field in the data flow.

[0065] The system automatically analyzes the dependency logic between data tables and extracts the processing logic of fields in upstream or downstream data tables. In the data link diagram, different icons or connection styles are used to identify different types of processing logic (such as data aggregation, conversion, calculation, etc.) to provide users with clear visual information.

[0066] Example description: In the financial field, suppose a bank needs to track the flow path of a customer transaction field and understand how the field is used in multiple systems. By generating an upstream and downstream data link diagram for the field, the system can display the upstream data source (such as an internal transaction database) and downstream destination (such as a risk assessment system) of the customer transaction amount field. The link diagram will identify the processing logic of the field in each system, such as whether data aggregation, conversion, or other calculations have been performed. In this way, banks can discover problems in the data processing chain more quickly, reduce communication costs in cross-departmental collaboration, and improve the efficiency of system problem locating.

[0067] By generating an upstream and downstream data link diagram of the target field, the data flow relationship of the target field in the system can be clearly displayed, significantly reducing the time developers spend manually tracking and analyzing dependencies. The link diagram intuitively displays the source, destination, generator and processing logic of the data, improves the transparency of data dependencies, and facilitates cross-team collaboration and rapid location of data problems.

[0068] S40, analyzing and determining the dependency link between the target field and the corresponding upstream and downstream fields based on the upstream and downstream data link diagram;

[0069] In this embodiment, the upstream and downstream data link diagram shows the flow path of the target field between data tables. The analysis of the dependency link is to identify and calculate the direct or indirect dependency relationship between the target field and the upstream and downstream fields through the link diagram. This helps to understand how the target field depends on other data tables or fields, and ensure the integrity and accuracy of the data.

[0070] Using graph-theory-based algorithms (such as depth-first search DFS or breadth-first search BFS), the nodes and edges in the data link graph are analyzed to determine the dependency path between the target field and the upstream and downstream fields. The system reads each node (data table) and the edges (field dependencies) between them from the link graph, and recursively analyzes the hierarchical structure of field dependencies.

[0071] Dependencies reflect the dependency hierarchy between the target field and other fields. Upstream fields provide data support to the target field, while downstream fields further reference the data results of the target field. By determining these dependencies, you can understand the processing and reference path of the target field.

[0072] Through the edges in the dependency link, the system marks all upstream fields that the target field depends on and the downstream fields that it affects. The implementation method may include an automated dependency path analysis tool that highlights the fields that have direct dependencies on the target field in the link diagram and lists the direction and level of the dependency.

[0073] The dependency paths between different fields may involve multiple processing types, such as aggregation, transformation, calculation, etc. Analyzing the types of dependency paths helps us understand how data is passed and processed between various fields.

[0074] Using the rule base or data log, the system can identify the processing type in each dependency path, such as aggregation (SUM, COUNT, etc.), conversion (field format conversion), or calculation (formula calculation). Through the visualization of the link diagram, the dependency path type can be displayed in different colors or annotations to facilitate users to identify different types of field dependencies.

[0075] Ensuring that the dependency links between the target field and the upstream and downstream fields are complete is one of the key steps. If there are omissions or errors in the dependency links, it may lead to data inconsistency or erroneous results. Therefore, the integrity analysis of the dependency links includes checking whether the dependency relationship between the fields is closed and whether there are incomplete dependency paths.

[0076] The system automatically scans the upstream and downstream links to ensure that the dependency relationship between the target field and its upstream and downstream fields is not interrupted. If the dependency chain is found to be incomplete, the system generates a warning and provides repair suggestions. The dependency database and metadata log can be used to detect whether there are missing nodes or incorrect connections in the dependency chain.

[0077] Key nodes refer to fields or tables that have a significant impact on the target field in the dependency chain. Extracting these nodes can help users identify which fields have the most significant effect on the final result of the target field. Based on weight analysis or field reference frequency, the system extracts key fields from the dependency chain and highlights them in the chain diagram to help users quickly identify key nodes in the dependency chain. Key node extraction can use the key path identification technology in the graph analysis algorithm to highlight the core influencing factors of the target field dependency chain.

[0078] Example: In the financial system, the transaction data of banks depends on multiple data tables. Through the dependency chain diagram, the bank can analyze the dependency relationship between a certain "transaction amount" field and its upstream fields (such as transaction record table, customer information table). At the same time, the system will show the destination of the field in the downstream table (such as risk control system, report generation system). Through automated dependency chain analysis, banks can quickly identify the complete path of field dependencies, ensure accurate data transmission, and discover key fields in the dependency chain, such as how the "customer ID" field affects transaction data.

[0079] By analyzing the dependency links between the target field and the upstream and downstream fields, the system can accurately identify and display the dependency relationship of the fields, helping users quickly understand the data flow path. This improves the efficiency of data dependency analysis, reduces the complexity of manual analysis of dependency paths, and ensures the integrity and accuracy of dependency links. In addition, the extraction of key nodes in dependency links enables data managers to quickly locate the fields that have the most significant impact on data results, improving the transparency of data management.

[0080] S50: Based on the dependency link, determine the data table and corresponding owner information that the target field depends on.

[0081] In this embodiment, the dependency link shows the association relationship between the target field and the upstream and downstream data tables. By analyzing the dependency link, all data tables that the target field depends on can be identified. The dependency relationship between each data table and other tables is clearly shown by the link diagram, which helps to identify the source and destination of the target field data.

[0082] Use the data processing module to parse the dependency link and determine which data tables provide data for the target field from the upstream and downstream link graph. Dependency link parsing can be achieved through a graph analysis algorithm, based on nodes (data tables) and edges (dependencies), to identify all related data tables in the dependency path.

[0083] Data tables are the core structures for storing field data, and all data of the target field depends on these data tables. By identifying the data tables in the dependency link, you can accurately locate the association between the target field and these tables. This helps understand how the target field obtains data and trace the flow path of data from the source to the target field.

[0084] The system lists all upstream and downstream data tables that provide data for the target field based on the link graph generated by the dependency link. The implementation method may include reading the association information of fields and tables from the dependency log or database dependency table, and automatically marking each data table that the target field depends on.

[0085] The owner of a data table is the team or individual responsible for maintaining and managing the table. Extracting owner information can help users understand the responsible party for the data table, facilitating collaboration and troubleshooting. Owner information usually includes team name, individual name, contact information, etc.

[0086] The system reads the owner information of each data table from the metadata information and associates this information with the data table in the dependency link. The extraction of owner information can be achieved through database query or records in the data management system to ensure that each table has a clear owner identification.

[0087] After determining all the data tables that the target field depends on, you need to associate the owner information of each table with the target field. This can help users quickly find the team or individual responsible for maintaining the relevant tables, thereby efficiently solving data-related problems.

[0088] The system associates each data table in the dependency chain with its owner information, and generates a report containing the target field's dependent data tables and their owners. This can be achieved by mapping the owner information with the data table fields through metadata management tools or database records, ultimately presenting a complete dependency chain and owner information.

[0089] After completing dependency link resolution and owner information extraction, the system needs to output a complete report showing the data tables that the target field depends on and the corresponding owner information. This report can be used as a reference for developers, data managers, or other relevant parties.

[0090] The system automatically generates a document or visual report containing all data tables that the target field depends on and their corresponding owners. The output report can be in PDF, HTML page or other data report format, and can be exported for subsequent review.

[0091] Example: In the bank's risk control system, the customer's risk assessment field depends on customer information, transaction records, and historical default data in multiple tables. Through the dependency chain diagram generated by the system, the bank can quickly identify all the data tables that the customer risk assessment field depends on. At the same time, the system will also extract the owner information of each table. For example, the "transaction record table" is maintained by the transaction department, and the "historical default data table" is managed by the risk control team. The display of owner information helps banks quickly locate problems and optimize data maintenance and cross-departmental collaboration.

[0092] By determining the data table that the target field depends on and the corresponding owner information based on the dependency link, the system can help users quickly identify the relationship between the target field and its dependent data table, and clarify the person responsible for the relevant data table. This automated dependency link analysis reduces the complexity of manual query and analysis and improves data management efficiency. In addition, outputting reports containing owner information makes cross-team collaboration and troubleshooting faster, and improves the transparency and accuracy of data maintenance.

[0093] The present invention relates to the fields of big data technology and financial technology, and discloses a method for confirming data dependency relationships, which includes importing metadata information containing field names, data table names, and data source information into a data processing module, obtaining target field names, and using the data processing module to retrieve target fields in the metadata information, and determining the upstream data source and downstream data destination of the target field; generating an upstream and downstream data link diagram of the target field, which is used to identify the data flow relationship, data generator, and processing logic; analyzing the dependency relationship between the target field and the upstream and downstream fields based on the link diagram, and further determining the data table and its owner information on which the target field depends. The present invention simplifies the complex process that originally required manual query and processing through automated analysis and link diagram generation, making data dependency relationships clearer, reducing the communication cost and query time of developers, and effectively improving the accuracy of data use and processing efficiency.

[0094] In one embodiment, the above S10 includes:

[0095] S101, collecting metadata information including field names, data table names, and data source information from multiple data resources, and assigning a unique identifier to each data table name;

[0096] S102, performing format standardization processing on the metadata information;

[0097] S103, performing data cleaning operations on the metadata information after the standardization process, including removing duplicate information or erroneous information;

[0098] S104: Import the processed metadata information into the database of the data processing module.

[0099] In this embodiment, metadata information is descriptive information about data, usually including field names, data table names, and data source information. The process of collecting metadata is to extract this information from multiple different systems or databases (data resources). In order to ensure the uniqueness of metadata information and facilitate subsequent retrieval, each data table name needs to be assigned a unique identifier (such as ID).

[0100] Collect metadata information from multiple data sources (such as SQL databases, NoSQL databases, file systems, etc.) through APIs or database connections. The system automatically traverses different database tables and structured data resources. Generate a unique identifier (UUID or auto-increment ID) for each data table name to ensure that data tables are not repeated in different data sources. The identifier can be stored in the database index for subsequent fast lookup.

[0101] Different data resources may use different formats to store metadata information. The purpose of standardization is to unify field names, data types, and data table structures to ensure that all metadata information can be operated according to the same rules in subsequent processing.

[0102] Use standard ETL tools (such as Apache Nifi, Talend, etc.) to convert the format of metadata information. For example, convert field names in different databases to a unified naming rule to ensure that the data table structure conforms to a unified standard (such as unifying to lowercase letters, deleting special characters, etc.). Define metadata mapping rules to ensure that field names and data table formats are unified in all data sources. For example, convert the "Customer Name" field from multiple sources to "customer_name".

[0103] Data cleaning is to remove redundant, duplicate and erroneous information from metadata to ensure the integrity and accuracy of metadata information. This step is crucial to ensure data quality.

[0104] Use algorithms or database query tools to find duplicate field names or table names in metadata. Redundant records can be detected by comparing unique identifiers or metadata content.

[0105] Automatically identify anomalies in metadata information (such as missing data, incorrect field type, etc.) and correct them according to the rules. For example, by setting data validation rules, automatically detect whether the field types in the data table meet expectations and automatically correct them when errors are detected.

[0106] After the metadata information is standardized and cleaned, it is imported into the database of the data processing module as the basis for subsequent data processing and dependency analysis. The database is specifically used to store the processed metadata information to ensure that this information can be quickly retrieved and analyzed.

[0107] Import the cleaned metadata into the database through SQL INSERT statements, bulk loading tools or APIs. Support relational databases (such as MySQL, PostgreSQL) or NoSQL databases (such as MongoDB, Cassandra). Create indexes for metadata information to speed up subsequent query operations. You can use B-tree indexes or hash indexes to optimize query performance and ensure that users can quickly retrieve and analyze metadata information.

[0108] This embodiment ensures the consistency and accuracy of data by collecting, standardizing and cleaning metadata information, and reduces duplicate and erroneous metadata information. The system automatically processes metadata, greatly reducing the possibility of manual intervention and improving the efficiency of data processing. The cleaned metadata information provides a high-quality data foundation for subsequent data analysis and dependency analysis, ensuring the integrity of the data dependency chain.

[0109] In one embodiment, the above S20 includes:

[0110] S201, receiving a target field name input by a user through a user interface;

[0111] S202, performing a search operation on the target field name in the metadata information by the data processing module to locate the data table where the target field is located;

[0112] S203, analyzing the associated table structure of the data table where the target field is located, tracing back and determining the upstream data source and downstream data destination of the target field;

[0113] S204, recording the information of the upstream data source and the downstream data destination in the data processing module.

[0114] In this embodiment, the system allows the user to enter the target field name through the interface, and the field name is used to identify the data field that the user needs to query. This step is the starting point of the entire data dependency analysis, ensuring that the system can accurately receive and identify the target field that needs to be tracked.

[0115] Use front-end technologies (such as HTML, JavaScript, React, etc.) to design a user-friendly input form that allows users to enter the name of the target field. The system interface can be a web-based dashboard or desktop application. After the user enters the field name, the system verifies the input to ensure that the field name exists in the database. If it does not exist, the system will give an error prompt.

[0116] Once the user enters the target field name, the data processing module will perform a search operation in the stored metadata information to find the data table to which the field belongs. The purpose of this step is to determine in which table the target field is stored, which facilitates subsequent dependency analysis.

[0117] Use SQL query or NoSQL command (such as SELECT or find operation) to search the target field name entered by the user in the metadata management system and locate the data table to which the field belongs. To speed up the search, you can use B-tree or hash index in the metadata table to ensure fast response when searching for field names. The search result will return the name of the data table where the target field is located and other related information (such as data table ID).

[0118] By analyzing the table structure of the data table where the target field is located and its association with other data tables, you can trace the upstream source (where the data of the field comes from) and downstream destination (how the data of the field is further used) of the field in the system. By analyzing the dependencies of the data, you can understand how the target field flows in multiple data tables.

[0119] Through the data processing module, the system can read the structure of the data table where the target field is located and analyze the foreign key constraints, dependencies or JOIN relationships between tables. This can be achieved by querying the data dictionary, metadata log or database constraint table. The system uses data flow analysis tools to track the data flow of the target field and determine the upstream source and downstream data table. In terms of implementation, the dependency path of the target field can be obtained by parsing the data flow log or querying the dependency table.

[0120] After the system identifies and tracks the upstream and downstream data tables of the target field, it needs to store this dependency information in the data processing module. This step helps with subsequent query, analysis, and display, providing users with complete field dependency information.

[0121] Through automated tools, the system records the upstream data source and downstream data destination of the target field in a dependency table or dependency log for subsequent retrieval. The system can use a relational database (such as MySQL, PostgreSQL) or a NoSQL database (such as MongoDB) to store dependency information and ensure that the upstream and downstream relationships between the field and the relevant data table are recorded for subsequent query and analysis.

[0122] Example description: In a bank's risk management system, the user wants to track the upstream source and downstream destination of the "Customer Risk Score" field. By entering the field name, the system automatically retrieves the "Risk Assessment Table" to which it belongs in the metadata management module and identifies the dependency relationship between the table and other tables. Through table structure analysis and data flow tracing, the system determines that the upstream data of the field comes from the "Customer Transaction History Table", while its downstream data is used by the "Credit Approval Table". Finally, the system records this dependency information in the data processing module for risk management personnel to analyze the use of the field.

[0123] In this embodiment, the target field name is input through the user interface, and the system can quickly locate the data table to which the field belongs and trace its upstream source and downstream destination, significantly improving the efficiency of data dependency analysis. Automated dependency tracking reduces the complexity of manual query and analysis, while ensuring the integrity and accuracy of dependency information and improving the transparency of data flow management. Recording dependency links makes subsequent analysis more efficient, and the time to locate and solve data problems is significantly reduced.

[0124] In one embodiment, in the above S30, based on the upstream data source and the downstream data destination, generating an upstream and downstream data link diagram of the target field includes:

[0125] S301, based on the upstream data source and downstream data destination of the target field, generate an upstream and downstream data link diagram of the target field using a graphical module;

[0126] S302, marking the name of each data table and field information associated with the target field in the data table in the upstream and downstream data link diagram;

[0127] S303, marking the data generator of each data table in the upstream and downstream data link diagram;

[0128] S304, describing the processing logic of each data flow process in the upstream and downstream data link diagram, wherein the processing logic includes data conversion, calculation or processing process;

[0129] S305, storing the upstream and downstream data link diagram in the data processing module.

[0130] In this embodiment, the data link diagram is a visual chart that shows the upstream and downstream dependencies of the target field in the data flow. It can clearly describe the path for the field to obtain data from the upstream data table and the downstream path for the data to be used and processed. The process of generating a link diagram requires the system to extract information from metadata and dependencies, and present the data flow path in a visual way through a graphical module.

[0131] Use the graphical library to generate a visual link diagram. The system automatically draws the upstream and downstream data flows of the target field based on the dependency chain parsed from the database. Nodes represent data tables, and edges represent dependency relationships. The analysis function in the dependency data processing module automatically extracts the upstream source and downstream destination of the target field, and the graphical module generates the connection path between fields.

[0132] The link diagram should not only show the name of the data table, but also clearly indicate the fields in each table that are directly related to the target field. This allows users to clearly see how the target field is related to other fields in a specific data table.

[0133] The specific field information (such as field name, data type, field ID, etc.) associated with the target field is marked on each node (data table) in the link diagram. This information is obtained through the metadata and dependency tables in the database. In the graphical module, each node is attached with the field information related to it, and users can click to view the detailed content of the fields in each table.

[0134] Data producers refer to the team, department or system responsible for generating or managing the data table. By marking the data producers, users can quickly identify the responsible party for each data table so that they can find the relevant person in charge when maintaining data and troubleshooting problems.

[0135] Relying on metadata information, extract the information of the generator of each data table (such as the owner department, development team, etc.), and mark the information next to the corresponding node in the link diagram. Associate the data table with the owner information, and the generator mark can be displayed as a team name, personal name or system name, so that users can see the responsible party at a glance.

[0136] The processing logic describes the processing process of data when it is transferred between upstream and downstream tables. This includes data conversion, aggregation, calculation, etc. By displaying the processing logic in the link diagram, users can understand how the data is processed and changed during the flow.

[0137] The system extracts the data processing logic from the data processing log or dependency table and displays the relevant processing operations (such as data cleaning, field calculation, data conversion, etc.) on the edge of the link diagram. Different symbols or colors can be used to identify different types of processing processes. On each data flow path, the processing logic describing the data conversion is attached to ensure that users can understand how the data flow is processed.

[0138] The generated link graph can be used as a visual representation of the dependent links and needs to be stored in the data processing module so that users can directly access the generated link graph during subsequent queries or analysis. This step ensures that the link graph is not only generated in real time, but also can be stored for a long time for subsequent use.

[0139] The generated link graph is stored in the database or file system of the data processing module in a graphic file format (such as SVG, PNG, JSON, etc.). Users can view, download or edit the link graph through the system interface. The system should provide a version management function to store historically generated link graphs so that users can compare the changes in dependencies at different time points.

[0140] Example description: In the customer management system of a bank, the user wants to understand the data source and usage of the "customer credit rating" field. Through this system, an upstream and downstream data link diagram of this field is automatically generated. The diagram marks the "customer transaction record table" (upstream source) and the "credit approval table" (downstream destination), and also marks the field name, generator (such as risk control team, credit approval team) and processing logic (such as data aggregation and risk score calculation). The link diagram is stored in the bank's data management system for the credit team and data analysts to call at any time to understand the data dependency and processing process.

[0141] This embodiment generates an upstream and downstream data link diagram of the target field, and the system provides an intuitive way to help users understand the upstream and downstream dependencies of the target field. The link diagram not only marks the data table name and field information, but also displays the data generator and the processing logic in each data flow, which improves the visualization of the data flow. Through the link diagram, users can quickly identify key data tables, data flow paths, and processing logic, thereby optimizing data management, improving data maintenance efficiency, and quickly locating data problems.

[0142] In one embodiment, the above S40 includes:

[0143] S401, analyzing the connection relationship between the target field and the corresponding upstream and downstream fields in the upstream and downstream data link diagram;

[0144] S402, based on the analysis of the connection relationship in the upstream and downstream data link diagram, identifying and determining the dependency path between the target field and the upstream and downstream fields;

[0145] S403, recording the field source information and field destination information in the dependency link in a data processing module.

[0146] In this embodiment, the upstream and downstream data link diagram shows the dependency path between the target field and other fields. The connection relationship refers to the transmission path of the data flow from the upstream to the downstream field. By analyzing the connection relationship between the fields in the link diagram, the system can identify how the target field and other fields are related to each other.

[0147] The system uses the graph analysis tool in the data processing module to automatically analyze the connection relationship between nodes (data tables, fields) and edges (dependencies) in the link graph. Graph theory algorithms (such as DFS or BFS) are used to parse the dependency paths in the link graph. The connection relationship between each field is displayed through a visual interface to help users understand the upstream and downstream associations between the target field and other fields.

[0148] A dependency path refers to the complete dependency chain between a target field and its upstream and downstream fields, indicating how data flows from an upstream field to a target field, or from a target field to a downstream field. Identifying dependency paths is a key step in analyzing the upstream and downstream associations of target fields and data processing flows.

[0149] Through the connection relationship in the dependency link, the system uses graph theory algorithms (such as the shortest path algorithm or the dependency link algorithm) to identify the complete dependency path. The system automatically identifies all upstream and downstream fields of the target field and identifies the key nodes in the dependency path. The dependency path is marked with different colors or line styles in the link diagram to clearly show how data flows from the upstream field to the target field, or from the target field to the downstream field.

[0150] The field source information and destination information represent where the target field gets data from and how the data is further transferred to the downstream. Recording this information in the data processing module can help users quickly query the upstream and downstream dependencies of the field.

[0151] The system stores the source and destination information of the fields in the dependency link in the dependency database of the data processing module. The field source and destination information includes metadata such as field name, data table name, data generator, etc. The system creates a dependency table for each field, which contains detailed information on the field source, destination, and dependency path. Users can quickly retrieve and view the dependency link of the target field through the query function.

[0152] By analyzing the connection relationship in the upstream and downstream data link diagram and identifying the dependency path of the target field, this embodiment enables the system to quickly and accurately locate the upstream source and downstream destination of the target field. This not only improves the efficiency of data dependency analysis, but also helps users have a more intuitive understanding of the complex data flow process. The field source and destination information in the dependency link is automatically recorded, ensuring the transparency of the data management process and providing data managers with effective tools to analyze and optimize the data processing process.

[0153] In one embodiment, the above S50 includes:

[0154] S501, analyzing the field relationship in the dependency link to determine the data table on which the target field depends;

[0155] S502, determining a unique identifier and name of each dependent data table;

[0156] S503, retrieving owner information of each dependent data table from the metadata information of the data resource, the owner information including the owner's name, contact information and department;

[0157] S504, associating the owner information with the corresponding data table;

[0158] S505, storing the associated owner information in a database of the data processing module.

[0159] In this embodiment, the dependency link shows the dependency path between the target field and other data tables. By analyzing these dependencies, the system can identify the upstream data tables that the target field depends on. Each target field has a dependency relationship with certain data tables, such as through a foreign key or an associated table connection.

[0160] Through graph theory algorithms (such as depth-first search DFS or breadth-first search BFS) or database-based dependency tables, the system can automatically analyze the field relationships in the dependency link and determine which data tables the target field has a dependency relationship with. The system parses the upstream source data table of the target field from the records of the dependency link and lists all directly dependent tables.

[0161] Each data table has a unique identifier and name in the system. The identifier is used to uniquely identify the data table, and the name is used for human-readable representation. The purpose of this step is to assign identifiers and names to each dependent data table to ensure accurate distinction in the system.

[0162] The system extracts the unique identifier (such as table ID) and name of each data table from the database or metadata system. If there is no ready-made identifier, the system automatically generates a UUID or other unique identifier. The system retrieves the name, structure, and identifier of each dependent data table by querying the metadata table to ensure that the dependencies between fields and tables are clearly represented.

[0163] Owner information includes details about the team, individual, or department responsible for maintaining a data table. This information is critical for managing and maintaining data tables, especially when collaborating across departments. Owner information typically includes name, contact information, and department.

[0164] The system retrieves the owner information of each dependent data table from the metadata repository. This information may be stored in a database management system, data management platform, or project management tool. The retrieval content includes the owner's name, contact number, email, team or department, etc. The system matches and associates the owner information with the corresponding data table based on the unique identifier of each data table to ensure that the responsible party for each table is clear and unambiguous.

[0165] The owner information of each data table must be associated with it so that the system can quickly find the person responsible for the table according to the needs. After the association, the user can quickly locate its owner information by querying a field or table.

[0166] The system creates an association table in the data processing module to associate the unique identifier and name of the data table with the owner information. The association table can be queried efficiently, helping users quickly retrieve the table and its owner information that the target field depends on. Using foreign key constraints in SQL databases or document nesting in NoSQL databases, the system associates the owner information with the data table to ensure data consistency.

[0167] All records of related data tables and owner information will be stored in the data processing module for subsequent query, update or analysis. The system needs to ensure that this information can be persisted and provide fast retrieval capabilities.

[0168] The system stores the associated records of the data table and the owner information in the database. You can use a relational database (such as MySQL, PostgreSQL) or a NoSQL database (such as MongoDB) for storage so that you can quickly access these records during subsequent queries. To speed up subsequent query operations, the system optimizes the index of the associated information between the owner information and the data table, and ensures that the owner information retrieval is fast and efficient by creating appropriate database indexes (such as B-tree or hash indexes).

[0169] By analyzing the dependency link and determining the data table and owner information that the target field depends on, this embodiment can provide users with a more transparent field management view, making data maintenance more efficient. The system automatically retrieves and associates the owner information of each data table, allowing users to quickly locate the maintenance responsible party, facilitating cross-team collaboration and problem troubleshooting. This automated information association and storage process reduces the complexity of manual query and management, and improves data management efficiency.

[0170] In one embodiment, after the above S50, the method further includes:

[0171] S601, generating a dynamic visual association diagram of the data table and the owner information to display the dependency relationship between the target field and the data table and the corresponding owner;

[0172] S602, analyzing the usage frequency and maintenance requirements of the data table to which the target field depends based on the owner information of the data table to which the target field depends in the data processing module;

[0173] S603: Based on the analysis result, generate an adjustment plan for the owner of the data table to which the target field depends.

[0174] In this embodiment, the dynamic visualization association diagram is used to show the relationship between the target field and the data table it depends on, as well as the relationship between the corresponding owners of the data table. Through the visualization tool, the user can clearly see how each data table depends on other tables, and how the owner (i.e., maintenance person) of each table is assigned. Dynamicity means that the chart can change in real time as the data is updated.

[0175] The system uses graphical tools (such as D3.js, Graphviz, etc.) to automatically generate association graphs that display target fields, data tables, and owner information. Nodes represent data tables, and the relationship between nodes and owners is displayed through edge connections. The system will update the association graph regularly or in real time based on changes in data table owner information to ensure that the dependency relationships and owner information seen by users are the latest. Users can view detailed information by clicking on a data table node or owner node, dynamically adjust the graph layout, and filter dependency links for specific fields.

[0176] Usage frequency refers to how often a data table is queried, used, or updated, and maintenance requirements refer to the number of times, workload, or other key maintenance indicators for these data tables. By analyzing this information, the system can help users understand which data tables require more maintenance resources and which owner has more maintenance work.

[0177] The system analyzes the usage frequency of the data table that each target field depends on based on metadata logs, query logs or operation records. The number of times each data table is accessed, updated or queried can be counted. The system counts the number of maintenance times and maintenance content of each data table based on maintenance logs and the owner's operation records. This includes maintenance actions such as repair operations, data table updates, and field addition or deletion. The system automatically generates a report on the usage frequency and maintenance requirements of each data table and displays it to the user to help formulate subsequent maintenance plans.

[0178] The adjustment plan is to optimize the owner allocation or maintenance responsibilities of the data table based on the analysis results. By analyzing the usage frequency, maintenance requirements and owner workload, the system can propose an owner adjustment plan to ensure the reasonable allocation of maintenance resources and improve data management efficiency.

[0179] Based on the frequency of use and maintenance requirements, the system automatically calculates the workload of the current owner of each data table. If an owner is responsible for too many tables or the maintenance frequency is too high, the system will recommend reallocating the responsibilities of these tables to other owners. The system generates an owner adjustment plan and provides charts or visualization tools to show the changes in the relationship between the original owner and the new owner. The user can choose to accept or manually adjust the plan. Based on the user's selection or automatic suggestion, the system will implement the owner adjustment plan into the database, update the association information between the data table and the owner, and record the adjustment history.

[0180] Example: In the bank's data management system, multiple data tables are dependent on the "customer information" field. The system generates a dynamic visual association diagram to display all the data tables and their owners that the "customer information" field depends on, and updates the owner information of these data in real time. By analyzing the usage frequency and maintenance requirements of each table, the system generates an optimized owner adjustment plan, reallocates overburdened data tables to other teams or personnel, and ensures that the bank's data management work is more efficient and resources are allocated reasonably.

[0181] This embodiment generates a dynamic visual association diagram, and the system intuitively displays the relationship between the target field and its dependent data table and owner, so that users can more clearly understand the dependency of the data table. The frequency of use and maintenance requirements analysis helps to optimize the maintenance plan of the data table, and through the automatic generation and execution of the owner adjustment plan, the system can reasonably allocate data maintenance responsibilities, improve data management efficiency and reduce the workload of the owner.

[0182] In one embodiment, a data dependency confirmation device is provided, and the data dependency confirmation device corresponds one-to-one to the data dependency confirmation method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the data dependency confirmation device of the present invention. Metadata import module 10, field search module 20, data link diagram generation module 30, dependency link analysis module 40 and dependency confirmation module 50. The functional modules are described in detail as follows:

[0183] The metadata import module 10 is used to import the metadata information of the data resource into the data processing module, wherein the metadata information includes the field name, the data table name and the data source information;

[0184] A field search module 20 is used to obtain a target field name, and based on the data processing module, search the target field name in the metadata information to determine the upstream data source and downstream data destination of the target field;

[0185] A data link diagram generating module 30, for generating an upstream and downstream data link diagram of the target field based on the upstream data source and the downstream data destination, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field;

[0186] A dependency link analysis module 40, configured to analyze and determine a dependency link between the target field and corresponding upstream and downstream fields based on the upstream and downstream data link diagram;

[0187] The dependency confirmation module 50 is used to determine the data table and corresponding owner information that the target field depends on based on the dependency link.

[0188] In one embodiment, the metadata import module 10 is specifically used for:

[0189] Collect metadata information including field names, data table names, and data source information from multiple data resources, and assign a unique identifier to each data table name;

[0190] Performing format standardization processing on the metadata information;

[0191] Perform data cleaning operations on the standardized metadata information, including removing duplicate or erroneous information;

[0192] The processed metadata information is imported into the database of the data processing module.

[0193] In one embodiment, the field search module 20 is specifically used to:

[0194] The name of the target field that receives user input through the user interface;

[0195] Performing a search operation on the target field name in the metadata information by the data processing module to locate the data table where the target field is located;

[0196] Analyze the associated table structure of the data table where the target field is located, trace back and determine the upstream data source and downstream data destination of the target field;

[0197] The information of the upstream data source and the downstream data destination is recorded in the data processing module.

[0198] In one embodiment, the data link diagram generation module 30 is specifically used to:

[0199] Based on the upstream data source and downstream data destination of the target field, generate an upstream and downstream data link diagram of the target field using a graphical module;

[0200] Annotating the name of each data table and field information associated with the target field in the data table in the upstream and downstream data link diagram;

[0201] Mark the data generator of each data table in the upstream and downstream data link diagram;

[0202] Describe the processing logic in each data flow process in the upstream and downstream data link diagram, wherein the processing logic includes the conversion, calculation or processing process of the data;

[0203] The upstream and downstream data link diagram is stored in the data processing module.

[0204] In one embodiment, the dependency link analysis module 40 is specifically configured to:

[0205] Analyze the connection relationship between the target field and the corresponding upstream and downstream fields in the upstream and downstream data link diagram;

[0206] Based on the analysis of the connection relationship in the upstream and downstream data link diagram, identify and determine the dependency path between the target field and the upstream and downstream fields;

[0207] The field source information and field destination information in the dependency link are recorded in the data processing module.

[0208] In one embodiment, the dependency confirmation module 50 is specifically configured to:

[0209] Analyze the field relationship in the dependency link to determine the data table on which the target field depends;

[0210] Determine the unique identifier and name of each dependent data table;

[0211] Retrieving owner information of each dependent data table from metadata information of the data resource, the owner information including the owner's name, contact information and department;

[0212] Associating the owner information with the corresponding data table;

[0213] The associated owner information is stored in the database of the data processing module.

[0214] In one embodiment, the dependency confirmation module 50 is specifically configured to:

[0215] Generate a dynamic visual association diagram of data tables and owner information to show the dependency relationship between the target field and the data table and the corresponding owner;

[0216] Based on the owner information of the data table to which the target field depends in the data processing module, analyzing the usage frequency and maintenance requirements of the data table to which the target field depends;

[0217] Based on the analysis result, an adjustment plan for the owner of the data table to which the target field depends is generated.

[0218] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a data dependency confirmation method service side.

[0219] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a user-side method for confirming a data dependency relationship

[0220] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0221] Importing metadata information of data resources into the data processing module, the metadata information includes field names, data table names and data source information;

[0222] Obtaining a target field name, and based on the data processing module, retrieving the target field name in the metadata information, and determining an upstream data source and a downstream data destination of the target field;

[0223] Based on the upstream data source and the downstream data destination, an upstream and downstream data link diagram of the target field is generated, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field;

[0224] Based on the upstream and downstream data link diagram, analyzing and determining the dependency link between the target field and the corresponding upstream and downstream fields;

[0225] Based on the dependency link, the data table and corresponding owner information on which the target field depends are determined.

[0226] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0227] Importing metadata information of data resources into the data processing module, the metadata information includes field names, data table names and data source information;

[0228] Obtaining a target field name, and based on the data processing module, retrieving the target field name in the metadata information, and determining an upstream data source and a downstream data destination of the target field;

[0229] Based on the upstream data source and the downstream data destination, an upstream and downstream data link diagram of the target field is generated, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field;

[0230] Based on the upstream and downstream data link diagram, analyzing and determining the dependency link between the target field and the corresponding upstream and downstream fields;

[0231] Based on the dependency link, the data table and corresponding owner information on which the target field depends are determined.

[0232] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0233] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0234] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0235] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for confirming data dependency, characterized in that: The following steps are involved: Importing metadata information of data resources into the data processing module, the metadata information includes field names, data table names and data source information; Obtaining a target field name, and based on the data processing module, retrieving the target field name in the metadata information, and determining an upstream data source and a downstream data destination of the target field; Based on the upstream data source and the downstream data destination, an upstream and downstream data link diagram of the target field is generated, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field; Based on the upstream and downstream data link diagram, analyzing and determining the dependency link between the target field and the corresponding upstream and downstream fields; Based on the dependency link, the data table and corresponding owner information on which the target field depends are determined.

2. The data dependency confirmation method according to claim 1, characterized in that: Import the metadata information of the data resource into the data processing module, the metadata information includes field name, data table name and data source information, including: Collect metadata information including field names, data table names, and data source information from multiple data resources, and assign a unique identifier to each data table name; Performing format standardization processing on the metadata information; Perform data cleaning operations on the standardized metadata information, including removing duplicate or erroneous information; The processed metadata information is imported into the database of the data processing module.

3. The data dependency confirmation method according to claim 1, characterized in that: Obtaining a target field name, retrieving the target field name in the metadata information based on the data processing module, and determining an upstream data source and a downstream data destination of the target field, including: The name of the target field that receives user input through the user interface; Performing a search operation on the target field name in the metadata information by the data processing module to locate the data table where the target field is located; Analyze the associated table structure of the data table where the target field is located, trace back and determine the upstream data source and downstream data destination of the target field; The information of the upstream data source and the downstream data destination is recorded in the data processing module.

4. The data dependency confirmation method according to claim 1, characterized in that: Based on the upstream data source and the downstream data destination, an upstream and downstream data link diagram of the target field is generated, including: Based on the upstream data source and downstream data destination of the target field, generate an upstream and downstream data link diagram of the target field using a graphical module; Annotating the name of each data table and field information associated with the target field in the data table in the upstream and downstream data link diagram; Mark the data generator of each data table in the upstream and downstream data link diagram; Describe the processing logic in each data flow process in the upstream and downstream data link diagram, wherein the processing logic includes the conversion, calculation or processing process of the data; The upstream and downstream data link diagram is stored in the data processing module.

5. The data dependency confirmation method as claimed in claim 1, characterized in that: Based on the upstream and downstream data link diagram, analyzing and determining the dependency link between the target field and the corresponding upstream and downstream fields, including: Analyze the connection relationship between the target field and the corresponding upstream and downstream fields in the upstream and downstream data link diagram; Based on the analysis of the connection relationship in the upstream and downstream data link diagram, identify and determine the dependency path between the target field and the upstream and downstream fields; The field source information and field destination information in the dependency link are recorded in the data processing module.

6. The data dependency confirmation method according to claim 1, characterized in that: Based on the dependency link, determining the data table and corresponding owner information that the target field depends on includes: Analyze the field relationship in the dependency link to determine the data table on which the target field depends; Determine the unique identifier and name of each dependent data table; Retrieving owner information of each dependent data table from metadata information of the data resource, the owner information including the owner's name, contact information and department; Associating the owner information with the corresponding data table; The associated owner information is stored in the database of the data processing module.

7. The data dependency confirmation method according to claim 1, characterized in that: After determining the data table and corresponding owner information that the target field depends on based on the dependency link, the method further includes: Generate a dynamic visual association diagram of data tables and owner information to show the dependency relationship between the target field and the data table and the corresponding owner; Based on the owner information of the data table to which the target field depends in the data processing module, analyzing the usage frequency and maintenance requirements of the data table to which the target field depends; Based on the analysis result, an adjustment plan for the owner of the data table to which the target field depends is generated.

8. A data dependency confirmation device, characterized in that: The data dependency confirmation device comprises: A metadata import module, used to import metadata information of data resources into a data processing module, wherein the metadata information includes field names, data table names and data source information; A field search module, used to obtain a target field name, retrieve the target field name in the metadata information based on the data processing module, and determine the upstream data source and downstream data destination of the target field; A data link diagram generation module, used to generate an upstream and downstream data link diagram of the target field based on the upstream data source and the downstream data destination, wherein the upstream and downstream data link diagram is used to identify the upstream and downstream data flow relationship, data generator and processing logic of the target field; A dependency link analysis module, used to analyze and determine the dependency link between the target field and the corresponding upstream and downstream fields based on the upstream and downstream data link diagram; The dependency confirmation module is used to determine the data table and corresponding owner information that the target field depends on based on the dependency link.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data dependency confirmation program stored in the memory and executable on the processor. When the data dependency confirmation program is executed by the processor, the steps of the data dependency confirmation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a data dependency confirmation program, which, when executed by a processor, implements the steps of the data dependency confirmation method according to any one of claims 1 to 7.

Images