Machine learning-based methods, systems, and electronic devices for multi-source heterogeneous data analysis.

Abstract

This disclosure provides a machine learning-based method for analyzing multi-source heterogeneous data, belonging to the field of data governance technology. This method parses and processes data flow logs from multi-source heterogeneous databases, and combines this with a graph database storing the first flow path and node dependencies in the lineage information set. This enables visualization and traceability of lineage information, making the data source and flow path clear and transparent. By determining the dynamic set of relationships between each flow node, it can dynamically capture the relationships between flow nodes, promptly identify inconsistent dependencies, and locate inconsistent dependency fields. Error fields are identified through inconsistent dependency fields for correction, generating a corrected data set. A second data flow path is constructed based on the corrected data set. After verifying the path integrity, an optimized lineage information set is generated, thus effectively improving the accuracy of lineage information and enabling it to accurately capture real-time changes in data flow.

Description

Technical Field

[0001] This disclosure relates to the field of data governance technology, specifically to a method, system, and electronic device for multi-source heterogeneous data analysis based on machine learning. Background Technology

[0002] Currently, with the diversification of data sources and the surge in data volume among financial institutions, data governance plays a crucial role in ensuring data security, improving decision-making efficiency, and meeting regulatory requirements. Traditional data governance methods often struggle to cope with the dual pressures of dynamic changes in data sources and privacy protection in complex data environments. They typically rely on static rules or manual intervention, which are ill-suited to scenarios with frequent changes in data sources. Furthermore, data sharing is prone to compliance issues due to privacy risks, particularly in cross-institutional collaborations where data silos further exacerbate governance difficulties. In addition, dynamic lineage analysis and secure sharing through federated learning have become key technical challenges in the governance of multi-source heterogeneous data. In cross-system data integration, if lineage analysis fails to accurately capture real-time changes in data flow, it may lead to erroneous data associations, affecting the accuracy of credit approval or risk assessment. This inaccuracy further complicates the challenges of secure sharing through federated learning.

[0003] The information disclosed in the background section is only for enhancing the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] In view of this, this disclosure provides a multi-source heterogeneous data analysis method based on machine learning, which can improve the accuracy of pedigree analysis while ensuring data security.

[0005] In a first aspect, embodiments of this application provide a multi-source heterogeneous data analysis method based on machine learning. The method includes: acquiring data flow logs from a multi-source heterogeneous database, the data flow logs including timestamps, IP addresses of data sources, IDs of flow nodes, and data packet sizes; determining a lineage information set based on the data flow logs, the lineage information set including data sources, a first data flow path, and flow node dependencies; using a graph database to store the first data flow path and the flow node dependencies to generate a graph database structure; assigning a unique identifier to each flow node in the graph database structure based on a node label allocation algorithm to generate a node identifier set; and performing graph traversal algorithms... The method involves: determining a dynamic set of relationships between each data transfer node based on the node identifier set; determining whether there are inconsistent dependencies in the dynamic set of relationships; if inconsistent dependencies exist, obtaining inconsistent dependency fields; determining a set of error fields based on the inconsistent dependency fields, the error fields including missing values, outliers, and inconsistent formats; repairing the set of error fields using a data cleaning algorithm to generate a repaired data set; constructing a second data transfer path based on the repaired data set; determining whether the second data transfer path has path integrity based on a path traversal algorithm; if the path integrity is determined to be present, generating an optimized lineage information set based on the second data transfer path.

[0006] Secondly, embodiments of this application provide a multi-source heterogeneous data analysis system based on machine learning. This system includes a first acquisition module, a first determination module, a first generation module, a second generation module, a second determination module, a first judgment module, a second acquisition module, a third determination module, a repair module, a construction module, a second judgment module, and a third generation module. Specifically, the first acquisition module acquires data flow logs from a multi-source heterogeneous database. The data flow logs include timestamps, IP addresses of data sources, IDs of flow nodes, and the size of data packets. The first determination module determines a lineage information set based on the data flow logs. The lineage information set includes data sources, a first data flow path, and flow node dependencies. The first generation module uses a graph database to store the first data flow path and the flow node dependencies to generate a graph database structure. The second generation module assigns a unique identifier to each flow node in the graph database structure based on a node label allocation algorithm to generate a node identifier set. The second determination module determines the dynamic relationships between each flow node based on a graph traversal algorithm and the node identifier set. The system comprises: a dynamic relation set; a first judgment module for judging whether inconsistent dependencies exist in the dynamic relation set; a second acquisition module for acquiring inconsistent dependency fields if inconsistent dependencies exist; a third determination module for determining a set of erroneous fields based on the inconsistent dependency fields, the erroneous fields including missing values, outliers, and inconsistent formats; a repair module for repairing the set of erroneous fields based on a data cleaning algorithm to generate a repaired data set; a construction module for constructing a second data flow path based on the repaired data set; a second judgment module for judging whether the second data flow path has path integrity based on a path traversal algorithm; and a third generation module for generating an optimized lineage information set based on the second data flow path if the path integrity is determined to be present.

[0007] This application provides a machine learning-based method for analyzing multi-source heterogeneous data. By parsing and processing data flow logs in a multi-source heterogeneous database, it accurately extracts information such as timestamps, IP addresses of data sources, and IDs of flow nodes. Combined with a graph database storing the first flow path and node dependencies in the lineage information set, it achieves visualization and traceability of lineage information, making the data source and flow path clear and transparent. Using node label allocation and graph traversal algorithms, it determines the dynamic set of relationships between each flow node, dynamically capturing these relationships, promptly identifying inconsistent dependencies, and locating inconsistent dependency fields to ensure the accuracy of data association. It identifies erroneous fields through inconsistent dependency fields and repairs them using data cleaning algorithms to generate a repaired data set. A second data flow path is then constructed based on this repaired data set. After verifying the path's integrity, an optimized lineage information set is generated. This effectively improves the accuracy of lineage information, enabling it to accurately capture real-time changes in data flow, thereby enhancing the accuracy of credit approval or risk assessment. It provides a high-quality and reliable foundation for subsequent cross-institutional data sharing and federated learning modeling, while enhancing the intelligence and automation of data governance, and helping to efficiently address the challenges of dynamic changes and complex relationships in a multi-source heterogeneous data environment. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments or conventional technologies of this disclosure, the accompanying drawings used in the description of the embodiments or conventional technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating a multi-source heterogeneous data analysis method based on machine learning provided in an exemplary embodiment of this application.

[0010] Figure 2 This is a flowchart illustrating a machine learning-based multi-source heterogeneous data analysis method provided in another exemplary embodiment of this application.

[0011] Figure 3 This is a flowchart illustrating a machine learning-based multi-source heterogeneous data analysis method provided in another exemplary embodiment of this application.

[0012] Figure 4 This is a flowchart illustrating a multi-source heterogeneous data analysis method based on machine learning, provided in another exemplary embodiment of this application.

[0013] Figure 5This is a flowchart illustrating a multi-source heterogeneous data analysis method based on machine learning, provided in another exemplary embodiment of this application.

[0014] Figure 6 This is a flowchart illustrating a multi-source heterogeneous data analysis method based on machine learning, provided in another exemplary embodiment of this application.

[0015] Figure 7 This is a flowchart illustrating a multi-source heterogeneous data analysis method based on machine learning, provided in another exemplary embodiment of this application.

[0016] Figure 8 This is a flowchart illustrating a multi-source heterogeneous data analysis method based on machine learning, provided in another exemplary embodiment of this application. Detailed Implementation

[0017] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are set forth to give a full understanding of embodiments of this disclosure.

[0018] The terms “a,” “one,” and “the” are used to indicate the existence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and that other elements / components / etc. may exist in addition to those listed. The terms “first” and “second” are used only as markers and are not a limitation on the number of objects.

[0019] Currently, with the diversification of data sources and the surge in data volume among financial institutions, data governance plays a crucial role in ensuring data security, improving decision-making efficiency, and meeting regulatory requirements. Traditional data governance methods often struggle to cope with the dual pressures of dynamic changes in data sources and privacy protection in complex data environments. They typically rely on static rules or manual intervention, which are ill-suited to scenarios with frequent changes in data sources. Furthermore, data sharing is prone to compliance issues due to privacy risks, particularly in cross-institutional collaborations where data silos further exacerbate governance difficulties. In addition, dynamic lineage analysis and secure sharing through federated learning have become key technical challenges in the governance of multi-source heterogeneous data. In cross-system data integration, if lineage analysis fails to accurately capture real-time changes in data flow, it may lead to erroneous data associations, affecting the accuracy of credit approval or risk assessment. This inaccuracy further complicates the challenges of secure sharing through federated learning.

[0020] This disclosure provides a multi-source heterogeneous data analysis method based on machine learning, such as... Figure 1 The method shown is a machine learning-based approach for analyzing multi-source heterogeneous data. This method may include the following steps:

[0021] Step S110: Obtain the data flow log from the multi-source heterogeneous database. The data flow log includes timestamps, IP addresses of data sources, IDs of flow nodes, and the size of data packets.

[0022] Step S120: Determine the lineage information set based on the data flow log. The lineage information set includes the data source, the first data flow path, and the dependency relationship of the flow nodes.

[0023] Step S130: Utilize a graph database to store the first data flow path and the dependency relationships of the flow nodes to generate a graph database structure;

[0024] Step S140: Based on the node label allocation algorithm, assign a unique identifier to each flow node in the graph database structure to generate a node identifier set;

[0025] Step S150: Based on the graph traversal algorithm, determine the dynamic association set between each flow node according to the node identifier set;

[0026] Step S160: Determine whether there are inconsistent dependencies in the dynamic association set;

[0027] Step S170: If an inconsistent dependency is determined, obtain the inconsistent dependency field;

[0028] Step S180: Determine the set of error fields based on the inconsistent dependency fields. Error fields include missing values, outliers, and inconsistent formats.

[0029] Step S190: Based on the data cleaning algorithm, repair the set of erroneous fields to generate a repaired data set;

[0030] Step S192: Construct a second data flow path based on the repaired data set;

[0031] Step S194: Based on the path traversal algorithm, determine whether the second data flow path has path integrity;

[0032] Step S196: If the path integrity is determined, an optimized bloodline information set is generated based on the second data flow path.

[0033] According to the machine learning-based multi-source heterogeneous data analysis method provided in this disclosure, the method can obtain data flow logs from a multi-source heterogeneous database. These logs include timestamps, IP addresses of data sources, IDs of flow nodes, and packet sizes. Based on the data flow logs, a lineage information set is determined, including data sources, a first data flow path, and flow node dependencies. A graph database is used to store the first data flow path and flow node dependencies to generate a graph database structure. A node label allocation algorithm is used to assign a unique identifier to each flow node in the graph database structure to generate a node identifier set. Finally, based on graph traversal... The algorithm determines the dynamic association set between each flow node based on the node identifier set; determines whether there are inconsistent dependencies in the dynamic association set; if inconsistent dependencies are found, obtains the inconsistent dependency fields; determines the set of erroneous fields based on the inconsistent dependency fields, including missing values, outliers, and inconsistent formats; repairs the set of erroneous fields based on the data cleaning algorithm to generate a repaired data set; constructs a second data flow path based on the repaired data set; determines whether the second data flow path has path integrity based on the path traversal algorithm; if path integrity is determined, generates an optimized lineage information set based on the second data flow path.

[0034] In the above method, data flow logs from multi-source heterogeneous databases are parsed and processed to accurately extract information including timestamps, IP addresses of data sources, and IDs of flow nodes. Combined with a graph database storing the first flow path and node dependencies in the lineage information set, the visualization and traceability of lineage information are achieved, making the data source and flow path clear and transparent. Using node labeling and graph traversal algorithms, a dynamic set of relationships between each flow node is determined, dynamically capturing these relationships, promptly identifying inconsistent dependencies, and locating inconsistent dependency fields to ensure the accuracy of data association. Inconsistent dependency fields are used to identify erroneous fields, which are then repaired using data cleaning algorithms to generate a repaired data set. A second data flow path is constructed based on this repaired data set. After verifying the path's integrity, an optimized lineage information set is generated. This effectively improves the accuracy of lineage information, enabling it to accurately capture real-time changes in data flow, thereby enhancing the accuracy of credit approval or risk assessment. It provides a high-quality and reliable foundation for subsequent cross-institutional data sharing and federated learning modeling, while enhancing the intelligence and automation of data governance, and helping to efficiently address the challenges of dynamic changes and complex relationships in a multi-source heterogeneous data environment.

[0035] The following is a detailed description of each step in the machine learning-based multi-source heterogeneous data analysis method provided in this disclosure:

[0036] In one embodiment of this disclosure, step S110 involves obtaining data flow logs from a multi-source heterogeneous database. These logs include a timestamp, the IP address of the data source, the ID of the flow node, and the size of the data packet. Specifically, obtaining data flow logs from the multi-source heterogeneous data system can be achieved through a distributed message queue such as Kafka. A Kafka cluster (3 nodes, processing 100,000 logs per second) is deployed. Each data flow log entry includes a timestamp, the IP address of the data source, the ID of the flow node, and the size of the data packet (e.g., 1024 bytes).

[0037] For example, in a financial institution's multi-source heterogeneous data system, a user credit data flow log was collected through a Kafka cluster. The log contained the following information: "time_stamp: 2025-08-01-14:30, source_ip: 192.168.1.101, node_id: NODE_007, packet_size: 1536 bytes". The timestamp "2025-08-01 14:30:22.567" precisely records the time of data flow. The data source IP "192.168.1.101" points to the user information database server, the flow node ID "NODE_007" corresponds to the processing node in the credit approval system, and the data packet size is 1536 bytes. It can be seen that this data flow log includes user basic information, credit scores, and other data. The data flow log is aggregated in real time using a distributed log collection tool, providing the original basis for analyzing data sources, flow paths, and node dependencies.

[0038] In one embodiment of this disclosure, in step S120, a lineage information set is determined based on the data flow log. The lineage information set includes the data source, the first data flow path, and the dependency relationship of the flow nodes. Specifically, if the data flow log records "timestamp: 2025-08-01 09:00:00, data source IP address: 192.168.1.102, flow node ID: flow nodes A, B, C, data packet size: 2048 bytes", then the data source (e.g., "192.168.1.101"), the first data flow path (e.g., "flow node A -> flow node B -> flow node C"), and the node dependency relationship (e.g., "flow node B depends on the output of flow node A, and flow node C depends on the output of flow node B") are extracted from the lineage information set formed after parsing.

[0039] In the above method, by extracting and structuring the data source, the first data flow path, and node dependencies from the data flow log, a clear lineage information set is formed. This not only enables full-link traceability of data from source to destination, making the "origin and destination" of each piece of data traceable, but also lays a standardized foundation for subsequent graph database structure storage and dynamic correlation analysis. It also improves the transparency and controllability of data governance in a multi-source heterogeneous data environment and provides a reliable lineage basis for cross-system data collaboration and model training.

[0040] In one embodiment of this disclosure, in step S130, a graph database is used to store the first data flow path and the dependencies between flow nodes to generate a graph database structure. Specifically, if the first data flow path is: flow node A -> flow node B -> flow node C, and the dependencies between flow nodes are: flow node B depends on the output of flow node A, and flow node C depends on the output of flow node B, then in the graph database (such as Neo4j), flow nodes A, B, and C will be used as nodes, and directed edges will be used to represent the first data flow path (such as A→B→C). Weights will be added to the edges (such as a weight of 0.8 from flow node A to B indicating that 80% of the data flow is successful), forming a "node-edge-weight" graph database structure.

[0041] In the above methods, the "node-edge" structure of graph databases is naturally adapted to the network relationship of data flow, and can intuitively store the direction of the path and the strength of node dependencies (the strength of node dependencies is judged according to the size of the weight value). Compared with traditional table structures, it is easier to trace the entire data link, which can significantly improve the efficiency and traceability of visualization management of multi-source heterogeneous data flow, and support the precision and efficiency of data governance.

[0042] In one embodiment of this disclosure, in step S140, based on a node label allocation algorithm, each flow node in the graph database structure is assigned a corresponding unique identifier to generate a node identifier set. Specifically, if the graph database structure contains flow nodes such as order generation nodes, payment processing nodes, and inventory update nodes, a unique identifier is assigned to each flow node. For example, the order generation node is assigned the identifier "8f3b2a1c-4d5e-11eb-ae93-0242ac130002", and the payment processing node is assigned the identifier "9g4c3b2d-6e7f-22fc-bf04-1353bd241113". These one-to-one identifiers are stored in the node attributes, forming a node identifier set containing the unique identifiers of all flow nodes.

[0043] In the above method, the unique identifier ensures the uniqueness of each flow node in the graph database structure. Even if the flow node name or attribute is duplicated, it can be accurately distinguished, providing a reliable foundation for the analysis of dynamic relationships between flow nodes. The unique identifier can quickly locate specific flow nodes and their associated paths, making it easy to track the complete data flow chain. This improves the accuracy and efficiency of node management in the graph database structure. At the same time, it provides a clear node identification basis for subsequent operations such as detecting inconsistent dependencies and repairing erroneous fields, ensuring the rigor of multi-source heterogeneous data flow analysis.

[0044] In one embodiment of this disclosure, in step S150, a dynamic set of relationships between various flow nodes is determined based on a graph traversal algorithm and a set of node identifiers. Specifically, if the set of node identifiers includes order nodes (e.g., a1b2c3d4), payment nodes (e.g., e5f6g7h8), and inventory nodes (e.g., i9j0k1l2), the graph traversal algorithm is used to traverse the graph database structure. Starting from the order node, the path weight is calculated along the directed edges. For example, for order → payment (path weight 0.9), 0.9 indicates that 90% of orders trigger payment, resulting in a dynamic set of relationships such as "order node → payment node (path weight 0.9), payment node → inventory node (path weight 0.8), order node → inventory node (path weight 0.3)," reflecting the real-time changes in the dependency strength between nodes as the business changes.

[0045] In the above method, the graph traversal algorithm can efficiently mine the dynamic relationships between flow nodes, including direct and indirect relationships. The weight values ​​can quantify the dependence strength and accurately capture dynamic changes. This provides a data foundation for subsequent detection of inconsistent dependencies, can promptly discover abnormal relationships, and makes the relationship logic between flow nodes clearer. It improves the depth and accuracy of multi-source heterogeneous data flow analysis and provides a reliable basis for monitoring and optimizing dynamic relationships in data governance.

[0046] In one embodiment of this disclosure, in steps S160-S170, it is determined whether there are inconsistent dependencies in the dynamic association set. If an inconsistent dependency is determined to exist, the inconsistent dependency field is obtained. The method also includes the following steps: Figure 2 As shown, the specific content is as follows:

[0047] Step S210: Obtain the edge weight values ​​between the current flow nodes. The edge weight values ​​are used to represent the dependence strength between each flow node.

[0048] Step S220: Obtain the edge weight values ​​between the current flow nodes. The edge weight values ​​are used to represent the dependency strength between each flow node.

[0049] Step S230: Determine whether the difference between the current edge weight value and the initial edge weight value is greater than a preset deviation threshold;

[0050] Step S240: If the deviation is greater than a preset deviation threshold, then an inconsistent dependency relationship is determined.

[0051] Step S250: Starting from any flow node, record all the third data flow paths corresponding to it, and calculate the path weight value corresponding to each third data flow path;

[0052] Step S260: Determine whether the difference between the path weight value corresponding to each third data flow path and the historical data flow path is greater than the second preset deviation threshold;

[0053] Step S270: If the deviation is greater than the second preset threshold, then mark it as a deviation flow path;

[0054] Step S280: Determine the inconsistent dependency fields based on the deviation flow path.

[0055] Specifically, in the graph database of an e-commerce platform, if the current edge weight value between an order node (e.g., o123) and a payment node (e.g., p456) is detected to be 0.3, a depth-first search algorithm is used to traverse the graph database structure to obtain an initial edge weight value of 0.8. The difference between the current edge weight value and the initial edge weight value is calculated to be 0.5, which is greater than the first preset deviation threshold of 0.2, indicating an inconsistent dependency relationship. Subsequently, starting from the order node, two third data flow paths are recorded as "order → payment → inventory" and "order → discount → payment", and the weight values ​​of each path are calculated to be 1.5 and 0.9, respectively. The path weight values ​​corresponding to their historical data flow paths are 0.5 and 0.3, respectively. The differences between the path weight values ​​corresponding to the two third data flow paths and the weight values ​​corresponding to the historical data flow paths are calculated to be 1 and 0.6, respectively. For example, if the second preset deviation threshold is 0.8, it can be seen that the first third data flow path (order → payment → inventory) is greater than 0.8 and is marked as a deviation flow path. The second third data flow path (order → discount → payment) is less than 0.8, so no action is required. Based on the above deviation flow paths, the "inventory" field of the payment node is determined to be an inconsistent dependency field.

[0056] The above method accurately locates inconsistent dependency fields in inconsistent dependencies by judging the deviation between edge weights and path weights, realizing automated detection of dependency anomalies, avoiding the subjectivity of manual judgment, and tracing back to specific fields. This provides clear targets for subsequent data cleaning and path optimization, effectively ensuring the consistency and reliability of associations in the flow of multi-source heterogeneous data.

[0057] In one embodiment of this disclosure, in step S180, a set of error fields is determined based on the inconsistent dependency fields. The error fields include missing values, outliers, and inconsistent formats. Specifically, if the "payment amount field" is determined to be an inconsistent dependency field in the transaction data flow of an e-commerce platform, this inconsistent dependency field exhibits three types of error fields in the data flow from the payment node (e.g., P789) to the order confirmation node (e.g., C123). For example, the "payment amount" field may be empty (missing value), displaying an amount as "-500 RMB" (an unreasonable outlier), or formatted as "300" (lacking a currency unit, inconsistent with the standard format "300 RMB"). These collectively constitute the set of error fields.

[0058] In the above method, data cleaning is made more targeted by identifying error field types such as missing values, outliers, and inconsistent formats. This precise positioning can avoid the waste of resources caused by indiscriminate cleaning, ensure that the subsequent repair process is efficient and reliable, lay a high-quality data foundation for generating an optimized lineage information set, and ultimately improve the accuracy and consistency of multi-source heterogeneous data flow.

[0059] In one embodiment of this disclosure, step S190, which involves repairing the erroneous field set based on a data cleaning algorithm to generate a repaired data set, further includes the following steps: Figure 3 As shown, the specific content is as follows:

[0060] Step S310: Standardize the set of error fields to generate a first standardized dataset;

[0061] Step S320: Calculate the covariance matrix of the first standardized dataset to determine the eigenvalues ​​and eigenvectors of the covariance matrix;

[0062] Step S330: Sort the eigenvalues ​​in order from maximum to minimum. In the sorting results, select the eigenvectors corresponding to the first five eigenvalues ​​to generate the feature matrix.

[0063] Step S340: Determine the corresponding repair scheme based on the type of the error field in the feature matrix.

[0064] Specifically, a set of erroneous fields in a certain financial credit data set includes missing values ​​(e.g., an empty "Income Certificate" field), outliers (e.g., a "Credit Score" of 999, far exceeding the reasonable range), and inconsistent formats (e.g., "Date of Birth" contains both "1994 / 10 / 01" and "1994-10-01"). Standardizing these erroneous fields normalizes the numerical fields to the [0,1] interval, generating a first standardized dataset. The covariance matrix of this first standardized dataset is calculated, yielding eigenvalues ​​of 5.2, 3.8, 2.1, 1.5, and 0.9 (all other eigenvalues ​​are less than 0.5). The eigenvectors corresponding to the first five eigenvalues ​​are selected to generate a feature matrix. Based on this, the corresponding repair scheme is determined according to the type of erroneous fields in the feature matrix.

[0065] In the above method, standardization eliminates the differences in the magnitude of fields in the erroneous field set, ensuring consistency in subsequent calculations. Covariance matrix and eigenvector analysis extract key features and relationships from the erroneous fields, making the repair plan more targeted and avoiding blind processing. This enables precise repair of erroneous fields, generating a high-quality repaired data set, providing reliable data support for constructing a complete second data flow path and an optimized lineage information set, ensuring the accuracy and consistency of data flow.

[0066] In one embodiment of this disclosure, step S340, determining the corresponding repair scheme based on the type of the error field in the feature matrix, further includes the following steps: Figure 4 As shown, the specific content is as follows:

[0067] Step S410: When there are missing values ​​in the feature matrix, obtain multiple neighbor values ​​corresponding to the missing values ​​based on the K-nearest neighbor interpolation algorithm;

[0068] Step S420: Perform a weighted average calculation on multiple neighbor values, and use the result as the missing value;

[0069] Step S430: When outliers exist in the feature matrix, calculate the standard scores of the outliers;

[0070] Step S440: Determine whether the standard score of the outlier exceeds the preset standard score threshold;

[0071] Step S450: If the score exceeds the preset standard threshold, the corresponding outlier is removed.

[0072] Step S460: When the feature matrix has inconsistent formats, use regular expressions to adjust the inconsistent formats to a unified format.

[0073] Specifically, if a bank's credit data feature matrix contains missing values ​​in the "Monthly Income" field, outliers in the "Loan Amount" field, and inconsistent formats in the "Account Opening Date" field, the following steps are taken: For missing values ​​in "Monthly Income," a K-nearest neighbor interpolation algorithm is used to select five neighbor (user) samples with similar credit ratings and occupational types. Their monthly incomes are 8000 yuan, 7500 yuan, 9000 yuan, 8500 yuan, and 7800 yuan, respectively. The weighted average is calculated to be 8160 yuan, which is used to fill in the missing values. For "Loan Amount," the occurrence of "10,000,000 yuan" is calculated to have a standard score of 3.2, exceeding the preset standard score threshold of 3.0, thus it is identified as an outlier and removed. For "Account Opening Date," formats such as "2020.05.12" and "2020 / 05 / 12" are used, and regular expressions are used to uniformly adjust them to "2020-05-12."

[0074] In the above method, missing values ​​are filled based on similar sample features using the K-nearest neighbor interpolation algorithm to ensure data correlation; outliers are identified and removed using standard scores to avoid interference from extreme data; and regular expressions ensure field standardization. These three elements work together to improve data quality. This method makes the repaired data more suitable for subsequent modeling and analysis, providing a reliable foundation for generating a complete second data flow path and an optimized lineage information set, effectively supporting the accurate governance and efficient flow of multi-source heterogeneous data.

[0075] In one embodiment of this disclosure, in step S192, a second data flow path is constructed based on the repaired data set. Specifically, the second data flow path is constructed based on the repaired data, eliminating erroneous fields in the first data flow path, ensuring the continuity and accuracy of data flow, laying the foundation for generating an optimized lineage information set, and ultimately improving the reliability and traceability of data flow in a multi-source heterogeneous data system, supporting cross-node and cross-system data collaboration and predictive model training.

[0076] In one embodiment of this disclosure, in steps S194-S196, based on a path traversal algorithm, it is determined whether the second data flow path has path integrity. If the path integrity is determined, an optimized lineage information set is generated based on the second data flow path. Specifically, if the second data flow path of a financial institution is "user account system → risk review node → credit approval node → loan record system", a path traversal algorithm (such as depth-first search, DFS) will start from the source node (user account system) and traverse each flow node and related edge layer by layer to check for broken or missing nodes or fields. If the traversal finds that each flow is completely transmitted after repair, and the input field of each flow node can match the output field of the previous flow node (such as the risk score accurately flowing from the review node to the approval node), then the path is determined to be complete. If the traversal finds that a certain flow node (such as the credit approval node) is missing the necessary "user credit field", then the path is determined to be incomplete and needs to be repaired again in the data cleaning stage. After confirming that the path is complete, an optimized lineage information set is generated based on the second data flow path.

[0077] In the above methods, path integrity verification ensures that the repaired data can flow reliably throughout the entire chain, avoiding information loss and improving the group defects of lineage information. The optimized lineage information set includes the repaired data flow path and field relationships, making data lineage clearer and more traceable. This provides high-quality lineage evidence for subsequent cross-institutional data sharing and federated learning modeling, while enhancing the accuracy and credibility of multi-source heterogeneous data governance and supporting the transparency and efficiency of data flow.

[0078] In one embodiment of this disclosure, in step S196, if path integrity is determined, after generating the optimized lineage information set based on the second data flow path, the following steps are further included: Figure 5 As shown, the specific content is as follows:

[0079] Step S510: Standardize the optimized bloodline information set to generate a second standardized dataset;

[0080] Step S520: Calculate the privacy budget for the second normalized dataset based on the differential privacy algorithm;

[0081] Step S530: Determine whether the privacy budget exceeds the preset privacy budget threshold;

[0082] Step S540: If it is determined that the preset privacy threshold has not been exceeded, Gaussian noise is added to the second standardized dataset to generate a perturbed dataset.

[0083] Step S550: Based on the federated learning framework, fuse the local model parameters of each institution in the perturbation dataset to generate a shared model parameter set;

[0084] Step S560: Determine whether the shared model parameter set has passed the consistency verification of each institution;

[0085] Step S570: If the consistency verification of each organization is satisfied, a secure shared dataset is generated.

[0086] Specifically, assuming the optimized lineage information set includes transaction amount (1000-5000 yuan), node transfer time (5-30 minutes), and node dependency strength (0.1-0.9), a second standardized dataset is generated using Z-score standardization. For example, a transaction amount of 4000 yuan is standardized to 1.0, a node transfer time of 20 minutes is standardized to 1.0, and a node dependency strength of 0.7 is standardized to 1.0. The privacy budget for this second standardized dataset is calculated using a differential privacy algorithm. In differential privacy, the privacy budget ε measures the strength of privacy protection (the smaller ε is, the stricter the privacy protection, and the slightly lower the data availability). For example, if ε = 1.2, which does not exceed the preset privacy budget threshold of 1.5, Gaussian noise with a standard deviation of approximately 0.417 can be added to generate a perturbed dataset. For example, the standardized value of a transaction amount of 1.0 is perturbed to 1.03; then, three financial institutions fuse their local model parameters based on a federated learning framework. The parameters of institutions A, B, and C are w1=[0.3,0.2], w2=[0.32,0.18], and w3=[0.28,0.22], respectively, to obtain a shared parameter set w=[0.3,0.2]. After consistency verification among the institutions (deviation ≤5%), the shared parameters and statistical features of the perturbed data are integrated to generate a secure shared dataset.

[0087] The methods described above protect data privacy and reduce the risk of data privacy breaches through differential privacy and noise addition. Federated learning frameworks can solve the data silo problem, enabling multiple institutions to securely share and update data in a timely manner. This results in a secure, shared dataset that supports multi-institutional collaborative modeling (such as credit risk prediction) while preventing privacy breaches, significantly improving the efficiency and security of utilizing multi-source heterogeneous data in cross-institutional scenarios.

[0088] In one embodiment of this disclosure, after step S570, if it is determined that the consistency verification of each organization is satisfied, and after generating the secure shared dataset, the following steps are further included: Figure 6 As shown, the specific content is as follows:

[0089] Step S610: Determine the training feature set based on the secure shared dataset;

[0090] Step S620: Determine the first initial model parameters based on the training feature set using the distributed gradient descent algorithm;

[0091] Step S630: Send the first initial model parameters to each mechanism;

[0092] Step S640: Based on local data, each institution calculates its corresponding local gradient update value;

[0093] Step S650: Summarize the local gradient update values ​​to generate the global gradient update values;

[0094] Step S660: Update the first initial model parameters based on the global gradient update value to generate the final model parameters;

[0095] Step S670: Generate a cross-institutional prediction model based on the final model parameters.

[0096] Specifically, when extracting training features from securely shared datasets, Principal Component Analysis (PCA) can be used to extract features from cross-institutionally shared medical datasets. For example, a securely shared dataset might include five indicators (age, blood pressure, blood glucose, BMI, and cholesterol) for 1000 patients, each normalized to a range of 0 to 1. The PCA algorithm calculates the covariance matrix, extracts the first two principal components, retains 80% of the variance, and generates a feature vector matrix (1000×2). The scikit-learn library in Python can be used to obtain a dimensionality-reduced training feature set, which reduces computational complexity and protects the privacy of the original data.

[0097] For example, three medical institutions each hold a portion of the data. They can determine the initial model parameters for training based on the dimensionality-reduced training feature set and send these parameters to each institution. Distributed training is then achieved using a federated learning framework (such as TensorFlow Federated). Each institution locally calculates the gradient of the logistic regression model, i.e., the update values ​​of each local gradient. Assuming an initial learning rate of 0.01, a batch size of 32, and 100 iterations, the local gradient update values ​​are aggregated to generate a global gradient update value, for example, [0.23, -0.15]. This gradient vector is then encrypted and transmitted to a central server. The encrypted gradient transmission mechanism can employ homomorphic encryption (such as the Paillier algorithm). The central server aggregates the encrypted gradients, decrypts them, and updates the initial model parameters, ensuring that the data does not leave its local machine. Finally, a cross-institutional prediction model is obtained through aggregated training. For example, a cross-institutional logistic regression model can be obtained, which can output the probability of a patient's disease risk.

[0098] In the above methods, the extraction of training feature sets reduces redundant information, encrypted transmission ensures data security, and distributed training improves the model's generalization ability, making it suitable for cross-institutional collaboration scenarios. Furthermore, each institution calculates local gradients based solely on its local data, without sharing original data. This strictly protects user privacy and complies with data security regulations while integrating data features from multiple institutions, making the prediction model more versatile. Distributed computing not only reduces the computational burden on individual institutions but also breaks down data silos, enabling cross-industry collaborative prediction in finance, payment, e-commerce, and other sectors. When adding a new institution, only initial parameters need to be synchronized and local gradients calculated; there is no need to reconstruct the prediction model. This makes it suitable for dynamically expanding multi-institutional collaboration scenarios, effectively balancing data security and model performance, and improving the efficiency and practicality of cross-institutional modeling.

[0099] In one embodiment of this disclosure, after step S670: generating the cross-institutional prediction model based on the final model parameters, the following steps are further included: Figure 7 As shown, the specific content is as follows:

[0100] Step S710: Determine whether the prediction accuracy of the cross-institutional prediction model does not exceed the first preset percentage threshold;

[0101] Step S720: If it is determined that the first preset percentage threshold has not been exceeded, then obtain the fourth data flow path in the optimized bloodline information set;

[0102] Step S730: Graph-based tracing algorithm, constructing a directed acyclic graph based on the fourth data flow path;

[0103] Step S740: Determine whether the edge weight value of the directed acyclic graph is greater than the preset path weight value;

[0104] Step S750: If the weight is greater than the preset path weight value, it is determined to be an abnormal flow path;

[0105] Step S760: Determine abnormal training features based on abnormal flow paths;

[0106] Step S770: Based on the feature reconstruction algorithm, optimize the abnormal training features to generate an optimized training feature set.

[0107] Specifically, the first preset percentage threshold (which can be 85%) serves as a benchmark for measuring model performance and determining whether the cross-institutional prediction model needs optimization. If the model's prediction accuracy falls below this threshold, the cause needs to be traced using the optimized lineage information set. A graph-based tracing algorithm (such as path reconstruction based on depth-first search) can be used to transform the fourth flow path into a directed acyclic graph, where nodes represent business steps, directed edges represent flow directions, and edge weights represent the normal flow probability of that step (calculated based on historical data). For example, the fourth data flow path extracted from the lineage information set could be "user registration → product browsing → order placement → payment → product delivery → confirmation of receipt". The above data flow path is constructed as a directed acyclic graph, where the nodes are user registration, product browsing, order placement, payment, product shipment, and confirmation of receipt. The directed edges and their weights are as follows: User registration → Product browsing (weight 0.9, 90% of registered users will browse products), Product browsing → Order placement (weight 0.3, 30% of browsing users will place an order), Order placement → Payment (weight 0.95, 95% of users will pay), Payment → Logistics shipment (weight 0.98, 98% of products will be shipped), Logistics shipment → Confirmation of receipt (weight 0.99, 99% of products will be received). The preset path weight is 0.25. It is found that one edge weight of "Product browsing → Order placement" is 0.3 > 0.25 (normal flow path), but the other edge weight of "Product browsing → Add to cart → Order placement" is 0.15 < 0.25 (abnormal flow path). This abnormal flow path may include abnormal characteristics such as shopping cart dwell time, product discount level, out-of-stock products, and product price increases. At this point, the "discount strength" can be quantified based on the feature reconstruction algorithm (e.g., 0-10%, 10%-30%, 30% and above), and the cross feature "discount strength × dwell time" can be constructed with "shopping cart dwell time" (<5 minutes, 5-15 minutes, >15 minutes respectively) to generate an optimized training feature set.

[0108] In the above method, by using the failure of the cross-institutional prediction model accuracy to reach the first budgeted percentage threshold as the judgment point, and employing a graph tracing algorithm to start from the fourth data flow path in the optimized lineage information set, abnormal flow paths and corresponding abnormal training features can be accurately located, avoiding the blindness of model optimization. This not only ensures the rationality of cross-institutional data flow but also effectively improves the prediction accuracy of the prediction model, enhancing the reliability of the prediction model in multi-institutional collaborative scenarios.

[0109] In one embodiment of this disclosure, after step S770: optimizing the abnormal training features based on the feature reconstruction algorithm to generate an optimized training feature set, the following steps are further included: Figure 8 As shown, the specific content is as follows:

[0110] Step S810: Obtain key features from the optimized training feature set to generate a simplified feature set;

[0111] Step S820: Based on the incremental learning algorithm, train the simplified feature set to generate the second initial model parameters;

[0112] Step S830: Update the cross-agency prediction model based on the second initial model parameters;

[0113] Step S840: Determine whether the prediction accuracy of the updated cross-institutional prediction model does not exceed the second preset percentage threshold;

[0114] Step S850: If it is determined that the second preset percentage threshold has not been exceeded, then computing resources are dynamically allocated to continuously optimize the second initial model parameters.

[0115] Specifically, the optimized training feature set is extracted, and three key features are selected: "discount strength × dwell time" (importance score 0.35), "logistics timeliness" (importance score 0.28), and "payment method" (importance score 0.22). The selection of key features is determined by their importance scores, with higher scores indicating greater importance, in order to generate a concise feature set. Based on the incremental learning algorithm, the simplified feature set is used to generate the second initial model parameters through three rounds of local iteration training. For example, w1=0.22 (corresponding to discount × stay), w2=0.18 (logistics timeliness), and w3=0.15 (payment method). The cross-institutional prediction model is updated accordingly, and its prediction accuracy is determined to be 87%, which does not exceed the second preset percentage threshold of 90%. Therefore, computing resources are dynamically allocated, increasing the number of GPU cores from 2 to 4, expanding the number of iterations from 10 rounds to 20 rounds, and fine-tuning the learning rate from 0.01 to 0.005. After continuous optimization, the second initial model parameters are updated, and the prediction model accuracy is improved to 95%, exceeding the second budget percentage threshold of 90%.

[0116] In the aforementioned method, extracting key features to generate a simplified feature set reduces the interference of redundant information on the cross-institutional prediction model, lowers computational complexity, and improves training efficiency. Furthermore, training and generating a second initial model parameter based on an incremental learning algorithm enables efficient updates while preserving the original model, avoiding resource waste from repeated training. Subsequently, updating the cross-institutional prediction model based on the updated model parameters allows for rapid iterative optimization of its performance, effectively enhancing its performance stability and practical value, and ensuring a more reliable predictive role in multi-institutional collaboration scenarios.

[0117] This disclosure also provides a multi-source heterogeneous data analysis system based on machine learning. The system may include a first acquisition module, a first determination module, a first generation module, a second generation module, a second determination module, a first judgment module, a second acquisition module, a third determination module, a repair module, a construction module, a second judgment module, and a third generation module. Specifically, the first acquisition module acquires data flow logs from a multi-source heterogeneous database. The data flow logs include timestamps, IP addresses of data sources, IDs of flow nodes, and the size of data packets. The first determination module determines a lineage information set based on the data flow logs. The lineage information set includes data sources, a first data flow path, and flow node dependencies. The first generation module uses a graph database to store the first data flow path and flow node dependencies to generate a graph database structure. The second generation module assigns a unique identifier to each flow node in the graph database structure based on a node label allocation algorithm to generate a node identifier set. The second determination module determines the dynamic relationships between each flow node based on the node identifier set using a graph traversal algorithm. The system comprises the following modules: a dynamic relation set; a first judgment module to determine whether inconsistent dependencies exist in the dynamic relation set; a second acquisition module to acquire inconsistent dependency fields if inconsistent dependencies are found; a third determination module to determine the set of erroneous fields based on the inconsistent dependency fields, including missing values, outliers, and inconsistent formats; a repair module to repair the set of erroneous fields using a data cleaning algorithm to generate a repaired data set; a construction module to construct a second data flow path based on the repaired data set; a second judgment module to determine whether the second data flow path has path integrity based on a path traversal algorithm; and a third generation module to generate an optimized lineage information set based on the second data flow path if path integrity is determined.

[0118] It should be noted that the embodiments of the machine learning-based multi-source heterogeneous data analysis system provided in this application can be used to execute the processing flow of the machine learning-based multi-source heterogeneous data analysis method embodiments described above. Its functions will not be repeated here, but can be referred to the detailed description of the above method embodiments.

[0119] As described above, the multi-source heterogeneous data analysis system based on machine learning provided in this disclosure analyzes and processes data flow logs in a multi-source heterogeneous database to accurately extract information such as timestamps, IP addresses of data sources, and IDs of flow nodes. Combined with a graph database storing the first flow path and node dependencies in the lineage information set, it achieves visualization and traceability of lineage information, making the data source and flow path clear and transparent. By using node label allocation algorithms and graph traversal algorithms, a dynamic set of relationships between each flow node is determined, enabling dynamic capture of relationships between flow nodes, timely detection of inconsistent dependencies, and location of inconsistent dependency fields, ensuring the accuracy of data association. Error fields are identified through inconsistent dependency fields, and data cleaning algorithms are used to repair these errors, generating a repaired data set. A second data flow path is constructed based on the repaired data set. After verifying the path integrity, an optimized lineage information set is generated. This effectively improves the accuracy of lineage information, enabling it to accurately capture real-time changes in data flow, thereby improving the accuracy of credit approval or risk assessment. It provides a high-quality and reliable foundation for subsequent cross-institutional data sharing and federated learning modeling, while enhancing the intelligence and automation of data governance, and helping to efficiently address the challenges of dynamic changes and complex relationships in a multi-source heterogeneous data environment.

[0120] This disclosure also provides an electronic device including one or more processors and memory resources represented by memory for storing instructions executable by the processor, such as application programs. The application programs stored in the memory may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor is configured to execute instructions to perform the aforementioned machine learning-based multi-source heterogeneous data analysis method.

[0121] The electronic device may also include a power supply component configured to perform power management of the electronic device, a wired or wireless network interface configured to connect the electronic device to a network, and an input / output (I / O) interface. The electronic device can be operated based on operating devices stored in memory, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0122] In one embodiment, a computer device, which may be a server, is also provided. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-source heterogeneous data analysis method based on machine learning.

[0123] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multi-source heterogeneous data analysis method based on machine learning. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0124] This disclosure also provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the aforementioned electronic device, the electronic device is able to execute a multi-source heterogeneous data analysis method based on machine learning, including: acquiring data flow logs from a multi-source heterogeneous database, the data flow logs including timestamps, IP addresses of data sources, IDs of flow nodes, and the size of data packets; determining a lineage information set based on the data flow logs, the lineage information set including data sources, a first data flow path, and flow node dependencies; using a graph database to store the first data flow path and flow node dependencies to generate a graph database structure; and assigning node labels to each flow node in the graph database structure based on a node label allocation algorithm. Each node is assigned a unique identifier to generate a node identifier set. Based on a graph traversal algorithm, a dynamic set of relationships between each flow node is determined according to the node identifier set. It is then determined whether inconsistent dependencies exist within the dynamic relationship set. If inconsistent dependencies exist, the inconsistent dependency fields are obtained. An error field set is determined based on the inconsistent dependency fields; error fields include missing values, outliers, and inconsistent formats. The error field set is repaired using a data cleaning algorithm to generate a repaired data set. A second data flow path is constructed based on the repaired data set. A path traversal algorithm is used to determine whether the second data flow path has path integrity. If path integrity is determined, an optimized lineage information set is generated based on the second data flow path.

[0125] This disclosure can take the form of a computer program product implemented on one or more storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0126] It should be noted that although the steps of the machine learning-based multi-source heterogeneous data analysis method of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps, such as omitting certain steps, combining multiple steps into one step, and / or decomposing a step into multiple steps, should all be considered part of this disclosure.

[0127] It should be understood that this disclosure is not limited to the detailed structure and arrangement of the modules in the machine learning-based multi-source heterogeneous data analysis system proposed in this specification. This disclosure can have other implementations and can be implemented and performed in various ways. The foregoing variations and modifications fall within the scope of this disclosure. It should be understood that this disclosure, as disclosed and defined in this specification, extends to all alternative combinations of two or more individual features mentioned or apparent in the text and / or drawings. All these different combinations constitute multiple alternative aspects of this disclosure. The embodiments described in this specification illustrate the best known mode for implementing this disclosure and will enable those skilled in the art to utilize this disclosure.

Claims

1. A method for analyzing multi-source heterogeneous data based on machine learning, characterized in that, The method comprises: acquiring a data flow transfer log in a multi-source heterogeneous database, the data flow transfer log comprising a timestamp, an IP address of a data source, an ID of a transfer node, and a size of a data packet; determining a blood relationship information set according to the data flow transfer log, the blood relationship information set comprising a data source, a first data flow transfer path, and a transfer node dependency relationship; storing the first data flow transfer path and the transfer node dependency relationship in a graph database to generate a graph database structure; assigning a corresponding unique identifier to each of the transfer nodes in the graph database structure based on a node label assignment algorithm to generate a node identifier set; determining a dynamic association relationship set between each of the transfer nodes based on the node identifier set according to a graph traversal algorithm; judging whether there is an inconsistent dependency relationship in the dynamic association relationship set; if it is judged that there is an inconsistent dependency relationship, acquiring an inconsistent dependency field; determining an error field set according to the inconsistent dependency field, the error field comprising a missing value, an abnormal value, and a format inconsistency; repairing the error field set based on a data cleaning algorithm to generate a repaired data set; constructing a second data flow transfer path according to the repaired data set; judging whether the second data flow transfer path has path integrity based on a path traversal algorithm; if it is judged that the path integrity is possessed, generating an optimized blood relationship information set according to the second data flow transfer path. 2.The machine learning based multi-source heterogeneous data analysis method of claim 1, wherein, The acquiring of the inconsistent dependency field if it is judged that there is an inconsistent dependency relationship comprises: acquiring an edge weight value between current transfer nodes, the edge weight value being used to represent a dependency strength between each of the transfer nodes; traversing the graph database structure based on a depth-first search algorithm to acquire a plurality of initial edge weight values; judging whether a difference between the current edge weight value and the initial edge weight value is greater than a first preset deviation threshold; if it is judged that the difference is greater than the first preset deviation threshold, it is judged that there is an inconsistent dependency relationship; taking any of the transfer nodes as a starting point, recording all third data flow transfer paths corresponding thereto, and calculating a path weight value corresponding to each of the third data flow transfer paths; judging whether a difference between the path weight value corresponding to each of the third data flow transfer paths and a path weight value corresponding to a historical data flow transfer path is greater than a second preset deviation threshold; if it is judged that the difference is greater than the second preset deviation threshold, the deviation transfer path is marked; determining the inconsistent dependency field according to the deviation transfer path. 3.The machine learning based multi-source heterogeneous data analysis method of claim 1, wherein, The repairing of the error field set based on the data cleaning algorithm to generate a repaired data set comprises: performing standardization processing on the error field set to generate a first standardized data set; calculating a covariance matrix of the first standardized data set to determine eigenvalues and eigenvectors of the covariance matrix; sorting the eigenvalues in order from a maximum value to a minimum value, and selecting the eigenvectors corresponding to the first five eigenvalues in the sorting result to generate a feature matrix; determining a corresponding repair scheme according to the type of the error field in the feature matrix. 4.The machine learning based multi-source heterogeneous data analysis method of claim 3, wherein, The determining of the corresponding repair scheme according to the type of the error field in the feature matrix comprises: When the feature matrix has the missing values, based on a K-neighbor interpolation algorithm, a plurality of neighbor values corresponding to the missing values are obtained; The plurality of neighbor values are subjected to a weighted average operation, and the operation result is taken as the missing value; When the feature matrix has an abnormal value, a standard score of the abnormal value is calculated; It is judged whether the standard score of the abnormal value exceeds a preset standard score threshold; If it is judged that the preset standard score threshold is exceeded, the corresponding abnormal value is removed; When the feature matrix has inconsistent formats, the inconsistent formats are adjusted to uniform formats by using a regular expression. 5.The machine learning based multi-source heterogeneous data analysis method of claim 1, wherein, After the blood relationship information set is generated based on the second data flow transfer path, the method further includes: The optimized blood relationship information set is subjected to standardization processing to generate a second standardized data set; Based on a differential privacy algorithm, a privacy budget of the second standardized data set is calculated; It is judged whether the privacy budget exceeds a preset privacy budget threshold; If it is judged that the preset privacy budget threshold is not exceeded, Gaussian noise is added to the second standardized data set to generate a perturbed data set; Based on a federated learning framework, local model parameters of each institution are fused in the perturbed data set to generate a shared model parameter set; It is judged whether the shared model parameter set passes the consistency verification of each institution; If it is judged that the consistency verification of each institution is satisfied, a secure shared data set is generated. 6.The machine learning based multi-source heterogeneous data analysis method of claim 5, wherein, After the secure shared data set is generated if the consistency verification of each institution is satisfied, the method further includes: A training feature set is determined based on the secure shared data set; Based on a distributed gradient descent algorithm, a first initial model parameter is determined based on the training feature set; The first initial model parameter is sent to each institution; Based on local data, each institution calculates a corresponding local gradient update value; The local gradient update values are aggregated to generate a global gradient update value; The first initial model parameter is updated based on the global gradient update value to generate a final model parameter; A cross-institution prediction model is generated based on the final model parameter.

7. The machine learning based multi-source heterogeneous data analysis method according to claim 6, characterized in that, After the cross-institution prediction model is generated based on the final model parameter, the method further includes: It is judged whether the prediction accuracy of the cross-institution prediction model does not exceed a first preset percentage threshold; If it is judged that the first preset percentage threshold is not exceeded, a fourth data flow transfer path in the optimized blood relationship information set is obtained; Based on a graph-based tracing algorithm, a directed acyclic graph is constructed based on the fourth data flow transfer path; It is judged whether an edge weight value of the directed acyclic graph is greater than a preset path weight value; If it is judged that the preset path weight value is greater, it is determined to be an abnormal transfer path; An abnormal training feature is determined based on the abnormal transfer path; Based on a feature reconstruction algorithm, the abnormal training feature is optimized to generate an optimized training feature set. 8.The machine learning based multi-source heterogeneous data analysis method of claim 7, wherein, After the abnormal training feature is optimized based on the feature reconstruction algorithm to generate the optimized training feature set, the method further includes: Key features in the optimized training feature set are obtained to generate a reduced feature set; train the reduced feature set based on an incremental learning algorithm to generate second initial model parameters; update the cross-institution prediction model according to the second initial model parameters; determine whether a prediction accuracy of the updated cross-institution prediction model does not exceed a second preset percentage threshold; if it is determined that the second preset percentage threshold is not exceeded, dynamically allocate computing resources to continuously optimize the second initial model parameters. 9.A machine learning based multi-source heterogeneous data analysis system, characterized in that, comprise: a first acquisition module configured to acquire data flow transfer logs in a multi-source heterogeneous database, the data flow transfer logs comprising a timestamp, an IP address of a data source, an ID of a transfer node, and a size of a data packet; a first determination module configured to determine a blood relationship information set according to the data flow transfer logs, the blood relationship information set comprising a data source, a first data flow transfer path, and a transfer node dependency relationship; a first generation module configured to store the first data flow transfer path and the transfer node dependency relationship in a graph database to generate a graph database structure; a second generation module configured to assign each of the transfer nodes in the graph database structure with a corresponding unique identifier based on a node label assignment algorithm to generate a node identifier set; a second determination module configured to determine a dynamic association relationship set between each of the transfer nodes based on a graph traversal algorithm according to the node identifier set; a first judgment module configured to determine whether there is an inconsistent dependency relationship in the dynamic association relationship set; a second acquisition module configured to acquire an inconsistent dependency field if it is determined that the inconsistent dependency relationship exists; a third determination module configured to determine an error field set according to the inconsistent dependency field, the error field comprising a missing value, an abnormal value, and a format inconsistency; a repair module configured to repair the error field set based on a data cleaning algorithm to generate a repaired data set; a construction module configured to construct a second data flow transfer path according to the repaired data set; a second judgment module configured to determine whether the second data flow transfer path has path integrity based on a path traversal algorithm; a third generation module configured to generate an optimized blood relationship information set according to the second data flow transfer path if it is determined that the path integrity is possessed.

10. An electronic device, comprising: comprise: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

Images