A multi-source data management method and device based on a federal mechanism

Abstract

The application discloses a kind of multi-source data management method and device based on federal mechanism, belong to computer data processing technical field, it includes the local data of each data source node is collected and extracts semantic summary, is formed distributed data source register by consensus verification;Cross-source collaborative analysis request is received, and is parsed as non-cyclic directed task graph;According to task graph, query is executed locally in each node, and collaborative calculation is completed by encrypted shared intermediate result, generates final analysis result and task execution log;Analysis task execution log is generated for each node Dynamic credit score;According to credit score, update node weight to adaptively optimize the node selection of subsequent task.The application adopts the closed-loop management strategy that multi-source heterogeneous data is semantically abstracted, collaborative analysis request is parsed as execution task graph, dynamic credit score is generated in combination with task execution log and is adaptively optimized according to node selection, and cross-domain data collaborative analysis and management of safe, efficient and intelligent can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a multi-source data management method and apparatus based on a federated mechanism. Background Technology

[0002] With the development of information technology, data has become a key production factor, widely distributed across the business systems of different institutions and departments, forming numerous data silos. To fully unlock the value of data, cross-domain and cross-institutional multi-source data collaborative analysis has become crucial. Federated mechanisms, as an emerging distributed collaborative computing paradigm, allow parties to conduct joint computations without sharing their local raw data, providing an effective technical path to solve data silos and ensure data security.

[0003] In existing technologies, collaborative analysis of multi-source data typically employs data centralization or API calls. Data centralization requires aggregating data from various sources to a central node for processing, which not only imposes significant storage and bandwidth demands but also poses serious risks of data leakage and privacy. To circumvent these issues, some solutions have begun to adopt federated learning frameworks, using model parameters rather than raw data for joint modeling. However, these frameworks are primarily designed for machine learning scenarios and lack sufficient support for more general data querying and management tasks, and they generally assume that all participating nodes are trustworthy and efficient.

[0004] In practical applications, existing technical solutions have significant shortcomings. First, the data formats, structures, and semantics of various data sources differ greatly, i.e., data heterogeneity, which poses a significant challenge to unified querying and analysis across sources. Second, in federated collaborative networks, the network conditions, computing power, and data quality of each node vary and change dynamically. Existing solutions generally lack dynamic assessment and management mechanisms for the reliability of node collaboration, easily leading to inefficient or even failed collaborative tasks. Finally, for complex analysis tasks, how to automatically plan the optimal distributed execution path and adaptively adjust it based on actual execution conditions is also a pressing problem that current technology needs to solve. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multi-source data management method and apparatus based on a federated mechanism. This method employs a closed-loop management strategy that semantically abstracts heterogeneous multi-source data, parses collaborative analysis requests into execution task graphs, generates dynamic reputation scores based on task execution logs, and adaptively optimizes node selection accordingly. This enables secure, efficient, and intelligent cross-domain data collaborative analysis and management.

[0006] The above objectives can be achieved through the following approach:

[0007] A multi-source data management method based on a federated mechanism includes: collecting local multi-source heterogeneous data from each data source node; performing knowledge distillation to extract semantic summary data representing the data structure and relationships; encapsulating the semantic summary data and the network address of the data source node into a node registration transaction, performing consensus verification and recording to form a distributed data source registry; receiving cross-source collaborative analysis requests input by users; parsing the execution logic of the cross-source collaborative analysis request based on the semantic summary data to generate an acyclic directed task graph; performing local execution query operations on the corresponding execution data source node according to the topological order of the acyclic directed task graph, and encrypting and securely sharing the intermediate results generated during the execution process to complete collaborative computation, generating a final analysis result and a task execution log; analyzing the response time, data completeness, and consistency of intermediate results of each execution data source node recorded in the task execution log, and calculating and generating a dynamic reputation score for each participating execution data source node based on the final analysis result; updating the weights of each data source node in the distributed data source registry based on the dynamic reputation score, and selecting data source nodes to participate in collaboration according to their weights when parsing new cross-source collaborative analysis requests in the future, thereby achieving adaptive optimization of federated data management.

[0008] Optionally, forming the distributed data source registry includes: cleaning and standardizing local multi-source heterogeneous data, mapping different types of data uniformly into structured feature vectors, and generating a standardized feature vector set; processing the standardized feature vector set, projecting it into a low-dimensional semantic space, hiding the original numerical details while preserving the data distribution characteristics, and generating semantic summary data; constructing a standardized node registration transaction with the semantic summary data and the network address of the data source node, verifying the integrity and format of the node registration transaction, and forming a distributed data source registry.

[0009] Optionally, generating the acyclic directed task graph includes: performing semantic parsing on the cross-source collaborative analysis request to extract query intent entities and operation logic predicates; retrieving the distributed data source registry, filtering execution data source nodes based on the matching degree between the query intent entities and the semantic summary data, generating corresponding atomic query operations for the execution data source nodes based on the operation logic predicates; analyzing the data input-output dependencies between the atomic query operations, constructing data flow paths as directed edges, using the atomic query operations as nodes, combining and outputting the acyclic directed task graph.

[0010] Optionally, the query intent entity includes: a core concept identifier and a context constraint descriptor, wherein: the core concept identifier is used to characterize the subject or attribute of the analysis target and provides a core index for matching calculations with the semantic summary data; the context constraint descriptor is used to limit the time, space or other related conditions of the data and serves as a filtering basis when screening the execution data source nodes.

[0011] Optionally, generating the final analysis result and task execution log includes: identifying the initial atomic query operation in the acyclic directed task graph and distributing the initial atomic query operation to the corresponding execution data source node to start local computation; after completing the local computation, homomorphic encryption is performed to encrypt the generated intermediate result data, and according to the data flow path defined in the acyclic directed task graph, it is securely transmitted to the next atomic query operation as execution input; when all atomic query operations in the acyclic directed task graph have been executed, the aggregated computation output is used as the final analysis result, and the identity identifier, start and end time, and status information during the execution process are collected and compiled to generate the task execution log.

[0012] Optionally, the method further includes: comparing and analyzing the planned execution path in the acyclic directed task graph with the actual execution data in the task execution log, quantifying the deviation in execution latency and resource consumption, and generating an execution deviation map.

[0013] Optionally, calculating and generating a dynamic reputation score for each participating data source node includes: extracting the response time, data completeness, and intermediate result consistency of each execution data source node from the task execution log, and performing normalization processing to generate a response time score, a data completeness score, and a result consistency score; analyzing the influence of the execution data source node on the final analysis result, and generating a result contribution score through feature importance analysis; and performing a weighted summation of the response time score, the data completeness score, the result consistency score, and the result contribution score, and updating it with time decay to generate a dynamic reputation score.

[0014] Optionally, the step of generating a result contribution score through feature importance analysis includes: for each execution data source node, removing the intermediate result data of the execution data source node from the task execution log, simulating and recalculating the collaborative computing process to generate a perturbed analysis result lacking the execution data source node; measuring the difference between the final analysis result and the perturbed analysis result, quantifying the magnitude of the result change, and generating a result contribution score.

[0015] Optionally, the adaptive optimization for implementing federated data management includes: updating the basic weights of nodes in the distributed data source registry based on the dynamic reputation scores of the execution data source nodes; and, when parsing new cross-source collaborative analysis requests, dynamically reducing the node selection priority of the corresponding deviation characteristics based on the basic weights and the performance bottlenecks identified by the execution deviation map, and performing weight and priority matching to combine nodes to participate in collaboration.

[0016] Based on the same inventive concept, this invention also provides a multi-source data management device based on a federated mechanism. The device includes: a federated node registration and data abstraction module, used to collect local multi-source heterogeneous data from each data source node, perform knowledge distillation to extract semantic summary data representing the data structure and relationships, encapsulate the semantic summary data and the network address of the data source node into a node registration transaction, perform consensus verification and record it, forming a distributed data source registry; a collaborative task planning module, used to receive cross-source collaborative analysis requests input by the user, parse the execution logic of the cross-source collaborative analysis request according to the semantic summary data, and generate an acyclic directed task graph; and a federated collaborative execution module, used to execute tasks according to the topological order of the acyclic directed task graph. Data source nodes perform local query operations and encrypt and securely share intermediate results generated during execution to complete collaborative computation, generating final analysis results and task execution logs. A dynamic reputation assessment module analyzes the response time, data completeness, and consistency of intermediate results for each data source node recorded in the task execution logs. Based on the final analysis results, it calculates and generates dynamic reputation scores for each participating data source node. An adaptive optimization and strategy adjustment module updates the weights of each data source node in the distributed data source registry based on the dynamic reputation scores. When parsing new cross-source collaborative analysis requests, it selects data source nodes to participate in collaboration based on their weights, achieving adaptive optimization of federated data management.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] 1. By constructing a closed-loop management framework integrating data abstraction, task planning, collaborative execution, reputation assessment, and adaptive optimization, efficient and intelligent cross-source data collaborative analysis is achieved while ensuring data sovereignty and privacy security. By extracting heterogeneous data into lightweight semantic summary data and performing distributed registration, the interoperability problem of multi-source data is effectively solved. Simultaneously, combined with federated execution and intermediate result encryption technologies, the original data is ensured to remain within the domain, guaranteeing the security of the entire data flow process.

[0019] 2. The introduced dynamic reputation evaluation mechanism can quantify and dynamically update the reliability of collaboration based on the actual performance and contribution of nodes. This mechanism can accurately identify and prioritize high-quality collaborating partners, avoiding the impact of poor performance or low data quality of individual nodes on the efficiency and accuracy of the overall analysis, thereby improving the robustness of the entire federated data management system and the credibility of the analysis results;

[0020] 3. By combining dynamic reputation scores with task execution deviation maps, adaptive optimization of federated data management is achieved. Instead of statically selecting data sources, the node selection strategy can be dynamically adjusted based on historical reputation and current task performance bottlenecks to achieve optimal resource allocation. This self-learning and continuous evolution capability enables it to adapt to complex and ever-changing network environments and node states, maintaining long-term, efficient, and stable operation.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a multi-source data management method based on a federation mechanism according to an embodiment of the present invention.

[0024] Figure 2 This is a dynamic reputation score evolution diagram of the data source node in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the adaptive node selection strategy according to an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of a multi-source data management device based on a federation mechanism according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1 One embodiment of the present invention proposes a multi-source data management method based on a federation mechanism. It adopts a closed-loop management strategy that performs semantic abstraction on multi-source heterogeneous data, parses collaborative analysis requests into execution task graphs, generates dynamic reputation scores based on task execution logs, and adaptively optimizes node selection accordingly. This enables secure, efficient, and intelligent cross-domain data collaborative analysis and management.

[0029] The method described in this embodiment specifically includes:

[0030] Collect local multi-source heterogeneous data from each data source node, perform knowledge distillation to extract semantic summary data representing the data structure and relationships, encapsulate the semantic summary data and the network address of the data source node into a node registration transaction, perform consensus verification and record it to form a distributed data source registry.

[0031] Receive a cross-source collaborative analysis request input by the user, parse the execution logic of the cross-source collaborative analysis request according to the semantic summary data, and generate an acyclic directed task graph;

[0032] Based on the topological order of the acyclic directed task graph, local execution query operations are performed on the corresponding execution data source nodes, and intermediate results generated during the execution process are encrypted and securely shared to complete collaborative computation, generating final analysis results and task execution logs.

[0033] The response time, data completeness, and consistency of intermediate results of each execution data source node recorded in the task execution log are analyzed. Based on the final analysis results, a dynamic reputation score is calculated and generated for each participating execution data source node.

[0034] Based on the dynamic reputation score, the weights of each data source node in the distributed data source registry are updated. When parsing new cross-source collaborative analysis requests in the future, the data source node is selected to participate in the collaboration according to the weight, thereby achieving adaptive optimization of federated data management.

[0035] By adopting a closed-loop management strategy that semantically abstracts multi-source heterogeneous data, parses collaborative analysis requests into execution task graphs, generates dynamic reputation scores based on task execution logs, and adaptively optimizes node selection accordingly, secure, efficient, and intelligent cross-domain data collaborative analysis and management can be achieved.

[0036] Optionally, the formation of the distributed data source registry includes:

[0037] Local multi-source heterogeneous data is cleaned and standardized, and different types of data are uniformly mapped into structured feature vectors to generate a standardized feature vector set;

[0038] Specifically, on each data source node, preprocessing is first performed on the multi-source heterogeneous data. This includes data cleaning steps to handle missing and outlier values, and data standardization steps. For example, for numerical data, min-max normalization can be used to map it to a uniform numerical range; for unstructured data such as text, word embedding technology is used to convert it into a high-dimensional feature vector; for image data, deep features are obtained through feature extraction components to generate image feature vectors. Through this preprocessing operation, data from diverse sources and in different formats are uniformly converted into a structured and dimensionally consistent set of standardized feature vectors.

[0039] The standardized feature vector set is processed and projected into a low-dimensional semantic space to hide the original numerical details while preserving the data distribution characteristics, thereby generating semantic summary data.

[0040] Specifically, to extract core semantic information from the original data while protecting its privacy, the standardized feature vector set is dimensionality reduced. By training with the optimization objective of minimizing the difference between the reconstructed data and the original data, the high-dimensional standardized feature vector set can be projected into a low-dimensional semantic space. This process preserves the relative distribution and clustering characteristics of data points—that is, the inherent relationships between data points—while obscuring the original precise numerical values, thereby generating semantic summary data that can characterize the core features of the data.

[0041] The semantic summary data and the network address of the data source node are used to construct a standardized node registration transaction. The integrity and format of the node registration transaction are verified to form a distributed data source registry.

[0042] Specifically, the semantic summary data generated in the aforementioned steps is encapsulated with the network address of the data source node that identifies its location, constructing a standardized node registration transaction with a unified data structure. This transaction is broadcast to the federated network, where consensus nodes verify the data integrity and format compliance of the node registration transaction according to a predetermined consensus protocol. Upon successful verification, the transaction is recorded on the distributed ledger. As various data source nodes successively complete their registration, a decentralized, tamper-proof distributed data source registry is ultimately formed.

[0043] Optionally, generating the acyclic directed task graph includes:

[0044] The cross-source collaborative analysis request is semantically parsed to extract the query intent entity and operation logic predicate;

[0045] Specifically, natural language processing (NLP) technology is used to perform in-depth processing on user-submitted cross-source collaborative analysis requests. The purpose of this processing is to accurately identify and extract two types of core information: the first type is the query intent entity, which represents the core target or data object of the analysis, such as a specific disease name or a geographical location; the second type is the operation logic predicate, which defines the calculations or logical operations that need to be performed on the query intent entity, such as calculating the average, counting, or performing association filtering.

[0046] The distributed data source registry is retrieved, and execution data source nodes are selected based on the matching degree between the query intent entity and the semantic summary data. Based on the operation logic predicate, corresponding atomic query operations are generated for the execution data source nodes.

[0047] Specifically, after extracting the query intent entity and operation logic predicates, the existing distributed data source registry is retrieved. For each query intent entity, the relevance (i.e., matching degree) of the data source node to the query intent is quantified by calculating the similarity between its feature vector and the semantic summary data vectors of each data source node in the registry. Based on this matching degree, all nodes with high relevance are selected to form execution data source nodes. Subsequently, based on the operation logic predicates extracted in the previous step, the macro-analysis task is decomposed, generating specific atomic query operations that can be executed locally for each selected execution data source node. For example, a cross-institutional "average age calculation" request will be decomposed into multiple atomic query operations for "local age summation and headcount statistics" executed on each institution node.

[0048] Analyze the data input-output dependencies between the atomic query operations, construct data flow paths as directed edges, treat the atomic query operations as nodes, and combine and output an acyclic directed task graph.

[0049] Specifically, dependency analysis is performed on all generated atomic query operations to identify the data input and output relationships between them, i.e., to determine which operations must wait for the results of other operations. Based on these dependencies, data flow paths are constructed as directed edges in the graph, and each atomic query operation is treated as a node in the graph. By combining all nodes and directed edges, a non-cyclic directed task graph that ensures no circular dependencies and has clear execution logic is finally output. This graph provides a distributed execution blueprint for subsequent federated collaborative execution.

[0050] Optionally, the query intent entity includes: a core concept identifier and a context constraint descriptor, wherein:

[0051] The core concept identifier is used to characterize the subject or attribute of the analysis target and provides a core index for matching and calculating with the semantic summary data;

[0052] Specifically, a core concept identifier (CPI) aims to capture the main content of a query from a user's original analysis request, such as a specific drug name in medical analysis. Technically, a CPI is converted into a standardized feature vector, which serves as a core index for calculating the match between the CPI and semantic summary data recorded in a distributed data source registry. The goal is to initially filter out candidate nodes related to the content topic from numerous data sources.

[0053] The context constraint descriptor is used to limit the time, space or other related conditions of the data, and serves as the filtering basis when selecting the execution data source node.

[0054] Specifically, a context constraint descriptor is used to precisely define the boundary conditions of the data. It extracts deterministic limiting information from the user request, such as a time range of "first quarter of 2024" or a spatial range of "East China". After a core concept identifier filters out an initial set of candidate nodes, the filtering rules provided by a context constraint descriptor are used to perform a secondary verification on the candidate node set, retaining only nodes that fully satisfy all boundary conditions as the final execution data source nodes.

[0055] Optionally, generating the final analysis results and task execution log includes:

[0056] Identify the initial atomic query operation in the acyclic directed task graph, and distribute the initial atomic query operation to the corresponding execution data source node to start local computation;

[0057] Specifically, the federated collaborative execution process is scheduled based on the generated acyclic directed task graph. First, the topology of the task graph is parsed to identify all nodes with an in-degree of zero; these nodes represent the initial atomic query operations. Then, these initial atomic query operations are distributed to the corresponding execution data source nodes specified in the task graph, thereby triggering these nodes to initiate computation processes on their local data.

[0058] After completing local computation, homomorphic encryption is used to encrypt the generated intermediate result data, and according to the data flow path defined by the acyclic directed task graph, it is securely transmitted to the next atomic query operation as the execution input.

[0059] Specifically, after an execution data source node completes its local computation, it generates intermediate result data. To ensure the confidentiality of the data during cross-node transfer, this intermediate result data is immediately subjected to homomorphic encryption. Homomorphic encryption allows specific mathematical operations to be performed directly on the ciphertext, and the decrypted result is identical to the result of performing the same operation on the plaintext. After encryption, the encrypted intermediate result data follows a data flow path defined by directed edges in an acyclic directed task graph, and is transmitted to the execution data source node where the next atomic query operation is located as its execution input.

[0060] Once all atomic query operations in the acyclic directed task graph have been completed, the aggregated computation output serves as the final analysis result. It also gathers the identity identifiers, start and end times, and status information during the execution process and compiles them to generate a task execution log.

[0061] Specifically, after all atomic query operations in a non-cyclic directed task graph have been executed, the process enters the result aggregation and log generation phase. All final computational outputs are aggregated for final computation or decryption, producing user-readable final analysis results. Simultaneously, throughout the entire execution process, metadata for each atomic query operation is continuously recorded, including the identity of the execution node, the start and end times of the operation, and the execution status information. After the final analysis results are generated, all this metadata is collected and organized, ultimately compiling into a structured task execution log containing complete execution process traceability information.

[0062] Optionally, the method further includes:

[0063] The planned execution path in the acyclic directed task graph is compared and analyzed with the actual execution data in the task execution log to quantify the deviation in execution latency and resource consumption, and generate an execution deviation map.

[0064] Specifically, this method is a process of quantitatively analyzing the post-event performance of collaborative tasks. The core inputs to this process are an acyclic directed task graph and a task execution log. The acyclic directed task graph contains planned execution path parameters, while the task execution log records the actual execution data for each atomic query operation. The analysis process first traverses each node and edge in the acyclic directed task graph and aligns the corresponding execution records from the task execution log. Next, the aligned planned values ​​are compared with the actual values, and the difference between the actual execution time and the planned time for each atomic query operation is calculated as the execution latency deviation. The difference between actual resource usage and estimated resource consumption is also quantified as the resource consumption deviation. After all deviation calculations are completed, these quantified deviation information are integrated to finally generate an execution deviation graph. This graph structurally reuses the topology of the acyclic directed task graph and labels the corresponding deviation information on its nodes and edges to visually indicate performance bottlenecks.

[0065] Optionally, calculating and generating dynamic reputation scores for each participating data source node includes:

[0066] The response time, data completeness, and intermediate result consistency of each execution data source node are extracted from the task execution log and normalized to generate response time score, data completeness score, and result consistency score.

[0067] Specifically, the process begins with a deep analysis of a task execution log, extracting three key raw performance metrics for each execution data source node: the first is response time, which is the total time a node takes from receiving an atomic query operation to returning intermediate result data; the second is data completeness, quantified by comparing the intermediate result data returned by the node with the expected format and fields; and the third is intermediate result consistency, assessed by cross-validating the node's results with those of other nodes in related calculations. Since the raw performance metrics have different dimensions and value ranges, they need to be normalized, for example, by using minimum-maximum scaling to uniformly map them to the range of 0 to 1, thereby generating standardized response time scores, data completeness scores, and result consistency scores.

[0068] The influence of the execution data source node on the final analysis result is analyzed, and a result contribution score is generated through feature importance analysis.

[0069] Specifically, this step aims to evaluate a node's contribution to the final value of the task, rather than merely its performance during execution. This feature importance analysis simulates a collaborative computation process lacking intermediate data for that node, yielding a perturbed analysis result. The node's contribution is determined by measuring the difference between this result and the original final analysis result. This quantified contribution value is also normalized to generate a result contribution score.

[0070] The response time score, data completeness score, result consistency score, and result contribution score are weighted and summed, and then updated with time decay to generate a dynamic reputation score.

[0071] Specifically, after obtaining the scores from the four dimensions mentioned above, they are combined into a comprehensive real-time performance score through a weighted summation. The weighting coefficients of each score can be configured according to the business scenario. To ensure that the reputation score reflects the long-term performance trend of a node and is more sensitive to recent behavior, a time decay method is used to update the node's dynamic reputation score. This update process utilizes an update factor to combine the currently calculated real-time performance score with the node's previous historical reputation score, thereby generating a comprehensive, dynamic reputation score that reflects its true collaborative value, such as... Figure 2 As shown, this depicts the long-term trend of how the reputation scores of different nodes are dynamically adjusted based on their respective performance after experiencing multiple collaborative tasks.

[0072] Optionally, the contribution score generated through feature importance analysis includes:

[0073] For each execution data source node, the intermediate result data of the execution data source node is removed from the task execution log, and the collaborative computing process is simulated and recalculated to generate a perturbation analysis result with the execution data source node missing.

[0074] Specifically, this method is an attribution analysis technique based on perturbation simulation. The process is iterative, evaluating the contribution of each execution data source node participating in the collaborative computation. For a specific target execution data source node, the first step of the analysis is to locate and logically remove all intermediate result data generated by that node from a task execution log that records the complete collaborative computation process. After removing the intermediate result data of the target node, a simulation recalculation of the original collaborative computation process is initiated. Subsequent atomic query operations that depend on the removed intermediate result data are adjusted accordingly, ultimately calculating a new result due to the lack of target node data—a perturbation-based analysis result.

[0075] The final analysis results are compared with the perturbation analysis results to measure the difference, quantify the magnitude of change in the results, and generate a result contribution score.

[0076] Specifically, the difference between the original final analysis results generated by normal collaboration of all nodes and the newly generated perturbed analysis results is measured to quantify the magnitude of change in the final result due to the absence of the target node. The method of difference measurement depends on the data type of the analysis results. For example, if the results are numerical, the absolute difference between the two is calculated; if the results are categorical labels, it is determined whether the labels have changed. This measurement process yields a numerical value that quantifies the magnitude of the change in the result. Finally, this value is normalized to map it to a standardized scoring range, ultimately generating a result contribution score that represents the magnitude of the contribution of the execution data source node.

[0077] Optionally, the adaptive optimization for implementing federated data management includes:

[0078] Based on the dynamic reputation score of the execution data source node, update the basic weight of the node in the distributed data source registry;

[0079] Specifically, this step is the first step in the dynamic closed-loop decision-making process. After a collaborative task is completed, a distributed data source registry is updated in real time based on a dynamic reputation score calculated for each executing data source node. This update process directly maps each node's dynamic reputation score to its base weight in the registry. This base weight reflects the node's long-term, comprehensive historical performance and value contribution, serving as the benchmark for subsequent node selection decisions.

[0080] When parsing new cross-source collaborative analysis requests, based on the basic weights and the performance bottlenecks identified by the execution deviation map, the node selection priority of the corresponding deviation features is dynamically reduced, and weight and priority matching is performed to combine nodes to participate in collaboration.

[0081] Specifically, when a new cross-source collaborative analysis request is received, the node selection process considers not only the nodes' base weights but also a dynamic decision factor: an execution bias graph generated from previous tasks. First, the characteristics of the new request are analyzed, and historical performance bottlenecks related to these characteristics are identified from the execution bias graph. Then, based on the nodes' base weights and the identified performance bottlenecks, the node selection priority is dynamically adjusted. This process is achieved through the calculation of a comprehensive selection score.

[0082] ,

[0083] in, The final comprehensive selection score for the node is the basis for the final decision on node selection in this process; The basic weight of a node reflects its long-term overall performance, and its value comes from the distributed data source registry. This is a performance adjustment factor, determined based on performance bottlenecks identified in an execution deviation graph. Its value ranges from 0 to 1. For nodes identified as performance bottlenecks within the current task type, the factor value is less than 1; for nodes with no relevant bottleneck records, the factor value is 1. Finally, the node combination with the highest overall selection score is chosen to participate in the collaboration, completing an intelligent node selection process that matches weights and priorities. Figure 3 As shown, this demonstrates how to select the optimal combination of collaborative nodes by comprehensively considering three dimensions: the node's dynamic reputation score, task-related performance, and data contribution.

[0084] Based on the same inventive concept, the present invention also provides a multi-source data management device based on a federation mechanism, such as... Figure 4 As shown, the device includes:

[0085] The federated node registration and data abstraction module is used to collect local multi-source heterogeneous data from each data source node, perform knowledge distillation to extract semantic summary data that represents the data structure and relationships, encapsulate the semantic summary data and the network address of the data source node into a node registration transaction, perform consensus verification and record it to form a distributed data source registry.

[0086] The collaborative task planning module is used to receive cross-source collaborative analysis requests input by the user, parse the execution logic of the cross-source collaborative analysis requests according to the semantic summary data, and generate an acyclic directed task graph.

[0087] The federated collaborative execution module is used to perform local execution query operations on the corresponding execution data source nodes according to the topological order of the acyclic directed task graph, and to encrypt and securely share the intermediate results generated during the execution process to complete collaborative computing, and generate the final analysis results and task execution logs.

[0088] The dynamic reputation assessment module is used to analyze the response time, data completeness, and consistency of intermediate results of each execution data source node recorded in the task execution log, and calculate and generate dynamic reputation scores for each participating execution data source node in combination with the final analysis results.

[0089] The adaptive optimization and strategy adjustment module is used to update the weights of each data source node in the distributed data source registry based on the dynamic reputation score. When parsing new cross-source collaborative analysis requests in the future, the module selects data source nodes to participate in the collaboration based on their weights, thereby achieving adaptive optimization of federated data management.

[0090] To verify the feasibility and effectiveness of this invention in practice, it was applied to a cross-regional financial risk assessment platform. This platform comprises a federation of three independent financial institutions: Data Node A, Data Node B, and Data Node C, aiming to jointly conduct a structural risk analysis of cross-market investment portfolios. Each institution holds a large amount of heterogeneous data from multiple sources, including transaction history, market sentiment texts, and alternative data across different asset classes. However, due to regulatory requirements and trade secrets, they cannot directly share the raw data.

[0091] To address the aforementioned issues, the platform deploys the method and apparatus of this invention. First, each data node processes its local data to form a distributed data source registry. Taking data node A as an example, it first cleans and standardizes its local transaction data, standardizing numerical data of different formats using the Z-score method; and converts unstructured market analysis reports into feature vectors using word embedding technology. These processes uniformly transform the diverse data into a high-dimensional set of standardized feature vectors. Subsequently, the high-dimensional vector set is reduced in dimensionality using a data compression function, generating semantic summary data that retains the inherent correlation of the data while hiding the original transaction details. Finally, this semantic summary data and data node A's network address are encapsulated into a standardized node registration transaction, which is broadcast to the federated network. After consensus verification by other nodes, the transaction is recorded on the distributed ledger. Data nodes B and C also perform the same process, ultimately forming a decentralized distributed data source registry.

[0092] When a risk analyst submits a cross-source collaborative analysis request on the platform: "Analyze the asset correlation between the 'upstream raw materials' and 'downstream vehicle manufacturing' sectors in the 'new energy vehicle industry chain' during the first quarter of 2024," an acyclic directed task graph is generated. First, the request is semantically parsed to extract the query intent entity and operational logic predicates. The core concept identifier is identified as "new energy industry chain asset correlation," and the context constraint descriptors are "Time: First Quarter of 2024" and "Sectors: Upstream Raw Materials, Downstream Vehicle Manufacturing." The feature vector of the core concept identifier is used to calculate similarity with the semantic summary data of each node in the registry. The calculation results show that the data of data nodes A and C are highly correlated with this topic. Next, the context constraint descriptors are used for filtering, confirming that nodes A and C both hold data for the specified time period and sectors, thus selecting them as the execution data source nodes. Subsequently, based on the operational logic predicate of "calculating correlation," the task is decomposed into multiple atomic query operations, and a clear acyclic directed task graph is constructed based on data dependencies.

[0093] During the federated collaborative execution phase, based on the task graph, the initial atomic query operations are distributed to data nodes A and C. Both institutions perform computations on their respective local data, and upon completion, homomorphically encrypt the resulting intermediate data. The encrypted data is then securely transmitted to the aggregation node specified in the task graph. The aggregation node, without decryption, directly performs calculations on the ciphertext to obtain the final encrypted correlation coefficient, and finally decrypts it to produce the final analysis result. Throughout the process, detailed metadata such as the execution node, start and end times, and execution status of each operation is recorded, and compiled into a complete task execution log after the task is completed.

[0094] After the task is completed, the adaptive optimization process begins. First, the planned execution path in the acyclic directed task graph is compared with the actual execution data in the task execution log to generate an execution deviation graph. The graph shows that although data node C contributes a lot of data, its computational resource consumption is 40% higher than expected, constituting a potential performance bottleneck.

[0095] Next, dynamic reputation scores were calculated for data nodes A and C. By parsing task logs, standardized scores were generated for both nodes in terms of response time and data completeness. Furthermore, feature importance analysis was used to assess their contribution: simulating a recalculation after removing data from node C revealed a significant decrease in the confidence level of the final analysis results, proving that node C provided crucial data and thus received a higher contribution score. After weighted summation and time decay updates of all scores, node A's dynamic reputation score increased due to its efficient performance, while node C's score, although affected by resource consumption, remained at a high level due to its high contribution.

[0096] In a subsequent stress test task with even higher computational resource requirements, the adaptive optimization mechanism proved effective. When selecting nodes, not only were the high base weights of nodes A and C considered, but the execution deviation graph was also consulted to identify node C as the bottleneck in computational resource consumption for this type of task. Therefore, node C's priority was dynamically lowered in this selection process, ultimately choosing to collaborate with node A and another node B with superior computational performance. This effectively avoided potential performance issues and optimized task execution efficiency.

[0097] Through the above embodiments, this invention achieves efficient and intelligent multi-source data collaborative analysis while ensuring data privacy and security. The closed-loop dynamic reputation assessment and adaptive optimization mechanism enables the entire federated system to learn from execution experience and continuously improve the overall performance and reliability of collaborative analysis.

[0098] It should be noted that the functional division and information interaction between the various modules described above are logical, but in terms of physical implementation, they can be integrated on the same software platform or deployed in a distributed manner. The connections between them represent data flow and control flow, aiming to collaboratively achieve the objectives of this invention. The above descriptions are merely exemplary embodiments of this invention and should not be construed as limiting the scope of protection of this invention.

Claims

1. A multi-source data management method based on a federated mechanism, characterized in that, The method includes: Collect local multi-source heterogeneous data from each data source node, perform knowledge distillation to extract semantic summary data representing the data structure and relationships, encapsulate the semantic summary data and the network address of the data source node into a node registration transaction, perform consensus verification and record it to form a distributed data source registry. The system receives a cross-source collaborative analysis request input by a user, parses the execution logic of the request based on the semantic summary data, and generates an acyclic directed task graph. The generation of the acyclic directed task graph includes: semantically parsing the cross-source collaborative analysis request to extract query intent entities and operation logic predicates; retrieving the distributed data source registry, filtering execution data source nodes based on the matching degree between the query intent entities and the semantic summary data, and generating corresponding atomic query operations for the execution data source nodes based on the operation logic predicates; analyzing the data input-output dependencies between the atomic query operations, constructing data flow paths as directed edges, and combining and outputting the acyclic directed task graph using the atomic query operations as nodes. The query intent entity includes a core concept identifier and a context constraint descriptor, wherein: the core concept identifier is used to characterize the subject or attribute of the analysis target, providing a core index for matching calculations with the semantic summary data; the context constraint descriptor is used to limit the time, space, or other related conditions of the data, serving as a filtering basis when filtering the execution data source nodes. Based on the topological order of the acyclic directed task graph, local execution query operations are performed on the corresponding execution data source nodes, and intermediate results generated during the execution process are encrypted and securely shared to complete collaborative computation, generating final analysis results and task execution logs. The response time, data completeness, and consistency of intermediate results of each execution data source node recorded in the task execution log are analyzed. Based on the final analysis results, a dynamic reputation score is calculated and generated for each participating execution data source node. Based on the dynamic reputation score, the weights of each data source node in the distributed data source registry are updated. When parsing new cross-source collaborative analysis requests in the future, the data source node is selected to participate in the collaboration according to the weight, thereby achieving adaptive optimization of federated data management.

2. The multi-source data management method based on a federation mechanism according to claim 1, characterized in that, The formation of the distributed data source registry includes: Local multi-source heterogeneous data is cleaned and standardized, and different types of data are uniformly mapped into structured feature vectors to generate a standardized feature vector set; The standardized feature vector set is processed and projected into a low-dimensional semantic space to hide the original numerical details while preserving the data distribution characteristics, thereby generating semantic summary data. The semantic summary data and the network address of the data source node are used to construct a standardized node registration transaction. The integrity and format of the node registration transaction are verified to form a distributed data source registry.

3. The multi-source data management method based on a federation mechanism according to claim 1, characterized in that, The generation of the final analysis results and task execution log includes: Identify the initial atomic query operation in the acyclic directed task graph, and distribute the initial atomic query operation to the corresponding execution data source node to start local computation; After completing local computation, homomorphic encryption is used to encrypt the generated intermediate result data, and according to the data flow path defined by the acyclic directed task graph, it is securely transmitted to the next atomic query operation as the execution input. Once all atomic query operations in the acyclic directed task graph have been completed, the aggregated computation output serves as the final analysis result. It also gathers the identity identifiers, start and end times, and status information during the execution process and compiles them to generate a task execution log.

4. The multi-source data management method based on a federation mechanism according to claim 1, characterized in that, The method further includes: The planned execution path in the acyclic directed task graph is compared and analyzed with the actual execution data in the task execution log to quantify the deviation in execution latency and resource consumption, and generate an execution deviation map.

5. The multi-source data management method based on a federation mechanism according to claim 1, characterized in that, The calculation and generation of dynamic reputation scores for each participating data source node includes: The response time, data completeness, and intermediate result consistency of each execution data source node are extracted from the task execution log and normalized to generate response time score, data completeness score, and result consistency score. The influence of the execution data source node on the final analysis result is analyzed, and a result contribution score is generated through feature importance analysis. The response time score, data completeness score, result consistency score, and result contribution score are weighted and summed, and then updated with time decay to generate a dynamic reputation score.

6. The multi-source data management method based on a federation mechanism according to claim 5, characterized in that, The contribution score generated through feature importance analysis includes: For each execution data source node, the intermediate result data of the execution data source node is removed from the task execution log, and the collaborative computing process is simulated and recalculated to generate a perturbation analysis result with the execution data source node missing. The final analysis results are compared with the perturbation analysis results to measure the difference, quantify the magnitude of change in the results, and generate a result contribution score.

7. The multi-source data management method based on a federation mechanism according to claim 4, characterized in that, The adaptive optimization for implementing federated data management includes: Based on the dynamic reputation score of the execution data source node, update the basic weight of the node in the distributed data source registry; When parsing new cross-source collaborative analysis requests, based on the basic weights and the performance bottlenecks identified by the execution deviation map, the node selection priority of the corresponding deviation features is dynamically reduced, and weight and priority matching is performed to combine nodes to participate in collaboration.

8. A multi-source data management device based on a federation mechanism, applied to a multi-source data management method based on a federation mechanism as described in any one of claims 1-7, characterized in that, The device includes: The federated node registration and data abstraction module is used to collect local multi-source heterogeneous data from each data source node, perform knowledge distillation to extract semantic summary data that represents the data structure and relationships, encapsulate the semantic summary data and the network address of the data source node into a node registration transaction, perform consensus verification and record it to form a distributed data source registry. The collaborative task planning module is used to receive cross-source collaborative analysis requests input by the user, parse the execution logic of the cross-source collaborative analysis requests according to the semantic summary data, and generate an acyclic directed task graph. The federated collaborative execution module is used to perform local execution query operations on the corresponding execution data source nodes according to the topological order of the acyclic directed task graph, and to encrypt and securely share the intermediate results generated during the execution process to complete collaborative computing, and generate the final analysis results and task execution logs. The dynamic reputation assessment module is used to analyze the response time, data completeness, and consistency of intermediate results of each execution data source node recorded in the task execution log, and calculate and generate dynamic reputation scores for each participating execution data source node in combination with the final analysis results. The adaptive optimization and strategy adjustment module is used to update the weights of each data source node in the distributed data source registry based on the dynamic reputation score. When parsing new cross-source collaborative analysis requests in the future, the module selects data source nodes to participate in the collaboration based on their weights, thereby achieving adaptive optimization of federated data management.

Images