Multi-source heterogeneous system-oriented trusted distributed data fusion scheduling platform and method

Abstract

The application relates to the technical field of data security processing, in particular to a trusted distributed data fusion scheduling platform and method for a multi-source heterogeneous system. Heterogeneous data normalization mapping is completed through a double-constraint semantic similarity matching algorithm, data ownership right confirmation and integrity protection are realized by combining sharding encryption and hash verification; a multi-dimensional node scheduling priority calculation model is constructed, and task trusted distribution is completed through an improved practical Byzantine fault tolerance consensus mechanism; distributed fusion calculation is completed by cooperating with homomorphic encryption and differential privacy mechanism, and finally, hash consistency verification and block chain storage are used to realize whole-process trusted traceability. The application significantly improves the heterogeneous data fusion accuracy and scheduling efficiency, guarantees the safety and credibility of the whole-process data fusion, and adapts to the fusion scheduling requirements of multi-industry cross-subject multi-source heterogeneous systems.

Description

Technical Field

[0001] This invention relates to the field of data security processing technology, specifically to a trusted distributed data fusion scheduling platform and method for multi-source heterogeneous systems. Background Technology

[0002] With the rapid development of IoT, distributed computing, and cross-domain data sharing technologies, heterogeneous systems with multiple architectures, protocols, and formats are prevalent in various industries such as manufacturing, government, and finance. Cross-system data fusion and scheduling have become core technologies for realizing data value mining. Simultaneously, data trustworthiness, scheduling rationality, and privacy security have become critical technical challenges that urgently need to be addressed in this field. Currently, relevant patent documents have disclosed corresponding solutions for multi-source heterogeneous data fusion and distributed scheduling technologies, as detailed below: Reference document 1 is an invention patent application with publication number CN120706742A entitled "Intelligent Dispatch and Decision System for Pipeline Network with Multi-Source Heterogeneous Data Fusion". This document discloses a multi-source heterogeneous data fusion and intelligent dispatch system adapted to pipeline network scenarios. The core components include a multi-modal data acquisition module, a spatiotemporal alignment and fusion hub module, a digital twin inference module, and an adaptive decision matrix module. It is compatible with multiple industrial protocols such as Modbus-TCP and OPC-UA through a heterogeneous protocol parsing unit, uses an improved ICP algorithm to achieve millimeter-level spatial alignment of multi-source data, combines a federated learning cleaning tower to complete cross-domain data quality assessment, generates pipeline network dispatch strategies through a two-layer reinforcement learning architecture, and uses a Hyperledger Fabric consortium blockchain to store decision process data and execution logs. This solution mainly solves the problems of severe data silos, lagging dynamic response, and insufficient intelligent decision-making in traditional pipeline network systems.

[0003] Comparative document 2 is an invention patent application with publication number CN120474706A entitled "A Multi-Source Heterogeneous Distributed Computing Power Fusion Scheduling Method and System". This document discloses a fusion scheduling scheme for distributed computing power resources. The core steps are to collect and preprocess heterogeneous scheduling source data in multiple dimensions to generate a joint scheduling dataset, perform key demand prediction processing on the task set based on a preset quantum channel topology and generate a key pre-distribution matrix, complete the multi-objective matching degree calculation between the key pre-distribution matrix and the real-time task queue based on a dynamic priority quantization model, perform adaptive key supplementation and migration path optimization after generating the allocation decision matrix, and finally update the quantum channel topology based on federated feature extraction technology and graph convolutional network. This scheme mainly solves the problems of resource scheduling compatibility conflicts and low task migration success rate caused by hardware heterogeneity in traditional distributed scheduling methods.

[0004] The aforementioned existing technical solutions still have many technical shortcomings and cannot meet the full-process trusted management requirements of generalized data fusion scheduling for multi-source heterogeneous systems. Firstly, the solution in Comparative Document 1 is highly adapted to vertical pipeline scenarios. Its data fusion and scheduling mechanism is designed for pipeline fluid simulation and operational condition scheduling, but it has not formed a generalized distributed data fusion scheduling architecture for multi-source heterogeneous systems. Therefore, it cannot adapt to general data fusion scenarios involving multiple entities and systems, such as government and finance. Furthermore, its data preprocessing stage does not achieve ownership confirmation and integrity verification at the fragment granularity, and the reliability of the standardized processing stage before data fusion is insufficient. Secondly, the solution in Comparative Document 2 focuses on the optimization of distributed computing resource scheduling, primarily addressing the adaptation issues of computing resource allocation and quantum key distribution. It does not systematically design for the semantic normalization of multi-source heterogeneous data itself or the privacy protection of the fusion computing process, and thus cannot achieve heterogeneous data... The entire process of data fusion and scheduling, from collection, normalization, scheduling and distribution, to fusion computing and result storage, is subject to closed-loop management. However, its node scheduling priority model does not comprehensively consider multi-dimensional parameters such as node trust rating and historical execution status, making it difficult to ensure both scheduling efficiency and task execution reliability. Thirdly, existing technologies for data fusion and scheduling of multi-source heterogeneous systems generally suffer from technical defects such as insufficient accuracy of data normalization semantic matching, difficulty in balancing the reliability and execution efficiency of distributed scheduling, and an inability to balance privacy protection and data accuracy in the fusion computing process. Furthermore, the lack of a full-process blockchain-based trusted storage and traceability mechanism makes it difficult to meet the complex requirements of data ownership definition, privacy and security protection, and fusion result reliability in the process of cross-entity multi-source heterogeneous data fusion. Summary of the Invention

[0005] The purpose of this invention is to provide a trusted distributed data fusion and scheduling platform and method for multi-source heterogeneous systems, so as to solve the problems mentioned in the background art, such as low semantic adaptation accuracy of multi-source heterogeneous data, imbalance between distributed scheduling reliability and execution efficiency, insufficient privacy protection in the fusion process, and poor traceability of the whole process.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A trusted distributed data fusion and scheduling method for multi-source heterogeneous systems includes the following steps: S1 collects raw data from a multi-source heterogeneous system, completes the format normalization mapping of the heterogeneous data through a semantic similarity matching algorithm, generates a unique ownership identifier based on the data ownership subject, performs fragmentation processing on the normalized data, and generates fragment hash verification values; the matching degree calculation formula of the semantic similarity matching algorithm is: ; In the formula, This represents the matching degree value of the semantic feature sequences corresponding to the m-th group and the n-th group of heterogeneous data to be matched; , These are the semantic feature sequences of the m-th and n-th heterogeneous data after standardization, respectively. Let S be the set of semantic features corresponding to the semantic feature sequence. , Corresponding to , The set of semantic features; for and The number of elements in the intersection. , They are respectively , The total number of elements; This is a function for calculating the length of a sequence of characters. for and The character length of the longest common semantic sequence. , They are respectively , Total character length; This is the function for calculating the maximum value. for and The maximum total character length; S2 constructs a distributed scheduling node cluster, calculates node scheduling priorities, and achieves trusted consensus on scheduling tasks through an improved practical Byzantine fault-tolerant consensus mechanism, distributing data sharding tasks to the corresponding target nodes; After receiving the data fragments, the S3 target node performs distributed fusion computation of the fragmented data based on homomorphic encryption and differential privacy mechanisms to generate fusion result fragments. S4 performs hash consistency verification on all shards of the fusion results. After the verification passes, the results are aggregated, the final fusion data is generated, and the data is stored on the blockchain.

[0007] Preferably, in step S2, the formula for calculating the node scheduling priority is: ; In the formula, Let i be the scheduling priority of node i. Let be the trust rating value of node i. Let i be the real-time load rate of node i. Let be the average link delay of node i. To prevent zero constant, The attenuation coefficient is... Let i be the time interval since node i last successfully completed the scheduled task.

[0008] Preferably, in step S2, the view switching threshold of the improved practical Byzantine fault-tolerant consensus mechanism is calculated using the following formula: ; In the formula, Set a threshold for view switching. This represents the number of faulty nodes within the cluster. This represents the total number of nodes in the cluster. , For preset weighting coefficients, This represents the average consensus latency of the cluster.

[0009] Preferably, in step S1, when processing the normalized data by sharding, the number of shards N is first determined based on the total length of the normalized data, the number of data ownership subjects, and the number of available nodes in the distributed scheduling node cluster. N is the smallest positive integer greater than or equal to the minimum number of consensus nodes in the cluster. Data sharding is completed using a fixed-length sharding method combined with boundary semantic alignment, and no cross-sharding is performed on a single semantically complete data unit. Based on the unique ownership identifier of the data ownership subject, a root key is generated through SHA-256 hash operation. Then, combined with the sequence number of each shard and the shard start offset, a unique symmetric encryption key for the corresponding shard is generated through the key derivation function KDF. The corresponding shard data is encrypted shard by shard using the AES-256-GCM symmetric encryption algorithm to generate shard ciphertext and encryption check code. The shard ciphertext, shard sequence number, encryption check code, and ownership identification are concatenated, and a unique shard hash check value for the shard is generated through the SM3 hash algorithm. The shard hash check value is then bound and stored with the corresponding shard sequence number.

[0010] Preferably, in step S3, during the distributed fusion computing process, the received encrypted data fragments are first subjected to homomorphic decryption verification. After the verification passes, based on preset sensitive attribute classification rules, the sensitivity levels of each data field within the fragments are divided into five levels, with the level value being positively correlated with the degree of sensitivity. The global sensitivity of the fusion computing is determined based on the type of fusion computing operator and the numerical range of the fragmented data. The fusion computing operator includes a summation operator, a mean operator, and a weighted aggregation operator. Differential privacy noise is generated based on the Laplace mechanism, and the probability density function of the noise is: ; In the formula, λ is the noise scale parameter. λ is positively correlated with the highest sensitivity level of the data shard and the global sensitivity of the fusion calculation, and negatively correlated with the preset privacy budget ε. The generated differential privacy noise is successively superimposed on the intermediate results of each round of fusion calculation to complete the distributed fusion calculation and generate fusion result shards.

[0011] On the other hand, the present invention also provides a trusted distributed data fusion scheduling platform for multi-source heterogeneous systems, for implementing the above-mentioned trusted distributed data fusion scheduling method for multi-source heterogeneous systems, including a multi-source heterogeneous data preprocessing module, a trusted distributed scheduling module, a privacy-enhancing fusion computing module, and a trusted verification and evidence storage module that are connected in sequence. The multi-source heterogeneous data preprocessing module is used to collect the original data of the multi-source heterogeneous system, complete the format normalization mapping of the heterogeneous data through the semantic similarity matching algorithm, generate a unique ownership identifier based on the data ownership subject, and perform fragment processing on the normalized data and generate fragment hash verification values. The trusted distributed scheduling module is used to build a distributed scheduling node cluster, calculate the node scheduling priority, complete the trusted consensus of the scheduling task through an improved practical Byzantine fault-tolerant consensus mechanism, and distribute the data sharding task to the corresponding target node. The privacy-enhanced fusion computing module is used to control the target node to receive data fragments and then perform distributed fusion computing of the fragmented data based on homomorphic encryption and differential privacy mechanisms to generate fusion result fragments. The trusted verification and evidence storage module is used to perform hash consistency verification on all shards of the fusion results. After the verification is passed, the results are aggregated, the final fusion data is generated, and the evidence is stored on the blockchain.

[0012] Preferably, the trusted distributed scheduling module has a built-in priority calculation unit, which calculates the node scheduling priority using the following formula: ; In the formula, Let i be the scheduling priority of node i. Let be the trust rating value of node i. Let i be the real-time load rate of node i. Let be the average link delay of node i. To prevent zero constant, The attenuation coefficient is... Let i be the time interval since node i last successfully completed the scheduled task.

[0013] Preferably, the trusted distributed scheduling module incorporates an improved practical Byzantine fault-tolerant consensus unit, which calculates the view switching threshold using the following formula: ; In the formula, Set a threshold for view switching. This represents the number of faulty nodes within the cluster. This represents the total number of nodes in the cluster. , For preset weighting coefficients, This represents the average consensus latency of the cluster.

[0014] Preferably, the multi-source heterogeneous data preprocessing module has a built-in fragment encryption unit, which includes a fragment partitioning component, a key derivation component, a symmetric encryption component, and a hash generation component. The data sharding component is used to determine the number of shards N based on the normalized total data length, the number of data ownership subjects, and the number of available nodes in the distributed scheduling node cluster. N is the smallest positive integer greater than or equal to the minimum consensus node number of the cluster. Data sharding is completed using a fixed-length sharding method combined with boundary semantic alignment. Single semantically complete data units are not split across shards. The key derivation component is used to generate a root key based on the unique ownership identifier of the data ownership subject through SHA-256 hash operation. Then, combined with the sequence number of each shard and the shard start offset, a unique symmetric encryption key for the corresponding shard is generated through the key derivation function KDF. The symmetric encryption component is used to encrypt the corresponding data fragments one by one using the AES-256-GCM symmetric encryption algorithm, generating fragment ciphertext and encryption verification code; the hash generation component is used to concatenate the fragment ciphertext, fragment sequence number, encryption verification code, and ownership confirmation identifier, generate a unique fragment hash verification value for the fragment using the SM3 hash algorithm, and bind and store the fragment hash verification value with the corresponding fragment sequence number.

[0015] Preferably, the privacy-enhancing fusion computing module has a built-in differential privacy protection unit, which includes a sensitivity classification component, a sensitivity calculation component, a noise generation component, and a noise superposition component. The sensitivity classification component is used to classify the sensitivity level of each data field in the fragmented data after the homomorphic decryption verification is passed, based on the preset sensitivity attribute classification rules. The sensitivity level is divided into five levels, and the level value is positively correlated with the sensitivity degree. The sensitivity calculation component is used to determine the global sensitivity of the fusion calculation based on the type of fusion calculation operator and the numerical range of the fragmented data. The fusion calculation operator includes a summation operator, a mean operator, and a weighted aggregation operator. The noise generation component is used to generate differential privacy noise based on the Laplace mechanism, and the probability density function of the noise is: ; In the formula, λ is a noise scale parameter, which is positively correlated with the highest sensitivity level of the data shard and the global sensitivity of the fusion calculation, and negatively correlated with the preset privacy budget ε. The noise superposition component is used to successively superimpose the generated differential privacy noise into each round of intermediate results of the fusion calculation to complete the distributed fusion calculation and output the fusion result shards.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention improves the normalization mapping accuracy of heterogeneous data with different architectures, protocols and formats by using a dual-constraint semantic similarity matching algorithm, which takes into account both the overlap of semantic feature sets and the contextual relevance of semantic sequences, and avoids the deviation of fused data caused by semantic mismatch. At the same time, it generates a unique ownership identifier based on the data ownership subject, and combines the symmetric encryption and hash verification mechanism at the fragment level to realize the ownership definition and integrity protection of data fragment granularity. It solves the technical problems of poor data consistency, ambiguous ownership and easy illegal tampering in the preprocessing stage before multi-source data fusion, and provides a standardized and highly reliable data source foundation for subsequent distributed fusion scheduling.

[0017] (2) The node scheduling priority calculation model designed in this invention integrates multiple parameters such as node trust rating, real-time load, link transmission performance and historical task execution status to achieve dynamic optimal distribution of scheduling tasks, effectively balance the computing load of cluster nodes, and avoid task execution delay caused by single-point overload. At the same time, the practical Byzantine fault-tolerant consensus mechanism is improved. The view switching threshold is dynamically adjusted by the proportion of faulty nodes and the average consensus latency of the cluster, which solves the technical problems of delayed master node fault response and redundant consensus process under the traditional fixed threshold mechanism. Under the premise of ensuring that the scheduling process is resistant to Byzantine attacks and the task distribution is trustworthy, the execution efficiency of the consensus process and the dynamic adaptation capability of the cluster are greatly improved.

[0018] (3) This invention completes the distributed fusion computation of fragmented data in the encrypted state through the synergistic application of homomorphic encryption and differential privacy mechanism, avoiding the risk of exposure of plaintext data during transmission and computation. At the same time, the differential privacy noise scale is dynamically adjusted based on the data sensitivity level and the global sensitivity of fusion computation, reducing the impact of noise on the accuracy of fusion results while meeting privacy protection requirements. Furthermore, by combining the fragmented hash consistency verification of fusion results with blockchain notarization, the immutability and traceability of data throughout the entire process from preprocessing, scheduling and distribution, fusion computation to result output are realized, ensuring the legality and credibility of the final fused data and adapting to the compliance and security requirements of cross-subject and cross-system multi-source heterogeneous data fusion. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are explained in detail together with the embodiments of the invention, but do not constitute a limitation thereof.

[0020] Figure 1 This is a block diagram of the trusted distributed data fusion and scheduling platform module for multi-source heterogeneous systems according to the present invention. Figure 2This is a flowchart of the trusted distributed data fusion and scheduling method for multi-source heterogeneous systems according to the present invention. Figure 3 This is a detailed diagram of the multi-source heterogeneous data preprocessing module of the present invention; Figure 4 This is a detailed diagram of the trusted distributed scheduling module of the present invention; Figure 5 This is a detailed diagram of the privacy-enhancing fusion computing and trusted verification and evidence storage module of the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figures 1-5 As shown, the trusted distributed data fusion scheduling platform for multi-source heterogeneous systems provided by the present invention includes: Multi-source heterogeneous data normalization and fragmentation preprocessing: First, raw data from multi-source heterogeneous systems with different architectures, protocols, and data formats are collected through a pre-defined interface adaptation component. This raw data includes structured, semi-structured, and unstructured data. Semantic features are extracted from the collected raw data to generate corresponding semantic feature sequences. Finally, a semantic similarity matching algorithm is used to complete the format normalization mapping of the heterogeneous data.

[0023] The formula for calculating the matching degree in semantic similarity matching algorithms is: ; In the formula, For the first to be matched Group and No. The matching degree value of the semantic feature sequences corresponding to the heterogeneous data sets. For the first Semantic feature sequences of heterogeneous data after standardization. For the first Semantic feature sequences of heterogeneous data after standardization. Semantic feature sequence The corresponding set of semantic features. for The corresponding set of semantic features. for The corresponding set of semantic features. for and The number of elements in the intersection. for The total number of elements. for The total number of elements. This is a function for calculating the length of a sequence of characters. for and The character length of the longest common semantic sequence. for Total character length. for Total character length. This is a function for calculating the maximum value. for and The maximum value.

[0024] When the matching degree value When the data is greater than or equal to a preset matching threshold, the two sets of heterogeneous data are mapped to the same standardized data template to complete the format normalization process. Based on the identity identifier of the data ownership subject and the unique data code, a unique data ownership identifier is generated through hash calculation. The normalized data is then fragmented. The number of fragments is determined based on the total length of the normalized data, the number of data ownership subjects, and the number of available nodes in the distributed scheduling node cluster. , The smallest positive integer greater than or equal to the minimum number of consensus nodes in the cluster is selected. Data sharding is performed using fixed-length sharding combined with boundary semantic alignment; semantically complete data units are not split across shards. Based on the unique ownership identifier of the data owner, a root key is generated through SHA-256 hashing. Then, combined with the sequence number and shard start offset of each shard, a unique symmetric encryption key for the corresponding shard is generated using the Key Derivation Function (KDF). The AES-256-GCM symmetric encryption algorithm is used to encrypt the corresponding shard data shard by shard, generating shard ciphertext and encryption checksums. The shard ciphertext, shard sequence number, encryption checksum, and ownership identifier are concatenated, and a unique shard hash checksum is generated using the SM3 hash algorithm. The shard hash checksum is then bound and stored with the corresponding shard sequence number.

[0025] Distributed Node Trusted Scheduling and Consensus Implementation: This step constructs a distributed scheduling node cluster to complete the trusted distribution and consensus confirmation of scheduling tasks. First, all nodes within the cluster are registered and rated for trustworthiness, generating a trustworthiness rating value for each node. It collects real-time operational status data from each node within the cluster, including real-time node load rates. Average link latency per node The time interval since the node last successfully completed a scheduled task. The scheduling priority of each node is calculated based on the above parameters. The formula for calculating node scheduling priority is: ; In the formula, For nodes The scheduling priority. For nodes The credibility rating value. For nodes Real-time load rate. For nodes The average link latency. To prevent zero constant. This is the attenuation coefficient. For nodes The time interval since the last successful completion of the scheduled task.

[0026] According to scheduling priority The nodes within the cluster are sorted from highest to lowest priority, and available nodes that meet the priority requirements are selected as target nodes. A reliable consensus for scheduling tasks is achieved through an improved Practical Byzantine Fault-Tolerant consensus mechanism. The view switching threshold for this improved mechanism is calculated using the following formula: ; In the formula, Set the view switching threshold. This represents the number of faulty nodes within the cluster. This represents the total number of nodes in the cluster. These are preset weighting coefficients. These are preset weighting coefficients. This represents the average consensus latency of the cluster.

[0027] When the master node response latency in the consensus process is greater than or equal to the view switching threshold Upon completion, the view switching process is automatically triggered, completing the master node replacement and consensus process restart. After consensus is achieved, the data sharding task is distributed to the corresponding target node, and the decryption key and shard hash verification value of the corresponding shard are distributed synchronously.

[0028] Privacy-Enhanced Distributed Data Fusion Computation: This step completes secure distributed fusion computation of fragmented data. After receiving the data fragments, the target node first decrypts the fragmented ciphertext using the fragment decryption key, then verifies data integrity using the fragment hash checksum. Upon successful verification, the encrypted preprocessing of the fragmented data is performed based on a homomorphic encryption mechanism. Based on preset sensitivity attribute classification rules, each data field within the fragment is classified into five sensitivity levels, with the level value positively correlated with the degree of sensitivity. The global sensitivity of the fusion computation is determined based on the type of fusion computation operator and the numerical range of the fragmented data. The fusion computation operators include summation, averaging, and weighted aggregation operators. Differential privacy noise is generated based on the Laplace mechanism, and the probability density function of the noise is: ; In the formula, For noise The probability density value. This is the noise scale parameter. The global sensitivity is positively correlated with the highest sensitivity level value of the data shards and is also positively correlated with the preset privacy budget. Negative correlation.

[0029] The generated differential privacy noise is successively superimposed onto the intermediate results of each round of fusion computation to complete the distributed fusion computation and generate fusion result fragments. A hash operation is performed on the fusion result fragments to generate fusion result hash values, and the fusion result fragments and their corresponding hash values ​​are synchronously uploaded to the scheduling master node.

[0030] Trustworthy Verification and Blockchain Evidence of Fusion Results: This step completes the consistency verification and trusted evidence storage of the fusion results. After the scheduling master node receives the fusion result shards and corresponding hash values ​​uploaded by all target nodes, it first performs hash consistency verification on all fusion result shards, comparing whether the hash value of the fusion result is consistent with the hash link of the shard calculation process, and discarding fusion result shards that fail the verification. All fusion result shards that pass the verification are sorted according to the shard number, and the results are aggregated to generate the final fusion data. The final fusion data is hashed to generate a unique evidence hash value, and the evidence hash value data ownership identifier, which identifies the node information of the fusion calculation process, is synchronized and stored on the consortium blockchain, completing the trusted traceability and tamper-proof evidence storage of the entire process.

[0031] Example 1: Data Fusion Scenario of Multi-Source Heterogeneous Systems in Industrial Internet of Things This embodiment is applied to an industrial Internet of Things (IoT) scenario. The multi-source heterogeneous system includes production equipment, sensor systems, MES systems, and ERP systems. The heterogeneous data formats include Modbus protocol data, JSON format data, and relational database table data.

[0032] Multi-source heterogeneous data preprocessing Three sets of heterogeneous data were collected, and semantic feature sequences were extracted from them respectively. .in , . , . . .

[0033] Substitute into the semantic similarity matching formula to calculate ; The preset matching threshold is 0.6. If the value exceeds the matching threshold, complete the normalization mapping of the two sets of data. Similarly, complete the normalization processing of all heterogeneous data to generate unique data ownership identifiers. Determine the number of shards. The data is sharded and encrypted, and a shard hash verification value is generated for each shard.

[0034] Distributed node scheduling and consensus Build a distributed node cluster. Select node 1 for priority calculation. , , , , , .

[0035] Substitute into the node scheduling priority formula to calculate ; Similarly, calculate the priority of all nodes and select the top 7 nodes as target nodes. , , , .

[0036] Substitute into the view switching threshold formula to calculate ; When the master node response latency is greater than or equal to 8.06ms, a view switch is triggered. After the consensus of the scheduling task is completed, the data is sharded and distributed to the target node.

[0037] Distributed converged computing After receiving the data fragments, the target node performs decryption and verification, classifies the data sensitivity level into three levels, uses a summation operator for fusion calculation, calculates a global sensitivity of 120, and sets a preset privacy budget. .

[0038] calculate Substituting into the Laplace noise probability density function, when hour ; Generate corresponding differential privacy noise, superimpose it onto the intermediate results of the fusion computation, complete the distributed fusion computation, and generate fusion result fragments.

[0039] Result Verification and Evidence Preservation All fusion result shards are subjected to hash consistency verification. After the verification is passed, the results are aggregated to generate the final fusion data. The data storage hash value is then stored on the industrial alliance blockchain.

[0040] Example 2: Cross-departmental multi-source heterogeneous data sharing and fusion scenario in government affairs This embodiment is applied to a cross-departmental data sharing scenario in government affairs. The multi-source heterogeneous systems include a household registration management system, a social security management system, a real estate registration system, and a market supervision system. The heterogeneous data formats include government XML data, structured government database table data, and unstructured certificate and license document data.

[0041] Multi-source heterogeneous data preprocessing Four sets of heterogeneous government data were collected, and semantic feature sequences were extracted from them respectively. .in , . , . . .

[0042] Substitute into the semantic similarity matching formula to calculate ; The preset matching threshold is 0.6. If the data exceeds the matching threshold, complete the normalization mapping of the two sets of data. Similarly, complete the normalization processing of all heterogeneous government data, generating unique data ownership identifiers. Determine the number of data shards. The data is sharded and encrypted, and a shard hash verification value is generated for each shard.

[0043] Distributed node scheduling and consensus Construct a distributed node cluster for the government alliance. Select node 3 for priority calculation. , , , , , .

[0044] Substitute into the node scheduling priority formula to calculate ; Similarly, calculate the priority of all nodes and select the top 10 nodes by priority as the target nodes. , , , .

[0045] Substitute into the view switching threshold formula to calculate ; When the master node response latency is greater than or equal to 4.5467ms, a view switch is triggered. After the consensus of the scheduling task is completed, the data is sharded and distributed to the target node.

[0046] Distributed converged computing After receiving the data fragments, the target node performs decryption and verification, classifies the data sensitivity level into four levels, uses the mean operator for fusion calculation, calculates the global sensitivity to be 85, and sets a preset privacy budget. .

[0047] calculate Substituting into the Laplace noise probability density function, when hour ; Generate corresponding differential privacy noise, superimpose it onto the intermediate results of the fusion computation, complete the distributed fusion computation, and generate fusion result fragments.

[0048] Result Verification and Evidence Preservation All fusion result shards are subjected to hash consistency verification. After the verification is passed, the results are aggregated to generate the final fusion data. The data storage hash value is then stored on the government alliance blockchain.

[0049] Example 3: Joint Fusion of Heterogeneous Data from Multiple Financial Institutions This embodiment is applied to a joint risk control scenario involving multiple financial institutions. The multi-source heterogeneous systems include bank credit systems, securities trading systems, insurance claims systems, and credit inquiry systems. The heterogeneous data formats include financial transaction flow data, risk control rating data, and unstructured credit report data.

[0050] Multi-source heterogeneous data preprocessing Five sets of heterogeneous financial data were collected, and semantic feature sequences were extracted from each set. .in , . , . . .

[0051] Substitute into the semantic similarity matching formula to calculate ; The preset matching threshold is 0.65. If the data exceeds the matching threshold, complete the normalization mapping of the two sets of data. Similarly, complete the normalization processing of all heterogeneous financial data to generate unique data ownership identifiers. Determine the number of shards. The data is sharded and encrypted, and a shard hash verification value is generated for each shard.

[0052] Distributed node scheduling and consensus Build a distributed node cluster for the financial alliance. Node 5 is selected for priority calculation. , , , , , .

[0053] Substitute into the node scheduling priority formula to calculate ; Similarly, calculate the priority of all nodes and select the top 13 nodes by priority as the target nodes. , , , .

[0054] Substitute into the view switching threshold formula to calculate ; When the master node response latency is greater than or equal to 5.05ms, a view switch is triggered. After the consensus of the scheduling task is completed, the data is sharded and distributed to the target node.

[0055] Distributed converged computing After receiving the data fragments, the target node performs decryption and verification, classifies the data sensitivity level into five levels, uses a weighted aggregation operator for fusion calculation, calculates a global sensitivity of 200, and sets a preset privacy budget. .

[0056] calculate Substituting into the Laplace noise probability density function, when hour ; Generate corresponding differential privacy noise, superimpose it onto the intermediate results of the fusion computation, complete the distributed fusion computation, and generate fusion result fragments.

[0057] Result Verification and Evidence Preservation All fusion result shards are subjected to hash consistency verification. After the verification is passed, the results are aggregated to generate the final fusion data. The data storage hash value is then stored on the financial consortium blockchain.

[0058] This invention improves the normalization mapping accuracy of heterogeneous data with different architectures, protocols, and formats by employing a dual-constraint semantic similarity matching algorithm that simultaneously considers the overlap of semantic feature sets and the contextual relevance of semantic sequences, thus avoiding data fusion bias caused by semantic mismatch. Furthermore, it generates unique ownership identifiers based on data ownership entities and combines shard-level symmetric encryption and hash verification mechanisms to achieve ownership definition and integrity protection at the data shard granularity level. This solves the technical problems of poor data consistency, ambiguous ownership, and susceptibility to illegal tampering in the preprocessing stage before multi-source data fusion, providing a standardized and highly reliable data source foundation for subsequent distributed fusion scheduling. The node scheduling priority calculation model designed in this invention integrates multiple parameters such as node trust rating, real-time load, link transmission performance, and historical task execution status to achieve dynamic optimal distribution of scheduling tasks, effectively balancing the computing load of cluster nodes and avoiding task execution delays caused by single-point overload. At the same time, it improves the practical Byzantine fault-tolerant consensus mechanism by dynamically adjusting the view switching threshold through the proportion of faulty nodes and the average consensus latency of the cluster. This solves the technical problems of delayed master node fault response and redundant consensus process under the traditional fixed threshold mechanism. Under the premise of ensuring the scheduling process is resistant to Byzantine attacks and the task distribution is trustworthy, it significantly improves the execution efficiency of the consensus process and the dynamic adaptation capability of the cluster. This invention utilizes the synergistic application of homomorphic encryption and differential privacy mechanisms to complete distributed fusion computation of fragmented data in the encrypted state, avoiding the risk of exposure of plaintext data during transmission and computation. Simultaneously, it dynamically adjusts the differential privacy noise scale based on data sensitivity levels and the global sensitivity of the fusion computation, reducing the impact of noise on the accuracy of the fusion results while meeting privacy protection requirements. Furthermore, by combining fragmented hash consistency verification of the fusion results with blockchain notarization, it achieves immutability and traceability of data throughout the entire process from preprocessing, scheduling and distribution, fusion computation to result output, ensuring the legality and credibility of the final fused data and adapting to the compliance and security requirements of cross-entity, cross-system, multi-source heterogeneous data fusion.

[0059] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A trusted distributed data fusion and scheduling method for multi-source heterogeneous systems, characterized in that, Includes the following steps: S1 collects raw data from multi-source heterogeneous systems, completes the format normalization mapping of heterogeneous data through semantic similarity matching algorithm, generates a unique ownership identifier based on the data ownership subject, performs fragmentation processing on the normalized data and generates fragment hash verification value. The formula for calculating the matching degree of the semantic similarity matching algorithm is as follows: ； In the formula, This represents the matching degree value of the semantic feature sequences corresponding to the m-th group and the n-th group of heterogeneous data to be matched; , These are the semantic feature sequences of the m-th and n-th heterogeneous data after standardization, respectively. Let S be the set of semantic features corresponding to the semantic feature sequence. , Corresponding to , The set of semantic features; for and The number of elements in the intersection. , They are respectively , The total number of elements; This is a function for calculating the length of a sequence of characters. for and The character length of the longest common semantic sequence. , They are respectively , Total character length; This is the function for calculating the maximum value. for and The maximum total character length; S2 constructs a distributed scheduling node cluster, calculates node scheduling priorities, and achieves trusted consensus on scheduling tasks through an improved practical Byzantine fault-tolerant consensus mechanism, distributing data sharding tasks to the corresponding target nodes; After receiving the data fragments, the S3 target node performs distributed fusion computation of the fragmented data based on homomorphic encryption and differential privacy mechanisms to generate fusion result fragments. S4 performs hash consistency verification on all shards of the fusion results. After the verification passes, the results are aggregated, the final fusion data is generated, and the data is stored on the blockchain.

2. The trusted distributed data fusion and scheduling method for multi-source heterogeneous systems according to claim 1, characterized in that, In step S2, the formula for calculating the node scheduling priority is: ； In the formula, Let i be the scheduling priority of node i. Let be the trust rating value of node i. Let i be the real-time load rate of node i. Let be the average link delay of node i. To prevent zero constant, The attenuation coefficient is... Let i be the time interval since node i last successfully completed the scheduled task.

3. The trusted distributed data fusion and scheduling method for multi-source heterogeneous systems according to claim 1, characterized in that, In step S2, the view switching threshold of the improved practical Byzantine fault-tolerant consensus mechanism is calculated using the following formula: ； In the formula, Set a threshold for view switching. This represents the number of faulty nodes within the cluster. This represents the total number of nodes in the cluster. , For preset weighting coefficients, This represents the average consensus latency of the cluster.

4. The trusted distributed data fusion and scheduling method for multi-source heterogeneous systems according to claim 1, characterized in that, In step S1, when processing the normalized data into fragments, the number of fragments N is first determined based on the total length of the normalized data, the number of data ownership subjects, and the number of available nodes in the distributed scheduling node cluster. N is the smallest positive integer greater than or equal to the minimum number of consensus nodes in the cluster. Data fragmentation is completed using fixed-length fragmentation combined with boundary semantic alignment, and no cross-fragmentation is performed on single semantically complete data units. Based on the unique ownership identifier of the data ownership subject, a root key is generated through SHA-256 hash operation. Then, combined with the sequence number of each fragment and the fragment starting offset, a unique symmetric encryption key for the corresponding fragment is generated through the key derivation function KDF. The corresponding fragment data is encrypted fragment by fragment using the AES-256-GCM symmetric encryption algorithm to generate fragment ciphertext and encryption check code. The fragment ciphertext, fragment sequence number, encryption check code, and ownership identification are concatenated, and a unique fragment hash check value for the fragment is generated through the SM3 hash algorithm. The fragment hash check value is then bound and stored with the corresponding fragment sequence number.

5. The trusted distributed data fusion and scheduling method for multi-source heterogeneous systems according to claim 1, characterized in that, In step S3, during the distributed fusion computing process, the received encrypted data fragments are first subjected to homomorphic decryption verification. After the verification is passed, based on the preset sensitive attribute classification rules, the sensitivity level of each data field in the fragment is divided into five levels, with the level value being positively correlated with the degree of sensitivity. Based on the type of fusion computation operator and the numerical range of the fragmented data, the global sensitivity of the fusion computation is determined. The fusion computation operators include summation operators, mean operators, and weighted aggregation operators. Differential privacy noise is generated based on the Laplace mechanism, and the probability density function of the noise is: ； In the formula, λ is the noise scale parameter. λ is positively correlated with the highest sensitivity level of the data shard and the global sensitivity of the fusion calculation, and negatively correlated with the preset privacy budget ε. The generated differential privacy noise is successively superimposed on the intermediate results of each round of fusion calculation to complete the distributed fusion calculation and generate fusion result shards.

6. A trusted distributed data fusion scheduling platform for multi-source heterogeneous systems, used to implement the trusted distributed data fusion scheduling method for multi-source heterogeneous systems as described in any one of claims 1-5, characterized in that, It includes a multi-source heterogeneous data preprocessing module, a trusted distributed scheduling module, a privacy-enhancing fusion computing module, and a trusted verification and evidence storage module that are connected in sequence. The multi-source heterogeneous data preprocessing module is used to collect the original data of the multi-source heterogeneous system, complete the format normalization mapping of the heterogeneous data through the semantic similarity matching algorithm, generate a unique ownership identifier based on the data ownership subject, and perform fragment processing on the normalized data and generate fragment hash verification values. The trusted distributed scheduling module is used to build a distributed scheduling node cluster, calculate the node scheduling priority, complete the trusted consensus of the scheduling task through an improved practical Byzantine fault-tolerant consensus mechanism, and distribute the data sharding task to the corresponding target node. The privacy-enhanced fusion computing module is used to control the target node to receive data fragments and then perform distributed fusion computing of the fragmented data based on homomorphic encryption and differential privacy mechanisms to generate fusion result fragments. The trusted verification and evidence storage module is used to perform hash consistency verification on all shards of the fusion results. After the verification is passed, the results are aggregated, the final fusion data is generated, and the evidence is stored on the blockchain.

7. The trusted distributed data fusion and scheduling platform for multi-source heterogeneous systems according to claim 6, characterized in that, The trusted distributed scheduling module has a built-in priority calculation unit, which calculates the node scheduling priority using the following formula: ； In the formula, Let i be the scheduling priority of node i. Let be the trust rating value of node i. Let i be the real-time load rate of node i. Let be the average link delay of node i. To prevent zero constant, The attenuation coefficient is... Let i be the time interval since node i last successfully completed the scheduled task.

8. The trusted distributed data fusion and scheduling platform for multi-source heterogeneous systems according to claim 6, characterized in that, The trusted distributed scheduling module incorporates an improved practical Byzantine fault-tolerant consensus unit, which calculates the view switching threshold using the following formula: ； In the formula, Set a threshold for view switching. This represents the number of faulty nodes within the cluster. This represents the total number of nodes in the cluster. , For preset weighting coefficients, This represents the average consensus latency of the cluster.

9. The trusted distributed data fusion and scheduling platform for multi-source heterogeneous systems according to claim 6, characterized in that, The multi-source heterogeneous data preprocessing module has a built-in fragment encryption unit, which includes a fragment partitioning component, a key derivation component, a symmetric encryption component, and a hash generation component. The data sharding component is used to determine the number of shards N based on the normalized total data length, the number of data ownership subjects, and the number of available nodes in the distributed scheduling node cluster. N is the smallest positive integer greater than or equal to the minimum consensus node number of the cluster. Data sharding is completed using a fixed-length sharding method combined with boundary semantic alignment. Single semantically complete data units are not split across shards. The key derivation component is used to generate a root key based on the unique ownership identifier of the data ownership subject through SHA-256 hash operation. Then, combined with the sequence number of each shard and the shard start offset, a unique symmetric encryption key for the corresponding shard is generated through the key derivation function KDF. The symmetric encryption component is used to encrypt the corresponding data fragments one by one using the AES-256-GCM symmetric encryption algorithm, generating fragment ciphertext and encryption verification code; the hash generation component is used to concatenate the fragment ciphertext, fragment sequence number, encryption verification code, and ownership confirmation identifier, generate a unique fragment hash verification value for the fragment using the SM3 hash algorithm, and bind and store the fragment hash verification value with the corresponding fragment sequence number.

10. The trusted distributed data fusion and scheduling platform for multi-source heterogeneous systems according to claim 6, characterized in that, The privacy-enhancing fusion computing module has a built-in differential privacy protection unit, which has built-in a sensitivity classification component, a sensitivity calculation component, a noise generation component, and a noise superposition component. The sensitivity classification component is used to classify the sensitivity level of each data field in the fragment after the homomorphic decryption verification has passed, based on the preset sensitivity attribute classification rules. The sensitivity level is divided into five levels, and the level value is positively correlated with the sensitivity. The sensitivity calculation component is used to determine the global sensitivity of the fusion calculation based on the type of the fusion calculation operator and the numerical range of the sharded data. The fusion calculation operator includes a summation operator, a mean operator, and a weighted aggregation operator. The noise generation component is used to generate differential privacy noise based on the Laplace mechanism, and the probability density function of the noise is: In the formula, λ is the noise scale parameter, which is positively correlated with the highest sensitivity level of data sharding and the global sensitivity of fusion calculation, and negatively correlated with the preset privacy budget ε. The noise superposition component is used to successively superimpose the generated differential privacy noise onto each round of intermediate results of the fusion computation, thereby completing the distributed fusion computation and outputting fusion result fragments.

Images