Distributed privacy computing big data processing system and method

Abstract

The invention relates to the technical field of privacy computing and big data processing, and discloses a distributed privacy computing big data processing system and method.The system comprises an edge computing layer, a privacy computing layer and a block chain layer, and the edge computing layer is deployed in the local of each data provider and comprises a data cleaning module and a dynamic desensitization module; the privacy calculation layer comprises a federal learning module, a homomorphic encryption module and a resource scheduling module, and the block chain layer comprises a right confirmation module, an auditing module and a hierarchical evidence storage module. Through deep fusion of federated learning, homomorphic encryption, block chain and other technologies, a three-layer architecture distributed privacy computing system of'edge computing layer-privacy computing layer-block chain layer 'is constructed, a corresponding computing method is provided, and the problems of efficiency, safety and auditing of existing privacy computing in big data processing are solved.

Description

Technical Field

[0001] This invention relates to the field of privacy computing and big data processing technology, specifically to a distributed privacy computing big data processing system and method based on federated learning, homomorphic encryption, and blockchain notarization. Background Technology

[0002] The field of big data analytics currently faces two major challenges: First, the fragmented storage of data across institutions creates "data silos," making cross-institutional collaborative computing difficult and hindering the full realization of data value. Second, increasingly stringent data privacy regulations pose a high risk of leakage when raw data is directly circulated, severely restricting the legal circulation and sharing of data.

[0003] While traditional privacy computing technologies have alleviated the aforementioned problems to some extent, they still have many shortcomings: In terms of computational efficiency, existing federated learning frameworks have iteration cycles of up to several weeks when processing petabyte-scale medical images or large-scale financial transaction data, which cannot meet the needs of real-time analysis; in terms of privacy protection strength, homomorphic encryption algorithms are prone to accuracy loss in complex data analysis (such as deep learning model training), and key management is complex, making it difficult to adapt to diverse computing scenarios; in terms of data ownership confirmation and auditing, traditional solutions lack a reliable data circulation process evidence storage mechanism, making it impossible to effectively trace data usage behavior, resulting in difficulties in resolving data usage disputes.

[0004] Among existing related technologies, Chinese patent CN115276950B proposed a privacy processing method that combines homomorphic encryption and multinomial computation, but it did not solve the resource scheduling problem of large-scale data parallel computing, and the computing efficiency could not meet the needs of practical applications; Yunnan Power Grid patent CN119379286A realized blockchain data ownership confirmation, but it did not integrate federated learning and dynamic encryption strategies, and could not take into account both the needs of data collaborative computing and the strength of privacy protection.

[0005] Therefore, there is an urgent need for an innovative privacy computing solution that balances computational efficiency, privacy protection strength, and trusted auditing capabilities to address the shortcomings of existing technologies. Summary of the Invention

[0006] To overcome the aforementioned technical deficiencies, this invention provides a distributed privacy computing big data processing system and method, which achieves efficient, secure, and auditable collaborative computing of cross-institutional data through a three-layer architecture design and deep integration of multiple technologies.

[0007] Therefore, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a distributed privacy computing big data processing system, the system comprising: an edge computing layer, a privacy computing layer, and a blockchain layer. The edge computing layer is deployed locally at each data provider and includes a data cleaning module and a dynamic desensitization module. The data cleaning module is used to preprocess the raw data to remove invalid information and interference data. The dynamic desensitization module is used to match the corresponding desensitization algorithm and desensitization granularity according to the data sensitivity level, perform data desensitization processing, and generate usable data that meets privacy protection requirements.

[0009] The privacy computing layer includes a federated learning module, a homomorphic encryption module, and a resource scheduling module. The federated learning module is used to select a collaborative computing mode of horizontal federated learning, vertical federated learning, or federated transfer learning based on the data distribution characteristics to perform federated learning on available data, and configures a secure aggregation protocol to ensure the privacy of model parameter interaction. The homomorphic encryption module is built based on the TFHE library and is used to encrypt model gradients using multiple neural networks in the ciphertext domain, while configuring alternative encryption schemes resistant to quantum attacks. The dynamic resource scheduling module dynamically allocates computing nodes and optimizes data transmission paths based on the task data volume and computational complexity.

[0010] The blockchain layer includes a rights confirmation module, an audit module, and a layered evidence storage module. The rights confirmation module uses the SM2 algorithm to generate data fingerprints, combined with smart contracts for automated registration and transaction data ownership and usage rights. The audit module uses zero-knowledge proof technology to compress and store the entire process operation log. The layered evidence storage module stores core data and auxiliary data in categories.

[0011] Furthermore, the data cleaning module preprocesses the raw data as follows:

[0012] For medical text data: Construct a medical professional vocabulary database, use semantic similarity algorithm to calculate the similarity between suspected erroneous words and standard words in the text, and automatically correct typos when the similarity is greater than the similarity threshold;

[0013] For medical image data: the Canny edge detection algorithm is used to extract the region of interest in the image, the region of interest is cropped to a standardized size of 128×128 pixels, and background interference areas are automatically removed;

[0014] For financial transaction data: The Isolation Forest algorithm is used to detect anomalies in transaction records. An anomaly score threshold is set, and when the anomaly score of a transaction record exceeds the anomaly score threshold, it is marked as a suspicious transaction.

[0015] Furthermore, the data sensitivity grading standard is specifically divided into three levels: A, B, and C. The definitions and corresponding desensitization strategies for each level are as follows:

[0016] Level A: This category corresponds to highly sensitive data, including genetic data and core financial account information. It employs an enhanced desensitization strategy that combines the k-anonymity algorithm with the differential privacy algorithm, where the k value is ≥20 and the differential privacy budget ε is 0.1~0.3.

[0017] Level B: This corresponds to moderately sensitive data, including medical record details and transaction records, and employs a moderate desensitization strategy that combines generalization algorithms with partial field masking algorithms.

[0018] Level C: This corresponds to low-sensitivity data, including basic health information and publicly available business data, and employs basic desensitization strategies such as simple replacement algorithms or low-intensity generalization algorithms.

[0019] The dynamic desensitization module has several built-in desensitization algorithms, including at least k-anonymity algorithm, differential privacy algorithm, generalization algorithm and permutation algorithm, which automatically match and execute the corresponding desensitization strategy according to the data sensitivity level.

[0020] Furthermore, the three collaborative computing modes and secure aggregation protocol of the federated learning module include:

[0021] Horizontal federated learning mode: After each participant trains the model based on local data, it only uploads the encrypted model gradient parameters. The aggregation node aggregates and calculates the gradient parameters through a secure aggregation protocol, generates global model parameters, and feeds them back to each participant.

[0022] Vertical federated learning model: Features among participants are aligned using privacy alignment technology, and cross-institutional collaborative feature computation is performed using encrypted matrix multiplication;

[0023] Federated transfer learning model: Employs domain-adaptive algorithms to reduce data distribution differences among different participants;

[0024] The secure aggregation protocol is based on a combination of secret sharing and Paillier homomorphic encryption algorithm, which makes it impossible for any participant to infer the data of other parties during the aggregation process.

[0025] Furthermore, the dynamic resource scheduling module is also used to preset a data volume threshold of 10GB and a computational complexity evaluation index system; when the amount of data to be processed is >10GB, it is automatically allocated to GPU computing nodes; when the amount of data to be processed is ≤10GB, nodes are allocated according to the computational complexity evaluation results, wherein high-complexity tasks of deep learning are preferentially allocated to edge computing nodes, and low-complexity tasks of simple data statistics are allocated to ordinary CPU nodes.

[0026] With network latency, node computing power utilization, and data transmission bandwidth as optimization objectives, the data transmission path between nodes is optimized through selection, crossover, and mutation operations using a genetic algorithm.

[0027] Furthermore, the full-process operation log of the audit module includes at least the data upload time, data preprocessing details, participant identity information, calculation task configuration parameters, model parameter transmission records, calculation result output time, and data access operation records;

[0028] The audit module uses zero-knowledge proof technology to transform the entire process operation log into concise proof information and store it on the blockchain.

[0029] Furthermore, the classification and storage rules of the hierarchical evidence storage module are as follows:

[0030] Core data is stored directly on the blockchain main chain, while auxiliary data is stored in the IPFS distributed file system and retrieved and accessed through IPFS content addressing technology. The core data includes model parameters, data fingerprints, ownership registration information, and final calculation results, while the auxiliary data includes data preprocessing intermediate results, temporary calculation data, and non-critical operation logs.

[0031] When the auxiliary data is updated, an update record is automatically generated and stored on the blockchain to keep the core data consistent with the auxiliary data.

[0032] Secondly, the present invention provides a distributed privacy computing big data processing method as described above, the method comprising:

[0033] Each data provider uses the data cleaning module of the edge computing layer to preprocess the local raw data using corresponding dedicated preprocessing technologies to remove invalid information and interference data; and uses the dynamic desensitization module to determine the data sensitivity level according to the data sensitivity classification standard, match the corresponding desensitization algorithm and desensitization granularity to perform data desensitization processing, and generate usable data.

[0034] The dynamic resource scheduling module of the privacy computing layer obtains the available data and corresponding computing task information. Based on the task data size, computing task type and computing complexity evaluation results, it allocates computing tasks to suitable computing nodes according to a preset task allocation strategy and optimizes the data transmission path between nodes through a load balancing mechanism.

[0035] Based on the privacy-preserving computation layer's federated learning module, each participant selects a collaborative computation mode—horizontal federated learning, vertical federated learning, or federated transfer learning—according to the data distribution characteristics. Under the protection of the homomorphic encryption module, each participant only interacts with encrypted model parameters or feature data, and completes global model training or collaborative computation through a secure aggregation protocol to generate computation results.

[0036] The blockchain layer's ownership confirmation module registers data ownership information, generates data transaction records, and stores them on the blockchain; the auditing module compresses and stores the entire process operation log; and the layered evidence storage module stores core data and auxiliary data on the blockchain main chain and IPFS respectively, according to classification storage rules.

[0037] Furthermore, a dual verification mechanism is adopted to verify the calculation results. After successful verification, the calculation results are fed back to each participant, and the verification report is uploaded to the blockchain for evidence storage. If the verification fails, an exception handling mechanism is triggered to trace the abnormal link and re-execute the corresponding steps.

[0038] Furthermore, the dual verification mechanism includes:

[0039] The calculation result is decrypted and compared with a preset plaintext calculation benchmark value to verify the accuracy of the calculation result;

[0040] By using the hash values ​​of model parameters and computation logs stored on the blockchain, the consistency between the computation behavior of each participant and the data transmission is verified, so as to prevent data tampering or malicious behavior.

[0041] If both verifications pass, the final calculation result will be fed back to each participant, and the verification report will be uploaded to the blockchain for evidence storage; if either verification fails, the exception handling mechanism will be triggered, the exception process will be traced, and the corresponding calculation steps will be re-executed.

[0042] Compared with the prior art, the present invention has the following technical effects:

[0043] 1) By using local preprocessing in the edge computing layer, dynamic resource scheduling and optimization of homomorphic encryption algorithms in the privacy computing layer, and layered evidence storage mechanism in the blockchain layer, the system can achieve a second-level response when processing tens of millions of data samples. The data throughput and computing efficiency are significantly improved compared with traditional solutions, and the efficiency bottleneck of large-scale data privacy computing is solved.

[0044] 2) By adopting a dynamic de-identification strategy and a dual encryption mechanism, combined with a secure aggregation protocol, the risk of data leakage is reduced, while balancing privacy and data availability, thus significantly enhancing the strength of privacy protection.

[0045] 3) By leveraging the immutability and zero-knowledge proof technology of blockchain, automated registration of data ownership and traceability and auditability of the entire process are achieved, solving the problems of difficult data ownership confirmation and auditing in traditional solutions. At the same time, the layered evidence storage mechanism reduces storage costs. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a block diagram of a distributed privacy computing big data processing system according to an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of a preferred embodiment of a distributed privacy computing big data processing method according to an embodiment of the present invention.

[0049] Figure 3 This is a flowchart of another preferred embodiment of a distributed privacy computing big data processing method according to an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0051] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 This invention provides a distributed privacy-preserving computing big data processing system, comprising: an edge computing layer 1, a privacy computing layer 2, and a blockchain layer 3. These three layers work in sequence to form a complete privacy computing closed loop. The three-layer architecture is described in detail below.

[0052] Edge computing layer 1 is deployed on local nodes of each data provider to avoid the risk of privacy leakage caused by the cross-institutional transmission of raw data. It includes data cleaning module 11 and dynamic de-identification module 12.

[0053] The data cleaning module 11 uses natural language processing and image recognition technologies to perform targeted preprocessing on different types of raw data, removing invalid information and interference data to ensure data quality.

[0054] Specifically, for medical text data, a medical professional vocabulary database is constructed, and the similarity between suspected erroneous words and standard words in the text is calculated based on a semantic similarity algorithm. When the similarity is greater than a similarity threshold, typos are automatically corrected, with a typo correction accuracy of no less than 98.7%, effectively avoiding subsequent calculation deviations caused by text errors. For medical image data, the Canny edge detection algorithm is used to extract regions of interest (ROIs) in the images, and the ROIs are cropped to a standardized size of 128×128 pixels, automatically removing irrelevant background areas and reducing data redundancy. For financial transaction data, the Isolation Forest algorithm is used to screen suspicious transaction records, and anomaly detection is performed on the transaction records. Anomaly scoring threshold is set, and when the anomaly score of a transaction record exceeds the anomaly scoring threshold, it is marked as a suspicious transaction. The anomaly transaction identification accuracy is no less than 99.2%, providing high-quality data for subsequent risk control modeling.

[0055] The dynamic desensitization module 12 presets data sensitivity classification standards, matches the corresponding desensitization algorithm and desensitization granularity according to the data sensitivity classification, performs data desensitization processing, and generates usable data that meets privacy protection requirements.

[0056] This embodiment categorizes data into three levels: A, B, and C. Level A corresponds to highly sensitive data such as genetic data and core financial account information; Level B corresponds to moderately sensitive data such as medical record details and transaction logs; and Level C corresponds to low-sensitivity data such as basic health information and publicly available business data. The module incorporates 12 de-identification algorithms, including k-anonymity, differential privacy, generalization, and permutation algorithms, dynamically matching the appropriate algorithm and granularity based on the data's sensitivity level. For Level A data, a robust de-identification strategy combining k-anonymity and differential privacy algorithms is employed, with k set to no less than 20 and the differential privacy budget ε set to 0.1~0.3, ensuring the privacy and security of extremely sensitive data. For Level B data, a moderate de-identification strategy combining generalization and partial field masking algorithms is used, preserving data usability while protecting privacy. For Level C data, a basic de-identification strategy using simple permutation or low-intensity generalization algorithms is employed, balancing privacy protection and computational efficiency.

[0057] The edge computing layer of this invention transforms raw data into usable data that meets privacy protection requirements through targeted data cleaning and dynamic desensitization processing, improving data preprocessing efficiency by more than 50% while avoiding the risk of privacy leakage when raw data is transmitted across institutions.

[0058] In a preferred embodiment, the privacy computing layer 2 is the core computing unit, including a federated learning module 21, a homomorphic encryption module 22, and a resource scheduling module 23, to realize collaborative computing and privacy protection of cross-organizational data.

[0059] The Federated Learning Module 21 supports three collaborative computing modes: horizontal federated learning, vertical federated learning, and federated transfer learning, adapting to different data distribution scenarios. Users can select one of these modes based on data distribution characteristics to perform federated learning on available data. A secure aggregation protocol based on secret sharing and Paillier homomorphic encryption is configured to prevent any participant from inferring data from others during the aggregation process, ensuring privacy and security during model parameter interaction.

[0060] In the horizontal federated learning model, each participant trains its model based on local data and only uploads encrypted model gradient parameters. The aggregation node aggregates all gradient parameters using a secure aggregation protocol, generates global model parameters, and feeds them back to each participant, preventing the leakage of original data. In the vertical federated learning model, privacy alignment technology is used to align features among participants, and encrypted matrix multiplication is used to achieve collaborative computation of cross-institutional features, overcoming collaboration barriers caused by differences in data dimensionality. In the federated transfer learning model, a domain adaptation algorithm is introduced to reduce the impact of differences in data distribution among different participants on model performance, improving the model's adaptation speed in the target scenario by more than 40%. In the medical image classification scenario, the accuracy of the model trained by this module is only 1.2% lower than that of centralized training. In the financial risk control modeling scenario, the model's prediction accuracy is less than 0.5% lower than that of the centrally trained model.

[0061] The homomorphic encryption module 22 is built based on the TFHE library and supports gradient encryption of models using various neural networks in the ciphertext domain. This embodiment supports 20 core neural network operations in the ciphertext domain, including convolution, pooling, fully connected layer computation, and activation function operations. For convolution operations, an optimized Fast Fourier Transform algorithm is used to accelerate ciphertext domain computation. For pooling operations, a statistical computation method in the ciphertext domain is used to preserve key features, improving the computational efficiency of deep learning in the ciphertext domain by more than three times compared to traditional homomorphic encryption schemes. The module also includes the NTRU algorithm as a quantum attack-resistant alternative encryption scheme. This algorithm, implemented using lattice-based cryptography principles, can resist quantum attacks using Shor's algorithm, meeting security protection needs for the next 10 years and addressing the quantum attack risks faced by traditional encryption algorithms.

[0062] The dynamic resource scheduling module 23 dynamically allocates computing nodes and optimizes data transmission paths based on task attributes such as data volume and computational complexity, thereby improving overall computing efficiency. Its task allocation strategy is as follows: A preset data volume threshold of 10GB and a computational complexity evaluation index system are used. When the amount of data to be processed exceeds 10GB, the task is automatically allocated to GPU computing nodes to leverage the parallel computing capabilities of GPUs and improve processing speed. When the amount of data to be processed is less than or equal to 10GB, nodes are allocated based on the computational complexity evaluation results. High-complexity tasks, such as deep learning, are preferentially allocated to edge computing nodes to reduce core network transmission pressure, while low-complexity tasks, such as simple data statistics, are allocated to ordinary CPU nodes to reduce hardware costs. The load balancing mechanism is implemented based on a genetic algorithm, optimizing network latency, node computing power utilization, and data transmission bandwidth. Through selection, crossover, and mutation operations in the genetic algorithm, the data transmission paths between nodes are optimized, reducing overall network latency by more than 40% and increasing the balance of node computing power utilization to over 85%, effectively solving the problem of uneven resource allocation in large-scale data parallel computing.

[0063] The privacy computing layer of this invention combines multi-mode federated learning, efficient homomorphic encryption, and intelligent resource scheduling to improve cross-organizational collaborative computing efficiency by more than 3 times while ensuring data privacy and security, thus meeting the needs of real-time analysis.

[0064] In a preferred embodiment, the blockchain layer 3 provides data ownership confirmation and auditing functions, including an ownership confirmation module 31, an auditing module 32, and a layered evidence storage module 33, to ensure the traceability and credibility of data use.

[0065] The rights confirmation module 31 uses the SM2 algorithm to generate a unique data fingerprint, which is combined with the smart contract's automated registration and transaction data ownership and usage rights. The specific process is as follows:

[0066] After the data provider uploads the original data, the rights confirmation module 31 performs a hash operation on the original data using the SM2 algorithm to generate a unique and tamper-proof data fingerprint. Based on the smart contract's preset data ownership registration terms, usage right authorization rules, and transaction settlement mechanism, after the data user submits a usage application, the smart contract automatically verifies the applicant's qualifications. Upon successful verification, the registration and authorization of data usage rights are completed, and a corresponding transaction record is generated. The module's transaction on-chain response time is less than 3 seconds, supports millions of concurrent evidence storage requests, and the transaction record includes key information such as data fingerprint, participant identification, authorization period, and intended use, achieving clear definition of data ownership and automated transactions.

[0067] Audit module 32 uses zero-knowledge proof technology to compress and store the entire process operation log, ensuring that the operation behavior is auditable. The entire process operation log includes data upload time, data preprocessing details, participant identity information, computation task configuration parameters, model parameter transmission records, computation result output time, and data access operation records. The module uses zero-knowledge proof technology to compress the entire process operation log, transforming it into concise proof information stored on the blockchain, reducing storage space by more than 80% compared to traditional log storage solutions. During the audit process, the verification mechanism of zero-knowledge proof can complete the verification of the authenticity and compliance of the operation behavior without exposing the original log content, with a verification accuracy rate of 100%, resolving the contradiction between high log storage costs and privacy protection in traditional audit solutions.

[0068] The layered evidence storage module 33 categorizes and stores core data and auxiliary data, balancing trustworthiness and storage costs. The categorization rules are as follows: core data such as model parameters, data fingerprints, ownership registration information, and final calculation results are directly stored on the blockchain main chain, ensuring data immutability and high trustworthiness; auxiliary data such as intermediate results during data preprocessing, temporary calculation data, and non-critical operation logs are stored in the IPFS distributed file system, enabling efficient data retrieval and access through IPFS content addressing technology. The module also features a data synchronization mechanism; when auxiliary data is updated, update records are automatically generated and stored on the blockchain, ensuring consistency between core data and auxiliary data while reducing overall storage costs by more than 60%.

[0069] This blockchain layer, through its functions of rights confirmation, auditing, and hierarchical evidence storage, enables the entire process of data use to be traceable and auditable, reducing the risk of data leakage to below 0.01% while significantly reducing storage costs.

[0070] Furthermore, the distributed privacy computing system of this invention supports three deployment modes: public cloud, private cloud, and hybrid cloud, adapting to the IT architecture needs of different organizations. It provides a visual process orchestration interface, including a task configuration module, parameter adjustment components, a progress monitoring panel, and a result display area. Non-technical personnel can easily configure data sharing tasks through drag-and-drop operations, with task configuration taking no more than 10 minutes, improving the system's ease of use. The system also features an anomaly detection module, which identifies malicious participants or abnormal computational behavior through statistical outlier analysis algorithms. The false alarm rate for anomaly detection is less than 0.5%. When an anomaly is detected, an alarm mechanism is automatically triggered and the relevant computational task is terminated to prevent data leakage or model damage, further enhancing the system's security.

[0071] Based on the aforementioned distributed privacy computing big data processing system, this invention also provides a distributed privacy computing big data processing method, such as... Figure 2 As shown, the method includes the following steps:

[0072] S1. Each data provider uses the data cleaning module of the edge computing layer to preprocess the local raw data using corresponding dedicated preprocessing technologies to remove invalid information and interference data; and uses the dynamic desensitization module to determine the data sensitivity level according to the data sensitivity classification standard, match the corresponding desensitization algorithm and desensitization granularity to perform data desensitization processing, and generate usable data.

[0073] Specifically, for medical text data, a medical professional vocabulary database is constructed using natural language processing technology. The semantic similarity between suspected erroneous words in the text and standard words in the vocabulary database is calculated. When the similarity is greater than a preset threshold of 0.85, it is identified as a typo and automatically corrected. For medical image data, the Canny edge detection algorithm is used to extract the region of interest in the image. The region of interest is then cropped to a standardized size of 128x128 pixels to remove background interference. For financial transaction data, the Isolation Forest algorithm is used to detect anomalies in transaction records. An anomaly score threshold is set. When the anomaly score of a transaction record exceeds the threshold, it is marked as a suspicious transaction and stored separately.

[0074] After preprocessing, the dynamic desensitization module matches the corresponding desensitization algorithm according to the data sensitivity level to complete the data desensitization process: Level A data adopts an enhanced desensitization strategy combining the k-anonymity algorithm and the differential privacy algorithm; Level B data adopts a moderate desensitization strategy combining the generalization algorithm and the partial field masking algorithm; and Level C data adopts a basic desensitization strategy using the simple substitution algorithm or the low-intensity generalization algorithm, ultimately generating usable data that meets privacy protection requirements.

[0075] Step S1 effectively improves data quality and privacy security through targeted preprocessing and dynamic desensitization, laying the foundation for subsequent collaborative computing. The data preprocessing time is reduced by more than 40% compared to traditional solutions.

[0076] S2. The dynamic resource scheduling module of the privacy computing layer obtains the available data and the corresponding computing task information. Based on the task data volume, computing task type and computing complexity evaluation results, it allocates the computing tasks to the appropriate computing nodes according to the preset task allocation strategy, and optimizes the data transmission path between nodes through the load balancing mechanism.

[0077] Specifically, the dynamic resource scheduling module of the privacy computing layer obtains preprocessed task information, including data size, computation task type, and complexity assessment results. According to the preset task allocation strategy, when the amount of data to be processed is greater than 10GB, the task is automatically allocated to GPU computing nodes; when the amount of data to be processed is less than or equal to 10GB, nodes are allocated according to the computational complexity assessment results, with high-complexity deep learning tasks preferentially allocated to edge computing nodes and low-complexity simple data statistics tasks allocated to ordinary CPU nodes.

[0078] Meanwhile, by using a load balancing mechanism based on genetic algorithms, with network latency, node computing power utilization, and data transmission bandwidth as optimization targets, the data transmission path between nodes is optimized, resulting in an overall network latency reduction of more than 40% and a node computing power utilization balance improvement of more than 85%.

[0079] Step S2 enables intelligent allocation and load balancing of computing resources, avoiding resource waste, improving overall computing efficiency, and reducing task response time from 15 seconds in the traditional solution to less than 3 seconds.

[0080] S3. Based on the privacy-preserving computation layer, each participant selects a collaborative computation mode of horizontal federated learning, vertical federated learning, or federated transfer learning according to the data distribution characteristics of the federated learning module. Under the protection of the homomorphic encryption module, each participant only interacts with encrypted model parameters or feature data, and completes global model training or collaborative computation through a secure aggregation protocol to generate computation results.

[0081] Each participant selects the corresponding collaborative computing mode based on the federated learning module: when the data features are the same but the samples are different, the horizontal federated learning mode is selected; when the data samples are the same but the features are different, the vertical federated learning mode is selected; and when the data distributions are significantly different, the federated transfer learning mode is selected.

[0082] Under the encryption protection of the homomorphic encryption module, each participant only interacts with the encrypted model parameters or feature data: After obtaining the gradient parameters by training the model locally, each participant encrypts the gradient parameters using the Paillier homomorphic encryption algorithm to generate encrypted gradients; each participant sends the encrypted gradients to the aggregation node, and at the same time, sends its own encryption key fragments to other participants through secret sharing technology; the aggregation node performs aggregation calculations on all encrypted gradients to generate a global encrypted gradient; the aggregation node sends the global encrypted gradient to each participant, and each participant reconstructs the key through key fragmentation, decrypts the global encrypted gradient, obtains the global gradient parameters, and completes one round of model aggregation.

[0083] Repeat the above process until the model training reaches the preset accuracy requirement or the number of iterations threshold, and generate the final calculation result.

[0084] This step, through the collaboration of federated learning and homomorphic encryption, achieves "data usable but not visible," ensuring privacy and security while keeping the accuracy loss of cross-institutional collaborative computing within 1.2%, thus meeting the needs of practical applications.

[0085] S4, the blockchain layer's ownership confirmation module registers data ownership information, generates data transaction records, and stores them on the blockchain; the audit module compresses and stores the entire process operation log; the layered evidence storage module stores core data and auxiliary data on the blockchain main chain and IPFS respectively according to the classification storage rules.

[0086] The blockchain layer's ownership confirmation module registers data ownership information: When a data provider uploads original data, the module calculates the hash value of the original data using the SM2 algorithm, generates a unique data fingerprint, and associates and stores the data fingerprint, data provider identity information, and data description information to complete ownership registration; After the data user submits a usage application, specifying the purpose, duration, and payment consideration, the smart contract automatically verifies the user's qualifications and the compliance of the application information. Once the verification is successful, the data provider and user reach a smart contract transaction, the smart contract automatically transfers the payment consideration to the data provider's account, and grants the user the right to use the data for the corresponding period, generating a transaction record containing the data fingerprint, identity information of both parties, authorization terms, and transaction amount, and uploading the transaction record to the blockchain.

[0087] The auditing module compresses and stores the operation logs of the entire calculation process, and uses zero-knowledge proof technology to transform the original log data into concise proof information and store it on the blockchain; the layered evidence storage module directly stores core data such as model parameters, data fingerprints, ownership registration information, and final calculation results on the main blockchain, stores auxiliary data in the IPFS distributed file system, and ensures the consistency between core data and auxiliary data through a data synchronization mechanism.

[0088] This step enables clear definition of data ownership and traceability of the entire process, improving the efficiency of tracing data usage disputes by more than 80% while reducing storage costs.

[0089] In a preferred embodiment, such as Figure 3 As shown, the present invention further includes step S5: verifying the calculation result using a dual verification mechanism; after the verification is passed, the calculation result is fed back to each participant, and the verification report is uploaded to the blockchain for evidence storage; if the verification fails, an exception handling mechanism is triggered to trace the exception and re-execute the corresponding steps.

[0090] The first layer of verification in this embodiment is the accuracy verification of the calculation result. After decrypting the ciphertext calculation result, it is compared with the preset plaintext calculation benchmark value, and the error is controlled within 0.5%. The second layer of verification is the data consistency verification. Through the model parameter hash value and calculation process log stored in the blockchain, the consistency of the calculation behavior of each participant and the data transmission is verified to ensure that there is no data tampering or malicious behavior.

[0091] Step S5 ensures the accuracy and reliability of the calculation results, keeping the error rate below 0.1% and improving the system's credibility.

[0092] The following examples illustrate the distributed privacy computing big data processing method of the present invention in specific application scenarios.

[0093] I. Cross-hospital diagnosis of medical imaging

[0094] 1. Data Preprocessing and Dynamic Desensitization: Hospital A uses the data cleaning module of the edge computing layer to extract the region of interest from the local CT image data using the Canny edge detection algorithm, and crops it to a standardized size of 128x128 pixels to remove background interference. The dynamic desensitization module identifies that the image data contains patient privacy information and classifies it as Class B sensitive data. A desensitization strategy combining generalization algorithm and partial field masking algorithm is used to remove privacy fields such as patient name and ID number.

[0095] 2. Dynamic allocation of computing resources: The dynamic resource scheduling module obtains task information. The task data volume is 15GB, which belongs to the high complexity of deep learning tasks. It is automatically allocated to the GPU edge node, and the data transmission path between the three hospital nodes A, B and C is optimized through genetic algorithm, reducing network latency by 45%.

[0096] 3. Federated Learning and Homomorphic Encryption Collaborative Computation: The three hospitals chose a horizontal federated learning model to jointly train the lung cancer screening model. Each hospital trained the model based on locally anonymized CT image data, encrypted the model gradient using the Paillier homomorphic encryption algorithm, and sent the encrypted gradient to the aggregation node. The aggregation node completed gradient aggregation through a secure aggregation protocol, generated a global gradient, and fed it back to each hospital. This process was repeated iteratively for 20 rounds.

[0097] 4. Data Ownership Confirmation and Layered Evidence Storage: The ownership confirmation module generates a unique fingerprint for CT image data using the SM2 algorithm, and three hospitals complete the data ownership registration; the audit module compresses and stores the operation logs of the entire process, including data preprocessing, model training, and gradient transfer; the layered evidence storage module stores model parameters, data fingerprints, and ownership information on the blockchain main chain, and stores intermediate gradient data during the training process in IPFS.

[0098] 5. Results Validation and Feedback: The trained model was validated on the test set of Hospital D. The model accuracy reached 94.3%, which was only 0.7% lower than that of centralized training. The calculation error was 0.3%, which meets the needs of medical diagnosis. The validation report was uploaded to the blockchain for evidence storage, and the final diagnostic model was fed back to the participating hospitals for clinical diagnosis.

[0099] II. Joint Modeling of Financial Risk Control

[0100] 1. Data Preprocessing and Dynamic Desensitization: A bank preprocessed 100,000 credit card transaction records. The isolated forest algorithm in the data cleaning module identified 123 suspicious transactions (accuracy rate of 99.2%). The dynamic desensitization module determined that the transaction records contained core account information and belonged to Class A sensitive data. A desensitization strategy combining k-anonymity (k=25) and differential privacy (ε=0.2) was adopted to ensure data privacy and security.

[0101] 2. Dynamic allocation of computing resources: The task data volume is 8GB, which is a high-complexity risk control modeling task. The dynamic resource scheduling module allocates it to the nearest GPU edge node, reducing the response time from 15 seconds to 3 seconds.

[0102] 3. Federated Learning and Homomorphic Encryption Collaborative Computation: The bank and multiple partner institutions chose a vertical federated learning model, achieved customer feature alignment through privacy alignment technology, and used ciphertext matrix multiplication of the homomorphic encryption module to achieve cross-institutional feature collaborative computation. The user credit scoring model was trained in the ciphertext domain, and the error between the calculation result and the plaintext calculation was 0.4%.

[0103] 4. Data ownership confirmation and hierarchical evidence storage: The ownership confirmation module completes the registration of ownership of transaction data. After the cooperating institution pays the data usage fee through smart contract, it obtains the right to use the data, and the transaction record is stored on the blockchain. The hierarchical evidence storage module puts the final credit scoring model and ownership information on the blockchain and stores the intermediate computational data in IPFS.

[0104] 5. Result Validation and Feedback: The accuracy of the credit scoring model is validated. The model's prediction accuracy reaches 98.5%, meeting the requirements of financial risk control. After validation, the model is used for credit card risk assessment by various institutions. The assessment results are fed back to all participants in real time, and the assessment records are uploaded to the blockchain audit.

[0105] In summary, this invention, through the deep integration of technologies such as federated learning, homomorphic encryption, and blockchain, constructs a three-layer distributed privacy computing system: an edge computing layer, a privacy computing layer, and a blockchain layer. It proposes corresponding computational methods and solves the efficiency, security, and auditing challenges of existing privacy computing in big data processing. The system boasts advantages such as high computational efficiency, strong privacy protection, good auditability, and wide adaptability. Third-party testing and industry pilot verification have shown that it has reached internationally leading levels in terms of functional completeness, performance indicators, and security strength, making it suitable for large-scale commercial application and effectively promoting cross-institutional data sharing and value release in fields such as healthcare, finance, and government.

[0106] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0107] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0108] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0109] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed privacy computing big data processing system, characterized in that, The system comprises an edge computing layer, a privacy computing layer and a blockchain layer, the edge computing layer is deployed locally at each data provider and comprises a data cleaning module and a dynamic de-sensitization module, the data cleaning module is used for pre-processing original data to remove invalid information and interference data, and the dynamic de-sensitization module is used for matching corresponding de-sensitization algorithms and de-sensitization granularity according to data sensitivity grading to perform data de-sensitization processing and generate available data meeting privacy protection requirements; The privacy computing layer comprises a federated learning module, a homomorphic encryption module and a resource scheduling module, the federated learning module is used for selecting a collaborative computing mode of horizontal federated learning, vertical federated learning or federated transfer learning according to data distribution characteristics to perform federated learning on available data, and configuring a secure aggregation protocol to ensure model parameter interaction privacy; the homomorphic encryption module is constructed based on a TFHE library and is used for encrypting model gradients of multiple neural networks in a ciphertext domain while configuring an anti-quantum attack alternative encryption scheme; and the dynamic resource scheduling module dynamically allocates computing nodes and optimizes data transmission paths according to task data volume and computing complexity. The blockchain layer comprises a right confirmation module, an audit module and a hierarchical storage module; the right confirmation module generates data fingerprints by using an SM2 algorithm and automatically registers and transacts data ownership and usage rights in combination with a smart contract; the audit module compresses and stores full-process operation logs by using zero-knowledge proof technology; and the hierarchical storage module classifies and stores core data and auxiliary data.

2. The distributed privacy computing big data processing system of claim 1, wherein, The data cleaning module specifically pre-processes original data as follows: For medical text data: a medical professional vocabulary library is constructed, and a semantic similarity algorithm is used to calculate the similarity of suspected wrong words in the text and standard words, and when the similarity is greater than a similarity threshold, the wrong words are automatically corrected; For medical image data: a Canny edge detection algorithm is used to extract an image region of interest, and the region of interest is cropped to a standardized size of 128x128 pixels to automatically remove background interference regions; For financial transaction data: an isolation forest algorithm is used to detect abnormalities in transaction records, and an abnormal score threshold is set, and when the abnormal score of a transaction record exceeds the abnormal score threshold, the transaction record is marked as a suspicious transaction.

3. The distributed privacy computing big data processing system of claim 1, wherein, The data sensitivity grading standard is specifically divided into three levels of A, B and C, and the definition of each level and the corresponding de-sensitization strategy are as follows: A level: corresponding to high-sensitive data including genetic data and core financial account information, a reinforced de-sensitization strategy combining a k-anonymity algorithm and a differential privacy algorithm is adopted, wherein k is greater than or equal to 20, and the differential privacy budget epsilon is 0.1-0.3; B level: corresponding to medium-sensitive data including medical record details and transaction flow, a moderate de-sensitization strategy combining a generalization algorithm and a partial field shielding algorithm is adopted; C level: corresponding to low-sensitive data including basic health information and public business data, a basic de-sensitization strategy of a simple substitution algorithm or a low-intensity generalization algorithm is adopted; The de-sensitization algorithms built in the dynamic de-sensitization module include at least a k-anonymity algorithm, a differential privacy algorithm, a generalization algorithm and a substitution algorithm, and the corresponding de-sensitization strategy is automatically matched and executed according to the data sensitivity level.

4. The distributed privacy computing big data processing system of claim 1, wherein, The three collaborative computing modes and the secure aggregation protocol of the federal learning module include: Horizontal federal learning mode: after each participant trains a model based on local data, only the encrypted model gradient parameters are uploaded, the aggregation node performs aggregation and calculation on the gradient parameters through a secure aggregation protocol, generates global model parameters, and feeds back to each participant; Vertical federal learning mode: align the features between participants through privacy alignment technology, and perform cross-institution feature collaborative calculation using encrypted matrix multiplication; Federal transfer learning mode: use domain adaptation algorithm to reduce the difference in data distribution between different participants; The secure aggregation protocol combines secret sharing and Paillier homomorphic encryption algorithm, so that any participant cannot infer other participant's data during the aggregation process.

5. The distributed privacy computing big data processing system of claim 1, wherein, The dynamic resource scheduling module is also used to preset a data volume threshold of 10GB and a computing complexity evaluation index system; when the amount of data to be processed is >10GB, it is automatically allocated to a GPU computing node; when the amount of data to be processed is ≤10GB, it is allocated to a node according to the evaluation results of the computing complexity, wherein high-complexity tasks of the deep learning type are preferentially allocated to edge computing nodes, and low-complexity tasks of the simple data statistics type are allocated to ordinary CPU nodes; Taking network delay, node computing power utilization rate, and data transmission bandwidth as optimization objectives, the data transmission path between nodes is optimized through selection, crossover, and mutation operations of genetic algorithm.

6. The distributed privacy computing big data processing system of claim 1, wherein, The audit module's full-process operation log includes at least data upload time, data preprocessing details, participant identity information, computing task configuration parameters, model parameter transmission record, computing result output time, and data access operation record; The audit module uses zero-knowledge proof technology to convert the full-process operation log into concise proof information and stores it in the blockchain.

7. The distributed privacy computing big data processing system of claim 1, wherein, The classification storage rule of the hierarchical storage module is: Core data is directly stored in the main chain of the blockchain, auxiliary data is stored in the IPFS distributed file system, and is retrieved and accessed through IPFS content addressing technology; the core data includes model parameters, data fingerprints, ownership registration information, and final computing results, and the auxiliary data includes data preprocessing intermediate results, temporary computing data, and non-critical operation logs; When the auxiliary data is updated, an update record is automatically generated and stored in the chain to keep the core data and the auxiliary data consistent.

8. The distributed privacy computing big data processing method of any one of claims 1 to 7, wherein, The method comprises: Each data provider uses corresponding exclusive preprocessing technology to preprocess local raw data through the data cleaning module of the edge computing layer, removes invalid information and interference data; and determines the data sensitivity level through the dynamic desensitization module according to the data sensitivity grading standard, matches the corresponding desensitization algorithm and desensitization granularity to perform data desensitization processing, and generates available data; The dynamic resource scheduling module of the privacy computing layer obtains the available data and corresponding computing task information, distributes the computing task to the adaptive computing node according to the preset task distribution strategy according to the task data size, computing task type, and computing complexity evaluation result, and optimizes the data transmission path between nodes through the load balancing mechanism; Each participant selects a collaborative computing mode of horizontal federated learning, vertical federated learning or federated transfer learning according to the data distribution characteristics based on the federated learning module of the privacy computing layer; under the protection of the homomorphic encryption module, each participant only interacts with the encrypted model parameters or feature data, completes global model training or collaborative computing through a secure aggregation protocol, and generates a computing result; The right module of the blockchain layer registers data ownership information, generates data transaction records and stores them on the chain; the audit module compresses and stores the whole-process operation log; the hierarchical storage module stores core data and auxiliary data in the blockchain main chain and IPFS respectively according to the classification storage rule.

9. The distributed privacy computing big data processing method of claim 8, wherein, The computing result is verified by using a double verification mechanism, and the computing result is fed back to each participant after the verification is passed, and the verification report is uploaded to the blockchain storage; if the verification fails, an abnormal handling mechanism is triggered to trace the abnormal link and re-execute the corresponding steps.

10. The distributed privacy computing big data processing method of claim 9, wherein, The double verification mechanism comprises: The computing result is decrypted and compared with the preset plaintext computing reference value to verify the accuracy of the computing result; The consistency of the computing behavior and data transmission of each participant is verified through the model parameter hash value and the computing process log stored in the blockchain, so that there is no data tampering or malicious behavior; If both verifications pass, the final computing result is fed back to each participant, and the verification report is uploaded to the blockchain storage; if any verification fails, an abnormal handling mechanism is triggered to trace the abnormal link and re-execute the corresponding computing steps.

Images