An engine system based on integrated privacy federated computing capabilities

Abstract

This invention provides an engine system based on integrated privacy-preserving federated computing capabilities, comprising: a federated computing coordinator receiving a federated computing task, breaking it down into multiple sub-queries, and distributing them to deployed local agent processes; the local agent processes executing the sub-queries to obtain local intermediate results, performing privacy processing on the local intermediate results based on a privacy computing protocol to obtain hidden intermediate results, and sending them to the federated computing coordinator; the federated computing coordinator using a federated aggregation algorithm to aggregate the hidden intermediate results to obtain a global computing result, which is then sent to a version control system; a blockchain module acquiring aggregation operation information, generating an operation summary based on the aggregation operation information, and writing it into the blockchain ledger; and the version control system generating new version data based on the global computing result, recording the data version evolution path based on a content-addressed storage structure and a copy-on-write mechanism. This application enables privacy-preserving collaborative computing of distributed spatiotemporal data, avoiding the risks of centralized processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an engine system based on integrated privacy federated computing capabilities. Background Technology

[0002] With the widespread application of sensitive spatiotemporal data such as geospatial data, remote sensing imagery, and meteorological data in fields such as smart cities, environmental monitoring, and emergency management, the demand for cross-institutional and cross-regional data collaborative analysis and modeling is increasing. However, existing technology platforms generally suffer from the following problems when processing such data.

[0003] Existing technologies such as Google Earth Engine, while possessing powerful centralized data processing capabilities, require data to be uploaded to a central server for processing. Their lack of native privacy protection mechanisms makes it difficult to guarantee data sovereignty, posing risks of data leakage and compliance. On the other hand, platforms like Open Data Cube, while supporting distributed deployment, have limited capabilities in version control, data traceability, and access management, making it difficult to support complex data lifecycle management. Therefore, existing technologies require sharing raw data, resulting in ineffective guarantees of data security, compliance, and auditability in the context of global data collaboration. Summary of the Invention

[0004] In order to enable privacy-preserving collaborative computing of distributed spatiotemporal data and help avoid the processing risks brought about by centralized computing, this application provides an engine system based on integrated privacy federated computing capabilities.

[0005] Firstly, this embodiment provides an engine system based on integrated privacy-preserving federated computing capabilities, the system comprising: The federated computing coordinator is used to receive joint computing tasks from various participants, break down the joint computing tasks into multiple sub-queries, and distribute them to the local agent processes deployed by each participant. Each local agent process executes the subquery locally to obtain local intermediate results, and performs covert processing on the local intermediate results based on a preset privacy computing protocol to obtain covert intermediate results, which are then sent to the federated computing coordinator. The federated computing coordinator also uses a federated aggregation algorithm to aggregate the hidden intermediate results to obtain global computing results, and sends them to the version control system. The blockchain module obtains the aggregation operation information corresponding to the aggregation process, generates an operation summary based on the aggregation operation information, and writes it into the blockchain ledger. The version control system generates new version data based on the global calculation results and records the data version evolution path based on the content-addressable storage structure and copy-on-write mechanism.

[0006] In some embodiments, the privacy computation protocol includes at least one of secure multi-party computation, homomorphic encryption, and differential privacy; The local agent process is configured to dynamically select and configure the privacy computing protocol based on the task type and constraints of the participants in the federated computing task. The federated aggregation algorithm is either a federated averaging algorithm or a secure aggregation protocol.

[0007] In some embodiments, generating an operation summary based on the aggregated operation information and writing it into the blockchain ledger includes: A cryptographic hash algorithm is invoked to generate an operation digest of the aggregated operation information; Construct a Merkle tree containing a summary of the operations; The root hash, timestamp, and operator identity information of the Merkle tree are written into a blockchain ledger based on a consortium blockchain via a smart contract.

[0008] In some embodiments, the system further includes: The security access control subsystem performs identity verification and authorization control on the federated computing task according to the preset permission policy, and uses an anonymous authentication protocol to hide the identity information of the participants during the authorization process; The dataset management interface provides a visual display and interactive function for version data, and supports users to access version information and perform data operations through a unified interface. The Handle identification system generates a unique persistent identifier for the operation digest and associates it with a parsing record, wherein the parsing record includes at least one of version location, historical version index and metadata.

[0009] In some embodiments, the security access control subsystem supports role-based access control and attribute-based access control, and works in conjunction with the blockchain module to write access authorization records into the blockchain ledger.

[0010] In some embodiments, the version control system also supports a multi-branch coexistence and differential comparison mechanism for tracking changes between versions, wherein the changes include at least one of data field differences, binary block differences, and geospatial data structured differences.

[0011] In some of these embodiments, the local agent process uses a high-performance remote procedure call framework for inter-node communication when executing the subquery, and combines it with a transport layer security protocol.

[0012] In some of these embodiments, when generating new version data, the version control system uses a copy-on-write mechanism to save only the difference blocks from the previous version and maintains the sharing of unmodified data through pointer references.

[0013] In some embodiments, the parsing record associated with the unique persistent identifier generated by the Handle identification system also includes a content fingerprint for verifying data integrity and version consistency.

[0014] In some embodiments, the secure access control subsystem provides an anonymous authentication protocol including at least one of ring signatures, group signatures, or zero-knowledge proofs.

[0015] By employing the aforementioned system, this application's system includes a federated computing coordinator, which receives joint computing tasks from various participants, breaks down the joint computing tasks into multiple sub-queries, and distributes them to the local agent processes deployed by each participant. Each local agent process executes the sub-queries locally to obtain local intermediate results, and performs anonymous processing on the local intermediate results based on a preset privacy computing protocol to obtain anonymous intermediate results, which are then sent to the federated computing coordinator. The federated computing coordinator also uses a federated aggregation algorithm to aggregate the anonymous intermediate results to obtain global computing results, which are then sent to the version control system. The blockchain module obtains the aggregation operation information corresponding to the aggregation processing, generates an operation summary based on the aggregation operation information, and writes it to the blockchain ledger. The version control system generates new version data based on the global computing results and records the data version evolution path based on a content-addressed storage structure and a copy-on-write mechanism. This application can realize privacy-preserving collaborative computing of distributed spatiotemporal data, helping to avoid the processing risks brought about by centralized processing. Attached Figure Description

[0016] Figure 1 This is an overall architecture diagram of an engine system based on integrated privacy federated computing capabilities, as provided in this application.

[0017] Figure 2 This is a schematic diagram of an engine system connection based on integrated privacy federated computing capabilities provided in this application.

[0018] Figure 3 This is a flowchart of the method provided in this application for generating operation summaries based on aggregated operation information and writing them into the blockchain ledger. Detailed Implementation

[0019] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but is consistent with the broadest scope claimed in this application.

[0020] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0021] The specific application scenario of this application is a distributed privacy computing platform for spatiotemporal data sharing and analysis. It is specifically suitable for secure collaborative computing and version management of sensitive spatiotemporal data such as geospatial data, remote sensing images, and meteorological data. It can be widely used in data collaboration and secure modeling environments among research institutions, government departments, enterprises, and international organizations in fields such as smart cities, environmental monitoring, emergency management, agricultural remote sensing, and public health.

[0022] In the field of smart cities, urban management departments need to integrate spatiotemporal data from multiple departments such as transportation, energy, and environmental protection for comprehensive analysis to optimize urban resource allocation and traffic scheduling. For example, road traffic data from the transportation department, spatiotemporal distribution data of electricity consumption from the energy department, and air quality monitoring data from the environmental protection department are all sensitive data. Due to data security and sovereignty considerations, each department is unwilling to share the raw data with other departments. However, the engine system proposed in this application can achieve collaborative computing of data from various departments without sharing the raw data, providing data support for the refined management of smart cities.

[0023] In environmental monitoring, environmental monitoring agencies in different regions possess spatiotemporal data such as meteorological data, hydrological data, and pollutant concentration monitoring data for their respective jurisdictions. When it is necessary to trace the source and assess the impact of cross-regional environmental pollution events, it is necessary to integrate these scattered data for analysis. Existing technologies often require agencies to share raw data, which poses a risk of data leakage. However, the engine system proposed in this application can ensure data privacy and security, realize collaborative analysis of cross-regional environmental data, and provide a scientific basis for environmental decision-making.

[0024] In the field of emergency management, after sudden events such as earthquakes and floods, it is necessary to quickly integrate spatiotemporal data from multiple departments, including emergency response, geology, meteorology, and transportation. This includes geographic information of the disaster area, meteorological early warning data, and traffic status data, in order to formulate effective emergency rescue plans. The system proposed in this application can quickly achieve collaborative computation of data from multiple departments while protecting data privacy, thus gaining valuable time for emergency rescue and reducing disaster losses.

[0025] In the field of agricultural remote sensing, agricultural research institutions, agricultural production enterprises, and government agricultural management departments possess a large amount of agricultural remote sensing data, such as remote sensing images of crop growth and soil moisture data. Through the system proposed in this application, this data can be collaboratively analyzed while protecting the data privacy of each institution, enabling functions such as crop growth monitoring, yield prediction, and pest and disease early warning, thus providing precise guidance for agricultural production.

[0026] In the field of public health, medical institutions and disease control centers in different regions possess their own sensitive spatiotemporal data, such as patient case data and infectious disease surveillance data. When a major infectious disease outbreak occurs, it is necessary to integrate this data for epidemic trend analysis, transmission path tracing, and evaluation of the effectiveness of prevention and control measures. This proposed system can guarantee data privacy and enable collaborative computing of public health data across regions and institutions, providing strong support for epidemic prevention and control decision-making.

[0027] Figure 1 This is a block diagram of the overall architecture of an engine system based on integrated privacy-preserving federated computing capabilities, as provided in this application. Figure 1 As shown, the engine system based on integrated privacy federated computing capabilities can be divided into a federated computing layer, a core service layer, a data processing layer, a storage module, and a data abstraction layer in terms of overall architecture. Each layer collaborates with the others to jointly realize privacy-preserving collaborative computing and version management of distributed spatiotemporal data.

[0028] The federated computing system comprises multiple federated nodes, including National Research Node A, National Research Node B, and National Research Node C. Each node deploys a data broker and corresponding privacy computing engine and protocols. Nodes communicate through secure channels to ensure data transmission security. The core service layer mainly includes modules such as data broker, spatiotemporal query engine, access control policy management, and metadata management. It is responsible for core services such as data query optimization, processing, and access control, providing core service support for upper-layer computing and ensuring compliant and efficient data processing. The data processing layer covers functions such as spatiotemporal indexing, ETL channels, data version control, persistent storage, and version tracking, realizing structured processing and version management of raw data, laying the foundation for subsequent storage and applications, and ensuring the continuity and traceability of data processing. The storage module includes blockchain notarization, version control system, object storage, and spatiotemporal database, respectively implementing tamper-proof notarization, version management, general object storage, and dedicated storage for spatiotemporal data, providing a secure and reliable storage environment for data and operation records. The data abstraction layer includes model fingerprinting, data abstraction, a handle system, unique persistent identifier generation, and parsing services, as well as a presentation space, interactive applications, a dataset repository, and a model repository. It realizes the abstract representation, unique identification, and parsing of data, improving the understandability and accessibility of data.

[0029] Figure 2 This is a block diagram of an engine system based on integrated privacy-preserving federated computing capabilities, as provided in this application. Figure 2 As shown, an engine system based on integrated privacy federated computing capabilities includes a federated computing coordinator, a local agent process, a blockchain module, a version control system, a secure access control subsystem, a dataset management interface, and a handle identification system.

[0030] The federated computing coordinator is the core coordinating body for federated computing tasks in the entire system. Its main function is to receive federated computing tasks from all participants, break down the federated computing tasks into multiple sub-queries, and distribute them to the local agent processes deployed by each participant.

[0031] During task decomposition, the federated computing coordinator formulates a global query plan based on factors such as the specific requirements of the joint computing task, data distribution, and the computing capabilities of each participant. This plan rationally breaks down complex joint computing tasks into multiple relatively independent sub-queries. For example, in a cross-regional joint computing task for crop yield prediction, the federated computing coordinator will break down the task into sub-query tasks targeting data from different regions, ensuring that each sub-query task can be efficiently executed locally by its corresponding participant.

[0032] Each local agent process executes the issued subquery locally, obtains local intermediate results, and performs privacy processing on these intermediate results based on a preset privacy computing protocol to obtain anonymous intermediate results before sending them to the federated computing coordinator. During subquery execution, the local agent employs a high-performance remote procedure call framework for inter-node communication, combined with transport layer security protocols to ensure the confidentiality and authentication of inter-node communication, preventing data theft or tampering during transmission.

[0033] Privacy-preserving computation protocols include at least one of secure multi-party computation, homomorphic encryption, and differential privacy. The local agent process is configured to dynamically select and configure the privacy-preserving computation protocol based on the task type of the federated computation task and the constraints of the participants.

[0034] When participants require precise calculations and the data dimensions are clearly defined, secure multi-party computation should be prioritized. For example, in collaborative computation of spatiotemporal data in the financial field, each participant needs precise calculation results, and the data dimensions are clearly defined. In this case, a secure multi-party computation protocol is adopted, allowing each participant to complete the computation task through writing without disclosing the original data, thus ensuring data privacy and the accuracy of the calculation results.

[0035] Homomorphic encryption is used when there is significant computational burden on one party and partial operations need to be performed on encrypted data. For example, in a medical spatiotemporal data system analysis project of a large medical institution, one institution has an extremely large amount of data, resulting in heavy computational burden. Furthermore, it needs to perform encrypted operations on some of the data to protect patient privacy. In this case, by choosing a homomorphic encryption protocol, the institution can encrypt the data locally before sending it to other participants. These other participants can then perform partial operations on the encrypted data, effectively reducing the computational burden on the first institution while protecting data privacy.

[0036] Differential privacy is used when a task has high performance requirements and only requires probabilistic privacy protection. For example, when performing large-scale statistical analysis on spatiotemporal data of urban traffic flow, the task has high computational performance requirements and does not require absolutely precise privacy protection. It only needs to ensure that the statistical characteristics of the data do not reveal individual privacy. In this case, a differential privacy protocol is used to add an appropriate amount of noise to the data, thereby achieving probabilistic privacy protection while meeting performance requirements.

[0037] After receiving the hidden intermediate results from the local agent processes of each participating party, the federated computing coordinator uses a federated aggregation algorithm to aggregate these results and obtain the global computation result, which is then sent to the version control system. The federated aggregation algorithm can be either a federated averaging algorithm or a secure aggregation protocol. In the federated averaging algorithm, the coordinator performs a weighted average of the model parameters (i.e., the hidden intermediate results) from each participating party to obtain the global model parameters (i.e., the global computation result). For example, in a task involving collaborative training of a medical diagnostic model by multiple institutions, each institution trains its model based on local data to obtain model parameters (i.e., hidden intermediate results). The coordinator uses a federated averaging algorithm to perform a weighted average of these parameters to generate global medical diagnostic model parameters, enabling collaborative model training while protecting the local data privacy of each institution.

[0038] Secure aggregation protocols employ more sophisticated encryption techniques and collaborative mechanisms to ensure that the confidential intermediate results of each participant are not leaked during the aggregation process, further enhancing the security of the aggregation process. For example, in collaborative computing of geospatial data involving national secrets, where data privacy and security requirements are extremely high, the federated dissemination coordinator uses a secure aggregation protocol to aggregate the confidential intermediate results of each participant through encrypted computing in collaboration with multiple parties, ensuring that data is not leaked during the aggregation process and safeguarding information security.

[0039] By working together with the federated computing coordinator and local agent processes, and combining dynamically selected privacy-preserving distributed protocols and appropriate federated aggregation algorithms, collaborative computing of distributed spatiotemporal data can be achieved without sharing the original data. This effectively protects data privacy and avoids the risk of data leakage caused by centralized processing.

[0040] The main function of the blockchain model is to obtain the aggregation operation information corresponding to the aggregation processing of the federated computing coordinator, generate operation summaries based on the aggregation operation information and write them into the blockchain ledger, so as to realize the immutable record of the aggregation operation and provide support for the verifiable storage of data and model processing flow. Figure 3 This is a flowchart illustrating the method provided in this application for generating operation summaries based on aggregated operation information and writing them into the blockchain ledger. For example... Figure 3 As shown, generating an operation summary based on aggregated operation information and writing it into the blockchain ledger includes the following steps: Step S101: Invoke the cryptographic hash algorithm to generate an operation digest of the aggregated operation information.

[0041] Step S102: Construct a Merkle tree containing the operation summary.

[0042] Step S103: Write the root hash, timestamp, and operator identity information of the Merkle tree into the blockchain ledger based on the consortium blockchain through a smart contract.

[0043] Cryptographic hash algorithms possess oneness and uniqueness; different input data will generate different hash values, i.e., operation digests, and the original input data cannot be deduced from the hash value. By hashing the aggregation operation information to generate a unique operation digest, the integrity and uniqueness of the aggregation operation information can be ensured. Once the aggregation operation information is tampered with, the corresponding operation digest will change, facilitating subsequent verification. The aggregation operation information includes the algorithm used for aggregation, the identifiers of each participant, the aggregation time, and relevant descriptive information about hidden intermediate results.

[0044] A Merkle tree is a binary hash tree where leaf nodes are the hash values ​​of data blocks (i.e., digest operations), and non-leaf nodes are the hash values ​​of their corresponding child nodes. By constructing a Merkle tree, multiple operation digests are integrated into a tree structure, improving data verification efficiency. When multiple aggregation operations exist, the operation digest of each aggregation operation can be used as a leaf node of the Merkle tree, and the hash values ​​of the parent nodes are calculated layer by layer upwards to ultimately generate the root hash of the Merkle tree.

[0045] By using Smart River Source, the root hash, timestamp, and operator identity information of the Merkle tree are written into a consortium blockchain ledger. A consortium blockchain is a blockchain jointly managed by multiple institutions, offering high security and controllability, making it suitable for cross-institutional data storage scenarios. Smart contracts are automated execution protocols based on blockchain, ensuring that operations written to the blockchain ledger follow preset rules, avoiding human intervention. By writing the Merkle tree root hash, timestamp (recording the time of the aggregation operation), and operator identity information (recording the entity performing the aggregation operation) into the consortium blockchain via smart contracts, this information is permanently stored in the blockchain ledger and cannot be tampered with.

[0046] For example, in a cross-agency environmental monitoring data collaborative computing project, after each time the federated computing coordinator completes the aggregation processing of data from various agencies, the blockchain module generates an operation summary and writes it to the blockchain ledger according to the above steps. When it is necessary to verify the authenticity and integrity of an aggregation operation, relevant personnel can obtain the root hash, timestamp, and operator identity information of the corresponding Merkle tree from the blockchain ledger, and simultaneously obtain the original operation information of that aggregation operation. They then recalculate the operation summary and construct the Merkle tree, comparing the recalculated root hash of the Merkle tree with the root hash stored in the blockchain ledger. If they match, it indicates that the information of the aggregation operation has not been tampered with, and the operation is authentic and valid. If they do not match, it indicates that the operation information may have been tampered with, requiring further verification. In this way, through the above functions of the blockchain module, immutable evidence of aggregation operations is achieved, making the data and model processing flow verifiable, and improving credibility and auditability.

[0047] Version control systems generate new version data based on the global computation results sent by the federated computing coordinator, and record the data version evolution path based on content-addressed storage structure and copy-on-write mechanism. They also support multi-branch coexistence and differential comparison mechanism to track changes between versions.

[0048] Version control systems employ a copy-on-write mechanism when generating new versions of data, saving only the difference blocks from the previous version and maintaining the sharing of unmodified data through pointer references. This mechanism significantly saves storage space and improves data storage efficiency. For example, in geospatial data processing, the initial version of a dataset contains a large amount of raster and vector data. When a partial modification is made to this version to generate a new version, the version control system does not copy the entire dataset, but only copies the modified parts, i.e., the difference blocks, and associates the new version data with the unmodified data blocks through pointers. In this way, the new version data only stores the difference blocks, while the unmodified data blocks reference the data from the initial version through pointers, avoiding duplicate data storage and significantly reducing storage space usage.

[0049] Content-addressed storage (CAD) refers to a version control system that identifies and locates data blocks using hash values ​​of the data content, rather than traditional file paths. Each data block has a unique hash value, allowing for quick location of the corresponding data block when data needs to be accessed. This storage structure makes data searching and access more efficient and also facilitates data integrity verification. If the content of a data block changes, its hash value will change accordingly. The version control system can detect data tampering promptly by monitoring changes in hash values, ensuring data integrity.

[0050] Version control systems support multiple branches, allowing users to experiment in parallel across different version chains, meeting the needs of different teams or research directions in multi-institutional collaborative development environments. For example, team A might focus on optimizing a crop growth monitoring model, while team B focuses on developing a pest and disease early warning model. The version control system can create independent branches for teams A and B, allowing each team to modify and update data on their respective branches, generating their own version data without affecting each other. When a team achieves a milestone, its branch can be merged with the main branch or other relevant branches to achieve the sharing and integration of results.

[0051] Differential comparison mechanisms are used to track changes between versions, including at least one of the following: data field differences, binary block differences, and geospatial data structured differences. Data field differences primarily refer to changes in metadata fields, such as changes to creation time, modification time, operator identifier, and data format. Binary block differences refer to differences at the binary level, applicable to version comparisons of unstructured data or binary files. Geospatial data structured differences are the types of differences held by geospatial data, including changes in geographic extent and statistical differences in pixel values. For example, when managing versions of remote sensing image data for a certain region, a differential comparison mechanism can clearly show changes in geographic extent between different versions, such as an expansion or contraction of the coverage area of ​​a certain version of remote sensing imagery compared to the previous version. It can also statistically analyze differences in pixel values, such as changes in pixel values ​​in vegetation cover areas due to seasonal variations. Version control systems can present these differences to users through a visual interface, such as using different colors to mark areas of geographic extent change and displaying statistical results of pixel value differences through charts, allowing users to intuitively understand the changes between versions and improving the transparency and traceability of data management. In this way, the aforementioned functions of the version control system enable effective recording and management of the data version evolution path, support multi-branch parallel development and version difference tracking, and improve data utilization efficiency and management convenience.

[0052] The secure access control subsystem performs identity verification and authorization control for federated computing tasks according to preset permission policies. During the authorization process, it uses an anonymous authentication protocol to hide the identity information of the participants. At the same time, it works in conjunction with the blockchain module to write access authorization records into the blockchain ledger.

[0053] The security access control subsystem supports role-based access control and attribute-based access control. The combination of these two access control methods enables fine-grained management of data access and sharing, meeting the flexible configuration requirements of permission policies across different organizations and user levels.

[0054] Role-based access control (RBAC) assigns access permissions based on a user's role within the organization. System administrators define different roles, such as data administrators, data analysts, and regular users, according to varying business needs and responsibilities, and assign corresponding access permissions to each role. For example, in a government department's spatiotemporal data management system, the data administrator role has full permissions, including data upload, modification, deletion, and permission assignment; the data analyst role has permissions for data query, analysis, and download; and the regular user role only has data query and browsing permissions. When a user logs into the system, the system automatically assigns appropriate permissions based on the user's role, and the user can only operate within the scope of their authorized permissions.

[0055] Attribute-based access control determines a user's access permissions based on multiple dimensions of attribute information, including user attributes, resource attributes, and environmental attributes. User attributes include the user's identity, department, and job level; resource attributes include data type, sensitivity level, and project; and environmental attributes include access time, location, and device. For example, sensitive geospatial data might only be accessible during working hours to senior analysts in a specific department using authorized internal devices. When a user requests access to this data, the security access control subsystem collects relevant user attribute information, data resource attribute information, and current environmental attribute information. It then makes a comprehensive judgment based on preset access control rules. If all rule conditions are met, access permission is granted; otherwise, the access request is denied.

[0056] In controlled sharing scenarios, the system can further incorporate anonymous authentication protocols to protect the identity of the requester, achieving the effect of hiding the user's true identity while proving access rights. Anonymous authentication protocols include at least one of ring signatures, group signatures, or zero-knowledge proofs.

[0057] Ring signatures are a digital signature technique where a signer selects multiple public keys from a group, mixes their private key with these public keys to form a ring, and then signs the message with their private key. Verifiers can only verify that the signature comes from a member of the ring, but cannot determine the specific signer's identity. For example, in a cross-institutional data sharing scenario, a user of institution A needs to access sensitive data of institution B. This user can use ring signature technology to form a ring with the public keys of several other users in institution A to sign the access request. After institution B's security access control subsystem verifies the signature's validity, it knows that the access request came from a user of institution A, but cannot determine which specific user, thus protecting the requester's identity privacy.

[0058] Similar to ring signatures, group signatures are also a type of anonymous signature technology. Signers belong to a specific group and use the group's public key to sign. Verifiers can only verify that the signature comes from the group, but cannot identify the specific signer. The difference is that group signatures typically have a group administrator responsible for managing member joining and leaving, and can open the signature panel to reveal the signer's identity when necessary, in response to potential security incidents.

[0059] Zero-knowledge proof is a cryptographic technique that allows a prover to convince a verifier that a statement is true without providing the verifier with any useful information. In secure access control, a requester can use zero-knowledge proofs to demonstrate to the system that they have permission to access specific data without providing their identity or other sensitive information. For example, if a requester possesses a credential that meets certain conditions, they can use a zero-knowledge proof to demonstrate to the system that they possess the credential and that it is valid. After verifying the validity of the proof, the system grants the requester access, but never knows the requester's specific identity or detailed information about the credential.

[0060] The secure access control subsystem works in conjunction with the blockchain module to write access authorization records into the blockchain ledger. These records include information such as authorization time, authorized recipient, authorized resources, and scope of authorized permissions. By writing this information into the blockchain ledger, the access authorization records are stored immutably, facilitating subsequent auditing and traceability. For example, when auditing access to specific data, auditors can query the blockchain ledger for all access authorization records for that data to understand which users or organizations gained access to the data, when, and the scope of those accesses. This ensures the compliance of data access and provides reliable evidence for potential security incident investigations. Through these functions of the secure access control subsystem, fine-grained control over data access and protection of the privacy of requesters are achieved, while ensuring the auditability of access authorization records and reducing the risk of unauthorized access and misuse of data.

[0061] The dataset management interface provides visualization and interactive functions for version data, allowing users to access version information and perform data operations through a unified interface. It also provides visualization rendering and interactive capabilities for version graphs, which can graphically display version chain relationships, branching situations, and differentiated content, improving the intuitiveness of data traceability and management.

[0062] The dataset management interface is developed based on a web platform, allowing users to easily access it through a browser without installing any additional client software, thus lowering the barrier to entry. The interface adopts an intuitive design style, presenting complex version data and management functions to users in a simple and clear manner.

[0063] In terms of visualization, the dataset management interface can display basic information about the dataset, such as dataset name, creation time, data type, data size, and project metadata. Simultaneously, the interface displays various versions of the dataset in a list or card format. Each version entry includes information such as version number, creation time, creator, version description, and a summary of the corresponding global computation results, allowing users to quickly understand the basic information of each version.

[0064] Users access version information and perform data operations through a unified interface, which includes a REST API and an SDK. The REST API uses a standard HTTP protocol and provides a series of API interfaces for users to call. The SDK provides developers with a more convenient programming interface, allowing them to directly call SDK methods in their local programs to access the functionality of the dataset management interface.

[0065] The visualization and interactive capabilities of the version graph are key features of the dataset management interface. The version graph graphically displays the version chain relationship of the dataset, with each version represented as a node. Nodes are connected by lines, clearly showing the evolutionary order of the versions. For multi-branch scenarios, different branches are distinguished by lines of different colors or styles, allowing users to intuitively see the development history and version changes of each branch.

[0066] When a user clicks on a version node, the interface displays detailed information about that version, including its metadata, the aggregation operation that generated it, and the differences from the previous version. The interface also supports comparative displays of differences between versions; users can select two or more versions for comparison, and the interface will visually demonstrate the differences in data fields, binary blocks, and geospatial data structure. Through these functions of the dataset management interface, users can intuitively and conveniently manage and trace data versions, understand the evolution of data and the differences between versions, and improve the efficiency and transparency of data management.

[0067] The Handle identification system generates globally unique persistent identifiers for operation summaries, datasets, and models, and associates them with resolution records. These resolution records include at least one of version location, historical version index, metadata, and content fingerprint, which are used to achieve unique data references, trusted access, integrity verification, and version consistency verification.

[0068] The unique and persistent identifiers generated by the Handle identification system are globally unique and persistent, unaffected by changes in data storage location, system upgrades, or organizational adjustments, and can stably identify data objects over a long period of time.

[0069] A resolution record is a set of information associated with a unique persistent identifier, containing key attributes and access information of the data object. Version location information indicates the storage address of the current version of the data. When a user queries a resolution record using the unique persistent identifier, they can obtain the data's storage location and thus access the data. The historical version index records the identifiers and related information of all historical versions of the data object. Users can use the historical version index to trace the version evolution of the data and access data from any historical version.

[0070] Metadata information includes the creation time of the data object, the creator, the data format, the data type, the data description, the project to which it belongs, and the associated standards and specifications. This information helps users understand the basic situation and background of the data object and determine whether the data meets their own usage needs.

[0071] A content fingerprint is a unique value obtained by hashing the content of a data object. It is used to verify data integrity and version consistency. After a user obtains the data, they can recalculate the content fingerprint and compare it with the content fingerprint stored in the parsing record. If they match, it means the data has not been tampered with during transmission or storage, and data integrity is guaranteed. If they do not match, it means the data may have been tampered with, requiring the data to be retrieved again or further security checks to be performed.

[0072] The Handle identification system also supports flexible configuration of the resolution service. Even when international cooperation is limited, it can retain its own resolution service to ensure identifier resolution capabilities. For example, in certain special circumstances where international network connections are unstable or restricted, making access to the globally unified Handle resolution service impossible, the system can activate its self-deployed resolution service. This ensures that users can correctly resolve data object information using a unique persistent identifier, guaranteeing data accessibility. Through these functions, the Handle identification system achieves unique identification, trusted access, and version traceability of data objects, ensuring data discoverability and accessibility across systems, organizations, and regions, thereby improving data sharing efficiency and utilization value.

[0073] This application uses a cross-regional air quality monitoring data collaborative analysis project as an example. The project involves environmental monitoring agencies in various regions, each possessing spatiotemporal air quality monitoring data within its respective jurisdiction. The project aims to integrate monitoring data from different regions, conduct cross-regional air quality collaborative analysis, predict air quality change trends, and provide a scientific basis for regional joint prevention and control of air quality issues.

[0074] The first step is system deployment and initialization. System deployment involves setting up local agent processes in environmental monitoring agencies across different regions, and establishing a federated computing coordinator, blockchain module, version control system, secure access control subsystem, dataset management interface, and handle identification system. All system modules are connected via a secure network to ensure secure communication.

[0075] System initialization includes user and role configuration, permission policy configuration, privacy computation protocol and aggregation algorithm configuration, and Handle identifier system initialization. User and role configuration is performed within the secure access control subsystem, assigning corresponding roles such as system administrator, data administrator, and data analyst. The system administrator is responsible for the overall system configuration and maintenance; the data administrator is responsible for uploading, version management, and permission allocation of their organization's data; and the data analyst executes subquery tasks and participates in collaborative analysis based on local data. Permission policy configuration configures permission policies based on role-based access control and attribute-based access control models. Privacy computation protocol and aggregation algorithm configuration pre-defines privacy computation protocols and federated aggregation algorithms based on project requirements and the data characteristics of each organization. In this project, due to the need for accurate calculation of the statistical results and trends of air quality indicators in various regions, and with clearly defined data dimensions, secure multi-party computation is prioritized as the primary privacy computation protocol; simultaneously, a federated averaging algorithm is configured as the federated aggregation algorithm to aggregate local intermediate results from various organizations. Handle identifier system initialization generates unique persistent identifiers for the basic datasets involved in the project and associates them with corresponding parsing records, including data storage location, metadata, historical version index, and content fingerprint.

[0076] Then, data uploading and version management are performed. For data uploading, data administrators from environmental monitoring agencies in each region upload their agency's spatiotemporal air quality monitoring data to the system via the web interface of the dataset management system. During the uploading process, the system verifies the data to ensure that the data format conforms to preset standards. After the data upload is complete, the version control system automatically generates an initial version of the dataset, assigns it a version number, and uses a copy-on-write mechanism to store the data in object storage and the spatiotemporal database. At the same time, the Handle identification system generates a new unique persistent identifier for this initial version of the data and updates the parsing record, including the location of the new version, the version index, and the content fingerprint.

[0077] Version recording and tracking are crucial because as projects progress, organizations may update or reprocess local data, generating new local data versions. Each time a new local data version is generated, the version control system records the differences between the new and previous versions, using pointers to reference unmodified data, thus saving storage space. Simultaneously, the new version information is updated to the dataset management interface, allowing users to view the dataset's version evolution path and trace the data's modification history.

[0078] Next, the joint computing task execution process is as follows. The first step is the submission of the joint computing task. That is, the data analyst of the project lead unit submits the joint computing task on the dataset management interface according to the project requirements. The task content is to calculate the average and maximum PM2.5 concentration of each region over the past month, as well as the spatial distribution trend of PM2.5 concentration across regions, and to specify the institutions participating in the joint computing.

[0079] The second step is task splitting and distribution. After receiving the joint computing task, the federated computing coordinator splits it into three sub-query tasks based on task requirements, the data distribution of participating institutions, and a pre-defined global query plan. These sub-query tasks correspond to the environmental monitoring centers in the three regions. The federated computing coordinator then distributes these three sub-query tasks to the local agent processes deployed by each participating institution using a high-performance remote procedure call framework combined with the TLS / SSL protocol.

[0080] The third step is local subquery execution and covert processing. Each participating institution's local agent process receives the subquery task and executes it locally. The local agent process coverstly processes the local intermediate results based on a pre-defined secure multi-party computation protocol. Under this protocol, the local agent process splits the local intermediate results into multiple secret shares. Each secret share itself does not contain any useful information; only by combining the secret shares of all participants according to a specific algorithm can the complete local intermediate result be recovered.

[0081] The fourth step is the aggregation of hidden intermediate results. The federated computing coordinator receives secret shares (hidden intermediate results) from the local agent processes of each participating agency and aggregates these secret shares using a pre-defined federated averaging algorithm. First, the coordinator performs a weighted average of the secret shares representing the average PM2.5 concentration in each region to obtain the average secret share of PM2.5 concentration across regions, thus reconstructing the average PM2.5 concentration across regions. Second, it aggregates the secret shares representing the maximum PM2.5 concentration in each region to determine the maximum value of PM2.5 concentration across regions. Finally, it aggregates the secret shares representing the spatiotemporal distribution characteristics of PM2.5 concentration in each region to generate spatial distribution trend data of PM2.5 concentration across regions. The result of these aggregation processes is the global calculation result.

[0082] The fifth step is aggregation operation notarization. The blockchain module obtains aggregation operation information from the federated computing coordinator, including the algorithm used for aggregation, the identifiers of each participating institution, the aggregation time, and descriptive information about hidden intermediate results. The blockchain module uses the SHA-256 cryptographic hash algorithm to generate an operation digest of this aggregation operation information, then constructs a Merkle tree containing the operation digest. Finally, a smart contract writes the root hash of the Merkle tree, the timestamp, and the operator's identity information into the consortium blockchain ledger based on Hyperledger Fabric, achieving immutable notarization of the aggregation operation.

[0083] The sixth step is the global computation result version management and display. The federated computation coordinator sends the global computation results to the version control system. The version control system generates new version data based on the global computation results, using a copy-on-write mechanism to save only the blocks that differ from the previous version—that is, the global computation result data—and references the unchanged original data blocks through pointers. Simultaneously, the version control system updates the data version evolution path, recording the new version information in the system.

[0084] The dataset management interface retrieves new version data from the version control system and presents it to users in a visual manner via a web interface. Users can view detailed data of the global calculation results through the interface, such as the average and maximum PM2.5 concentrations for each region, as well as spatial distribution trend maps of PM2.5 concentrations across regions. Simultaneously, users can obtain new version data through the REST API or SDK interfaces provided by the interface for further analysis and report generation.

[0085] Finally, data access and security control are implemented. The first step is data access request and authentication. Researchers need to access the global computational results data in the project dataset for related academic research. The researcher first submits a data access request through the dataset management interface, entering their username and password. Upon receiving the access request, the security access control subsystem verifies the user's identity, checking the correctness of the username and password. Simultaneously, based on the attribute-based access control policy, it checks whether the user's attributes, resource attributes, and environment attributes comply with the preset access permission rules.

[0086] The second step involves anonymous authentication and authorization. Since the research institution and the environmental monitoring agency involved in the project do not have a direct cooperative relationship, the secure access control subsystem employs a ring signature anonymous authentication protocol during the authorization process to protect the researchers' privacy. The researcher's local client generates a ring containing their own public key and the public keys of researchers from multiple other research institutions, ring-signs the access request, and sends the signed request to the secure access control subsystem. After verifying the validity of the ring signature, the system confirms that the access request originates from a group of researchers within the research institution, but cannot determine the specific requester's identity, thus protecting the researcher's privacy. After confirming that the user's identity verification is successful and meets the anonymous authentication requirements, the secure access control subsystem grants the researcher access to the new version of the project dataset according to a preset permission policy, allowing them to view and download global calculation results data, but prohibiting them from accessing the original monitoring data from various regions. Simultaneously, the secure access control subsystem sends this access authorization record to the blockchain module, writing it into the consortium blockchain ledger via a smart contract, achieving immutable evidence storage of the access authorization record, facilitating subsequent auditing and traceability.

[0087] The third step involves data integrity verification. After researchers download the global computation results data through the dataset management interface, data integrity verification is required to ensure that the data has not been tampered with during transmission. Researchers first use the Handle identification system to query the parsing record corresponding to the unique persistent identifier of the new version of the project dataset, obtaining the content fingerprint of that version of the data. Then, researchers calculate the content fingerprint of the downloaded data locally and compare it with the content fingerprint in the parsing record. If they match, the data integrity is guaranteed, and the data has not been tampered with; if they do not match, the data may have been tampered with during transmission, and researchers need to resubmit the data access request and download the data. This application, through the collaborative work of core modules such as the federated computing coordinator, blockchain module, version control system, secure access control subsystem, dataset management interface, and Handle identification system, effectively solves the problems of insufficient data privacy protection, limited version management capabilities, and poor auditability in existing technologies for distributed spatiotemporal data processing, achieving privacy-preserving collaborative computing and versioned management of distributed spatiotemporal data.

[0088] It should be understood that although the steps in the flowcharts in the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps, and they can be performed in other orders.

[0089] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An engine system based on integrated privacy federated computing capabilities, characterized in that, The system is used for privacy-preserving collaborative computing and version management of sensitive spatiotemporal data in geospatial data, remote sensing imagery, and meteorological data, including: The federated computing coordinator is used to receive joint computing tasks from various participants, break down the joint computing tasks into multiple sub-queries, and distribute them to the local agent processes deployed by each participant. Each local agent process executes the subquery locally to obtain local intermediate results, and performs covert processing on the local intermediate results based on a preset privacy computing protocol to obtain covert intermediate results, which are then sent to the federated computing coordinator. The federated computing coordinator also uses a federated aggregation algorithm to aggregate the hidden intermediate results to obtain global computing results, and sends them to the version control system. The blockchain module obtains the aggregation operation information corresponding to the aggregation process, generates an operation summary based on the aggregation operation information, and writes it into the blockchain ledger. The version control system generates new version data based on the global calculation results and records the data version evolution path based on the content-addressable storage structure and copy-on-write mechanism; When executing the sub-query, the local agent process uses a high-performance remote call framework for inter-node communication, combined with a transport layer security protocol. The content-addressable storage structure refers to the version control system identifying and locating data blocks by the hash value of the data content, rather than by the traditional file path. When generating new version data, the version control system uses a copy-on-write mechanism to save only the blocks that differ from the previous version and maintains the sharing of unmodified data through pointer references. The security access control subsystem performs identity verification and authorization control on the federated computing task according to the preset permission policy, and uses an anonymous authentication protocol to hide the identity information of the participants during the authorization process; The dataset management interface provides a visual display and interactive function for version data, and supports users to access version information and perform data operations through a unified interface. The Handle identification system generates a unique persistent identifier for the operation digest and associates it with a resolution record, wherein the resolution record includes at least one of version location, historical version index, and metadata; The unique persistent identifier generated by the Handle identification system is associated with a parsing record that also includes a content fingerprint, used to verify data integrity and version consistency. The secure access control subsystem provides anonymity authentication protocols including at least one of ring signatures, group signatures, or zero-knowledge proofs. The version control system also supports multi-branch coexistence and a differential comparison mechanism to track changes between versions. These changes include differences in data fields, binary blocks, and geospatial data structure. The geospatial data structure differences include changes in geographical range and statistical differences in pixel values. The version control system presents these differences to the user through a visual interface, including marking areas of geographical range change with different colors and displaying statistical results of pixel value differences through charts.

2. The system of claim 1, wherein, The privacy computation protocol includes at least one of secure multi-party computation, homomorphic encryption, and differential privacy; The local agent process is configured to dynamically select and configure the privacy computing protocol based on the task type and constraints of the participants in the federated computing task. The federated aggregation algorithm is either a federated averaging algorithm or a secure aggregation protocol.

3. The system of claim 1, wherein, The step of generating an operation summary based on the aggregated operation information and writing it into the blockchain ledger includes: A cryptographic hash algorithm is invoked to generate an operation digest of the aggregated operation information; Construct a Merkle tree containing a summary of the operations; The root hash, timestamp, and operator identity information of the Merkle tree are written into a blockchain ledger based on a consortium blockchain via a smart contract.

4. The system according to claim 1, characterized in that, The security access control subsystem supports role-based access control and attribute-based access control, and works in conjunction with the blockchain module to write access authorization records into the blockchain ledger.

Images