Decentralized multi-party collaborative data processing method, system, device, medium and product

Abstract

The application discloses a kind of decentralized multi-party collaborative data processing method, system, equipment, medium and product, involve multi-party security calculation field, this method includes: in symbol plane executes target host program to build position-aware computing graph row by row;During construction, each participant node uses hybrid evaluation strategy in physical plane to execute position-aware computing graph;Wherein, if all input data of the current graph node of position-aware computing graph is ready locally, trigger early evaluation to carry out asynchronous calculation;If the value of current graph node is explicitly dependent on control flow or output statement, trigger delayed evaluation to carry out on-demand calculation;In the case where target host program is executed, all participant nodes obtain the same position-aware computing graph and local cache value of itself completely.This application can realize end-to-end delay optimization and fine-grained control of data access permission in a centerless scheduling collaborative computing architecture.

Description

Technical Field

[0001] This application relates to the field of secure multi-party computation, and in particular to a decentralized multi-party collaborative data processing method, system, device, medium, and product. Background Technology

[0002] With the development of big data, cloud computing, and distributed computing technologies, the demand for multi-party collaborative computing across organizations and autonomous systems is growing, and it is widely used in scenarios such as federated learning, privacy computing, and industrial internet data fusion. Existing distributed computing frameworks can be mainly divided into three categories: First, general-purpose distributed frameworks centered on a central scheduler, relying on a central node to manage global tasks; however, control plane communication is complex, violating the principles of data sovereignty and decentralization. Second, dedicated protocol frameworks for privacy computing, employing synchronous RPC models primarily for cryptographic protocol implementation; however, high latency on wide area networks is linearly accumulated, resulting in low execution efficiency. Third, compiler / tracing frameworks for tensor computing, which split single programs into multiple devices for parallel execution through static or tracing compilation; these are well-suited for data center intranet environments, but lack optimization for cross-domain data sovereignty and wide area network latency. Therefore, existing distributed computing frameworks mainly suffer from defects such as central dependency, latency sensitivity, and coarse data access control, making it difficult to meet the practical application requirements of multi-party data sharing and peer-to-peer collaborative computing on wide area networks. Summary of the Invention

[0003] In view of the above-mentioned defects or deficiencies in the related technologies, the purpose of this application is to provide a decentralized multi-party collaborative data processing method, system, device, medium and product, which can realize end-to-end latency optimization and fine-grained control of data access permissions in a collaborative computing architecture without central scheduling.

[0004] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a decentralized multi-party collaborative data processing method, the method comprising: All participating nodes in the collaboration process parse the computation request sent by the external client to obtain the target host program identifier; All participating nodes load the target host program based on the target host program identifier; the target host program contains embedded site identifiers of the participating nodes that indicate the executing program nodes; During the execution of the target host program, all participating nodes will decouple the execution flow into parallel symbolic plane and physical plane; The target host program is executed line by line in the symbolic plane to construct a location-aware computation graph; the location-aware computation graph includes a graph node set, a directed edge set, and a variable location mapping table; the information of each graph node in the graph node set includes a globally unique identifier, a function reference, the site identifier of the participating nodes of the executor node, and the graph node status; the information of each edge in the directed edge set includes an upstream graph node, a downstream graph node, the data variable identifier flowing through the corresponding edge, a data source node, a destination node, and an access control list; During the construction process, each participating node executes the location-aware computation graph using a hybrid evaluation strategy on the physical plane; wherein, if all input data of the current graph node of the location-aware computation graph is ready locally, early evaluation is triggered for asynchronous computation; if the value of the current graph node is explicitly dependent on the control flow or output statement, delayed evaluation is triggered for on-demand computation. During the early or delayed evaluation process, if the owner of the input data required by the current graph node is a remote participant node, the cross-node data transmission protocol is triggered. In response to the cross-node data transmission protocol, the participant node that needs the data actively initiates a data retrieval command, and at the same time, the participant node that provides the data pre-materializes the target data to the temporary storage area based on its local computing state to complete the cross-node data delivery. After the target host program is executed, all participating nodes obtain the exact same location-aware computation graph and their respective local cache values.

[0005] Optionally, the step of executing the target host program line by line in the symbol plane to construct a location-aware computation graph includes: The target host program is parsed line by line in the symbol plane to generate an operator stream; When generating each operator in the operator flow, the operator type is determined. If the operator type is a computation operator, the input data of the computation operator is subjected to a common invariant check. If the check passes, a graph node and data dependency edge are added at the corresponding position of the initial position-aware computation graph to generate the current position-aware computation graph. If the operator type is a transport operator, update the variable position mapping table, update the owner node of the source variable corresponding to the transport operator to the destination node; add the destination node to the outgoing edge access control list of the upstream graph node corresponding to the source variable; If the operator type is a synchronization operator, the holder of the private data corresponding to the synchronization operator generates a criterion result and broadcasts the criterion result to other participating nodes; after receiving the criterion result, all participating nodes align their instruction pointers (IP) and enter the same branch to continue executing the target host program. After traversing the operator stream, the position-aware computation graph is generated.

[0006] Optionally, performing a common-invariant check on the input data of the computation operator includes: Verify whether the variable position mapping table Loc[x] of each input variable of the computation operator is equal to the site identifier of the target participant node of the computation operator; if it is not equal, it indicates that the operands corresponding to the input variables are distributed in the memory space of different participant nodes, throw a Fusion Gap exception, and prompt the developer to move the data to the same participant node through the δ_move operator.

[0007] Optionally, if the verification passes, adding graph nodes and data dependency edges at the corresponding positions in the initial location-aware computation graph to generate the current location-aware computation graph includes: After the co-occurrence check passes, the value of the current number of graph nodes plus 1 will be used as the globally unique identifier of the newly added graph node; Create a new graph node in the graph node set, and add a directed edge from the upstream graph node corresponding to the input variable to the new graph node in the directed edge set for each input variable; If the site identifier of the participant of the expected execution program node is equal to the site identifier of the newly added graph node, then the node state of the new graph node is set to pending state, and early evaluation is triggered in the physical plane; if the site identifier of the participant of the expected execution program node is not equal to the site identifier of the newly added graph node, then the node state of the new graph node is marked as symbolic state, retaining only the graph structure information without undertaking physical execution.

[0008] Optionally, the step of triggering early evaluation for asynchronous computation if all input data for the current graph node of the location-aware computation graph is ready locally includes: If the upstream graph node of the current graph node has completed early evaluation, the local cached value of the upstream graph node is read directly; if the upstream graph node is still in the process of early evaluation, the early evaluation completion event of the upstream graph node is waited for; if the execution participant node of the upstream graph node is not the current participant node, the cross-node data transmission protocol is triggered to pull input data from the participant node that provides the data; after all the input data of the upstream graph node is in place, the current participant node performs local computation on the physical plane and writes the computation result into the local state of the current graph node.

[0009] Optionally, the step of triggering delayed evaluation for on-demand computation if the value of the current graph node is explicitly dependent on the control flow or output statement includes: Iterate through each input variable of the current graph node and recursively perform asynchronous evaluation operations on the input variables to make the input variables ready. The asynchronous evaluation operations include: if the input variable has been evaluated early, directly obtain and return the cached value of the input variable; if the input variable is in the early evaluation execution state, suspend the current thread and wait for the evaluation completion event of the input variable; if the input variable is in the suspended state, recursively trigger delayed evaluation of the input variable to backtrack up the data dependency chain to activate the ancestor node; if the target execution participant node of the input variable is not the current participant node, trigger the cross-node data transmission protocol to actively pull the value of the input variable from the participant node that owns the input variable. After detecting that all input variables of the current graph node are in place, the local operation method corresponding to the current graph node is executed on the physical plane, and the calculation result is written to the local state as input for the downstream synchronization contact.

[0010] Secondly, this application provides a decentralized multi-party collaborative data processing system, which includes multiple participating node nodes with unique site identifiers; all participating node nodes execute the steps of the decentralized multi-party collaborative data processing method described in the first aspect above.

[0011] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the decentralized multi-party collaborative data processing method described in the first aspect above.

[0012] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the decentralized multi-party collaborative data processing method described in the first aspect.

[0013] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the decentralized multi-party collaborative data processing method described in the first aspect.

[0014] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a decentralized multi-party collaborative data processing method, system, device, medium, and product. By introducing a decoupling mechanism between the symbolic plane and the physical plane execution flow, it achieves parallel processing of computational logic construction and underlying resource execution. Asynchronous computation can be performed simultaneously with the construction of a location-aware computation graph using a hybrid evaluation strategy, effectively hiding the system overhead of the graph construction phase and significantly improving the execution efficiency and resource utilization of distributed computing. Through early and delayed evaluation, asynchronous computation can be triggered when local data is ready to fully utilize idle computing power, and computation can be performed on demand when control flow depends to avoid unnecessary resource consumption, thus realizing the computational efficiency and resource utilization of distributed computing. The system optimizes resource scheduling; through the design of a location-aware computation graph and variable location mapping table, it can accurately perceive the data distribution status. Combined with the collaborative mechanism of data receiver actively pulling and data sender pre-materializing in the cross-node data transmission protocol, it effectively reduces the handshake latency and bandwidth consumption in cross-node communication, and solves the bottleneck of frequent data movement in the distributed environment. In addition, by embedding site identifiers and access control lists, it ensures that each participating node ultimately obtains a completely consistent computation graph and cached values ​​under independent execution. While ensuring strong consistency of multi-party collaborative computation results, it realizes secure and controllable data interaction and efficient parallelism of the computation process. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram of the structure of a decentralized multi-party collaborative data processing system provided in an embodiment of this application; Figure 2 A flowchart illustrating a decentralized multi-party collaborative data processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a symbol-physical dual-plane decoupling architecture provided in an embodiment of this application; Figure 4 This is a partial structural schematic diagram of a location-aware computation graph provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the construction process of a location-aware computational graph provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the relationship between multi-participant nodes and the Fusion Gap provided in an embodiment of this application; Figure 7 This is a schematic diagram of cross-node transmission of a location-aware computation graph provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] In one exemplary embodiment, such as Figure 1 As shown, a decentralized multi-party collaborative data processing system is provided. This decentralized multi-party collaborative data processing system includes multiple participating node nodes with unique site identifiers. In this collaborative session, all participating node nodes obtain the same host program and execute the decentralized multi-party collaborative data processing method based on the host program.

[0020] In one exemplary embodiment, such as Figure 2 As shown, a decentralized multi-party collaborative data processing method is provided. In this embodiment, the method includes the following steps S101 to S107. Wherein: In step S101, all participating nodes in the collaboration parse the computation request sent by the external client to obtain the target host program identifier.

[0021] It should be noted that the external client sends computation requests to each participating node via gRPC or HTTP interface; after receiving the computation request, each participating node parses the host program identifier and the list of participants carried in the computation request, assigns a unique process sequence number to this collaborative session, and starts an independent host program tracking container on each participating node accordingly.

[0022] After the host program traces the container and starts, it loads the private data of its participating node into its local memory according to the declarations in the host program. The private data comes from three sources: (a) generated out of thin air by executing a specific function locally on the participating node (such as random number generation); (b) read from the local file system; and (c) obtained from a registered local HTTP private interface. Regardless of the method used, the private data exists only in the local memory of its own participating node and is not transmitted to other participating nodes. Each participating node holds the same copy of the host program and its own private data, laying the foundation for the decentralized execution of subsequent steps.

[0023] In step S102, all participating nodes load the target host program based on the target host program identifier; the target host program contains embedded site identifiers of participating nodes that indicate the executing program nodes.

[0024] In step S103, during the execution of the target host program, all participating nodes decouple the execution flow into parallel symbol plane and physical plane.

[0025] Step S104: Execute the target host program line by line in the symbol plane to construct a location-aware computation graph; the location-aware computation graph includes a graph node set, a directed edge set, and a variable location mapping table; the information of each graph node in the graph node set includes a globally unique identifier, function reference, site identifier of the participating nodes of the executing program node, and graph node status; the information of each edge in the directed edge set includes upstream graph node, downstream graph node, data variable identifier flowing through the corresponding edge, data source node, destination node, and access control list.

[0026] In an example embodiment, constructing a location-aware computation graph may include: parsing the target host program line by line in the symbol plane to generate an operator stream; determining the operator type for each operator in the operator stream; if the operator type is a computation operator, performing a common invariant check on the input data of the computation operator; if the check passes, adding graph nodes and data dependency edges at the corresponding positions in the initial location-aware computation graph to generate the current location-aware computation graph; if the operator type is a transmission operator, updating the variable position mapping table and updating the owner node of the source variable corresponding to the transmission operator to the destination node; adding the destination node to the outgoing edge access control list of the upstream graph node corresponding to the source variable; if the operator type is a synchronization operator, the holder of the private data corresponding to the synchronization operator generates a criterion result and broadcasts the criterion result to other participating nodes; after receiving the criterion result, all participating nodes align their instruction pointers (IP) and enter the same branch to continue executing the target host program; and generating a location-aware computation graph after traversing the operator stream.

[0027] It should be noted that the above-mentioned co-location invariant verification of the input data of the computation operator includes: verifying whether the variable position mapping table Loc[x] of each input variable of the computation operator is equal to the site identifier of the target participant node of the computation operator; if it is not equal, it indicates that the operands corresponding to the input variables are distributed in the memory space of different participant nodes, throws a Fusion Gap exception, and prompts the developer to move the data to the same participant node through the δ_move operator.

[0028] If the verification passes, a new graph node and data dependency edge are added to the corresponding position of the initial position-aware computation graph to generate the current position-aware computation graph. This includes: after the co-invariant verification passes, the value of the current graph node count plus 1 is used as the globally unique identifier of the new graph node; a new graph node is created in the graph node set, and a directed edge from the upstream graph node corresponding to the input variable to the new graph node is added to the directed edge set for each input variable; if the site identifier of the expected execution program node is equal to the site identifier of the new graph node, the node state of the new graph node is set to the pending state, and early evaluation is triggered in the physical plane; if the site identifier of the expected execution program node is not equal to the site identifier of the new graph node, the node state of the new graph node is marked as the symbolic state, retaining only the graph structure information without undertaking physical execution.

[0029] In step S105, during the construction process, each participating node executes the location-aware computation graph in the physical plane using a hybrid evaluation strategy; wherein, if all input data of the current graph node of the location-aware computation graph is ready locally, early evaluation is triggered to perform asynchronous computation; if the value of the current graph node is explicitly dependent on the control flow or output statement, delayed evaluation is triggered to perform on-demand computation.

[0030] It should be noted that during asynchronous computation, if the upstream graph node of the current graph node has completed early evaluation, the local cached value of the upstream graph node is read directly; if the upstream graph node is still in the process of early evaluation, the process waits for the early evaluation completion event of the upstream graph node; if the execution participant node of the upstream graph node is not the current participant node, the cross-node data transmission protocol is triggered to pull input data from the participant node that provides the data; after all the input data of the upstream graph node is in place, the current participant node performs local computation on the physical plane and writes the computation result into the local state of the current graph node.

[0031] When performing on-demand computation, each input variable of the current graph node is traversed, and asynchronous evaluation operations are recursively performed on the input variables to ensure that the input variables are in place. The asynchronous evaluation operations include: if the input variable has been evaluated early, the cached value of the input variable is directly obtained and returned; if the input variable is in the early evaluation execution state, the current thread is suspended and the evaluation completion event of the input variable is waited for; if the input variable is in the suspended state, delayed evaluation is recursively triggered on the input variable to backtrack up the data dependency chain to activate the ancestor node; if the target execution participant node of the input variable is not the current participant node, the cross-node data transmission protocol is triggered to actively pull the value of the input variable from the participant node that owns the input variable; after detecting that all input variables of the current graph node are in place, the local operation method corresponding to the current graph node is executed on the physical plane, and the calculation result is written to the local state as the input of the downstream synchronization contact.

[0032] In step S106, during the early evaluation or delayed evaluation process, if the owner of the input data required by the current graph node is a remote participant node, the cross-node data transmission protocol is triggered. In response to the cross-node data transmission protocol, the participant node that needs the data actively initiates a data retrieval command, and at the same time, the participant node that provides the data pre-materializes the target data to the temporary storage area based on its local computing state to complete the cross-node data delivery.

[0033] It should be noted that, in the process of completing cross-node data delivery, the method also includes: the data-providing participating node verifies the data-requiring participating node's pull permission based on the access control list of the corresponding edge in the location-aware computing graph; after the permission verification is passed, the data-requiring participating node retrieves the target data from the temporary storage area to complete the cross-node data delivery.

[0034] In step S107, after the target host program is executed, all participating nodes obtain the exact same location-aware computation graph and their respective local cache values.

[0035] By introducing a decoupling mechanism between the symbolic plane and the physical plane execution flow, parallel processing of computational logic construction and underlying resource execution is achieved. Asynchronous computation can be performed using a hybrid evaluation strategy while constructing the location-aware computation graph, effectively hiding the system overhead of the graph construction phase and significantly improving the execution efficiency and resource utilization of distributed computing. Through early evaluation and delayed evaluation, asynchronous computation can be triggered when local data is ready to make full use of idle computing power, and computation can be performed on demand when control flow depends to avoid invalid resource consumption, achieving optimal scheduling of computing resources. Through the design of the location-aware computation graph and variable location mapping table, the data distribution status can be accurately perceived. Combined with the collaborative mechanism of data receiver actively pulling and data sender pre-materialization in the cross-node data transmission protocol, the handshake latency and bandwidth occupation of cross-node communication are effectively reduced, solving the bottleneck of frequent data movement in the distributed environment.

[0036] To understand steps S101 to S107 above, specifically: like Figure 1 As shown, the system in this application adopts a single-program decentralized execution model (SPDE model), that is, all participating nodes run the exact same host program, and each participating node dynamically determines its role in this collaborative session based on its own site ID. In other words, each participating node only performs local computation on the program node marked with its own site ID.

[0037] like Figure 3 As shown, each participating node decouples its execution flow into two parallel planes during this collaborative session: the symbolic plane and the physical plane. The symbolic plane runs on the main thread, responsible for maintaining the GSC (Global Sequence Counter), intercepting operators, and incrementally building the Position-Aware Computation Graph (PACG), without participating in physical computation. The physical plane asynchronously executes local computations of graph nodes on a separate background worker thread (event loop), while also accepting cross-node data write-backs. The symbolic plane and physical plane are decoupled through the event loop, ensuring that the position-aware computation graph construction in the symbolic plane is not blocked by I / O or computation in the physical plane.

[0038] That is, each participating node maintains a local Position-Aware Computation Graph (PACG), in the following form: Where V is the set of graph nodes, E is the set of directed edges, and Loc is the variable position mapping table; and a global sequence counter (denoted as κ) with an initial value of 0 is also maintained. Initially, the set of graph nodes V, the set of directed edges E, and the variable position mapping table Loc are all empty sets.

[0039] like Figure 4 As shown, each node in the location-aware computation graph corresponds to a computation operator, described by a quadruple: UID (globally unique identifier), func (function reference), target (Site ID of the participating site of the node to be executed), and status (node ​​status, with values ​​of pending, scheduled, executing, completed, or symbolic). The target attribute is a core extension that distinguishes location-aware computation graphs from ordinary computation graphs, used to support data sovereignty constraints and role projection.

[0040] In the location-aware computation graph, each directed edge e corresponds to one cross-operator data dependency, i.e.: ; The meanings of each field are as follows: v_src and v_dst are the upstream and downstream graph nodes, respectively; var is the identifier of the data variable flowing through the edge; src_site and dst_site are the data source node and destination node, respectively; acl is the access control list, which only lists the set of destination nodes that are allowed to consume the data; the auxiliary mapping Loc:VarID→Site ID records the current owner node of each variable.

[0041] During the execution of the host program line by line, such as Figure 5As shown, S1 performs initialization, making the graph node set V, directed edge set E, and variable position mapping table Loc empty sets; S2 creates a Python interpreter to intercept operations, method calls, attribute accesses, and index retrievals involving remote variables RV and virtual variables DV in real time; S3, during interception, each intercepted operation is encapsulated as an operator, forming an operator stream in the order of interception. Each operator in the operator stream carries the following information: operator type (including δ_comp operator, δ_move operator, or δ_sync operator), target execution node identifier, input variable list, and function references. Because the SPDE model guarantees that all participating nodes execute the same host program, the operator streams generated independently by each participating node are strictly identical in content and order, and the resulting position-aware computation graphs are also identical; S4, for each received operator, determine the operator type and classify the current operator into one of the three primitive categories: δ_comp operator, δ_move operator, or δ_sync operator; where, if the current operator type is δ_comp operator, a new graph node is added to the graph node set V, and its incoming edge is added to the directed edge set E; if the current operator type is... The δ_move operator updates the variable position mapping table without modifying the animated graph node set V and the directed edge set E. If the previous operator type is δ_sync, it triggers control plane broadcasting, also without modifying the animated graph node set V and the directed edge set E. The global sequence counter is atomically incremented only during the δ_comp operator, and the incremented value is used as the globally unique identifier of the new graph node. In S5, after processing each operation operator, it is determined whether there are other operators. If so, S2 is executed; otherwise, S6 is executed, and the final position-aware computation graph and the value of the global sequence counter are output.

[0042] It should be noted that when the operator type is δ_comp, firstly, a co-occurrence invariant check is performed, that is, for each input variable x of the current operator, it is checked whether the current variable position mapping table Loc[x] is equal to the target execution node target of the current operator. For example... Figure 6 As shown, if any input variable does not meet this condition, it means that the data participating in the local computation is distributed in the memory space of different participating nodes, and a Fusion Gap exception is thrown, prompting the developer to first move the data to the same participating node using the δ_move operator.

[0043] After passing the co-occurrence invariant check, the global sequence counter performs an atomic increment on κ: The incremented value serves as the globally unique identifier for the newly added graph node.

[0044] Subsequently, a new graph node v_new=(UID,func,target,status) is created in the graph node set V, and a directed edge is added to the directed edge set E for each input variable x, pointing from the upstream graph node corresponding to the input variable x to the new graph node v_new. This directed edge carries the data variable identifier var, the data source node src_site, the destination node dst_site, and the access control list acl field, where acl is derived from the downstream consumer set topology.

[0045] Finally, role projection is performed: if target equals the Site ID of the participating node, then the node status of the new graph node v_new is set to pending and enters the physical plane scheduling process; otherwise, the status is marked as symbolic, retaining only graph structure information without undertaking physical execution; its formal semantics are: Each participating node independently performs atomic increment on the same δ_comp operator, resulting in a globally unique identifier that is inherently identical, requiring no cross-node RPC coordination—this is implicit sequence consensus.

[0046] When the operator type is δ_move, the global sequence counter κ is not incremented, nor are new graph nodes added to the graph node set V; instead, the following two updates are made directly to the existing position-aware computation graph (i.e., the current position-aware computation graph constructed by the previous δ_comp): First, the variable position mapping table is updated, changing the owner node of the source variable to the destination node, i.e., variable position mapping table Loc[x]←dst_site; Second, the destination node dst_site is added to the outgoing edge access control list acl of the upstream graph node corresponding to the input variable x, for use in deriving the access control list acl field when the downstream δ_comp operator constructs incoming edges. The above two updates complete the symbolic state transition of variable ownership and the registration of access control information. The actual physical data transmission is delayed until it is pulled by the receiver on demand; its formal semantics are: It should be noted that, for the case of broadcasting public data using .at_public(), this application equivalently expands it to registering access control lists (ACLs) with multiple destination nodes, making the public data equivalent in the graph structure to a set of N access control lists that allow destination nodes.

[0047] When a branch of the host program depends on the value of a private remote variable RV (e.g., in "if private_x>0", private_x is private data of a participating node), a δ_sync operator is automatically inserted at that branch instruction. Upon intercepting the δ_sync operator, neither the global sequence counter κ is incremented, nor are the set of nodes V and directed edges E of the animated graph modified. Instead, the following synchronization process is triggered: the owner node holding the remote variable RV evaluates the branch criterion and broadcasts the result to other participating nodes; all other participating nodes block and wait at the branch instruction; after receiving the criterion result, a participating node aligns its instruction pointer IP and enters the same branch to continue executing the host program; its formal semantics are: When the branch criterion is of the host's native type (e.g., common inputs such as integers and booleans), since the SPDE model guarantees that each participating node holds the same common parameters, each participating node can independently obtain the same branch result without communication or the need to insert the δ_sync operator.

[0048] The evaluation of graph nodes in the physical plane employs a hybrid triggering mechanism of early evaluation and lazy evaluation. Early evaluation is tentatively committed to the physical plane as soon as a graph node is created, while lazy evaluation is activated on demand in a chained manner only when downstream nodes require the value of the current node. Cross-node data transmission is triggered on demand when a remote variable RV is encountered during the evaluation process. Its decision-making strategy is described by the piecewise priority function α(v): Where IsLocal(v) means that v.target equals the Site ID of this participant node; AllParentsEager(v) means that all upstream graph nodes corresponding to the remote variable RV input of graph node v have been determined to be early evaluated (i.e., their input chains are all located locally on this participant node); Demanded(v) means that the value of graph node v is required by a synchronization trigger, including early evaluation trigger and delayed evaluation trigger; where: Early evaluation triggering occurs as follows: When a new graph node v_new is created in the graph node set V, if_early_evaluate is immediately invoked. Specifically, if v_new.target equals the current participant node and all upstream graph nodes corresponding to its remote variable RV inputs are early evaluation nodes, then at the same time the graph node is created, the do_evaluate task of that node is added as a coroutine to the physical plane's event loop. The physical plane's worker threads then asynchronously execute local computations. The main thread does not wait for the coroutine to complete and continues to return to the symbol plane to intercept subsequent operators. This mechanism allows the physical computation of the δ_comp operator to proceed in parallel with the construction of the tracking / position-aware computation graph for subsequent operators, making full use of the physical plane's idle time.

[0049] On the physical plane, the early evaluation coroutine concurrently initiates evaluation proxies for all remote variable RV inputs of the new graph node v_new. Specifically: if an upstream graph node has also completed early evaluation, its local cached value is read directly; if the upstream graph node is still undergoing early evaluation, its early evaluation completion event is awaited; if the upstream graph node is not currently in the participating node's node range (i.e., ...), ... Figure 7 As shown, if the target of v_new is the site identifier of the participating node (but an input variable belongs to another participating node), then cross-node fetching is triggered. After all the inputs of the upstream graph nodes are in place, the physical plane calls the corresponding operation method to perform the calculation and writes the result to the local state of the participating node; all subsequent operations accessing the same node value directly reuse this cached result.

[0050] LazyEvaluate and chained activation work as follows: Graph nodes that do not meet the early evaluation condition (i.e., all their input chains are located locally on the participating node) are suspended by default upon creation. This means they are neither committed to the physical plane nor trigger any cross-node transfers. They only enter the lazy evaluation channel when the value of the graph node is explicitly needed.

[0051] This application's deferred evaluation has two types of synchronization contacts: one is a branch criterion synchronization contact: when a conditional branch of the form "if private_rv>0:" appears in the host program, or when the interpreter initiates a Boolean evaluation of the remote variable RV, the actual value of the graph node must be obtained immediately to complete the branch decision; the other is a result output synchronization contact: when the host program executes an output statement or explicitly calls .evaluate() on a remote variable RV to export its value to the local machine (e.g., printing, file writing, HTTP return), the actual value of the graph node must be obtained immediately. Each of the above synchronization contacts corresponds to the host thread's call to RV.evaluate(), thus entering the deferred evaluation channel.

[0052] Delayed evaluation is performed in a chained activation process as follows: First, acquire the asynchronous lock on the current graph node. If the current graph node has already completed evaluation, return its cached result directly. Otherwise, for each remote variable RV input p of the current graph node, recursively call the asynchronous evaluation method of p. If p has already been evaluated early, return the cached value of p directly. If p is being evaluated early, wait for its completion event. If p is in a suspended state (neither evaluated early nor delayed), continue to recursively trigger delayed evaluation on p, activating it back to the ancestor node level by level. If p is not in the current graph node (p.target ≠ this node), the data receiver (the participating node that needs the data) actively triggers a fetch, retrieving its value from the owner node of p. After all inputs are in place, the current graph node executes the local operation method in the physical plane and writes the result to the local state as the input for subsequent downstream synchronization contacts.

[0053] The delayed evaluation mechanism causes graph nodes that actually affect the visible results of this node to incur physical computation costs; for symbolic graph nodes (which account for the vast majority) whose target is not the site identifier of this participant node and are not required by any synchronization touchpoint of this participant node, their local state remains suspended until the end of the collaboration session, thereby significantly reducing the local overhead of non-target graph nodes.

[0054] It should be noted that the triggering of the aforementioned cross-node data transmission (the issuance of Pull and Prepare commands) can include: when a graph node of this participating node enters the evaluation process (either early or delayed evaluation is acceptable), and the upstream graph node proxies by one of its remote variables RV input p is not in this participating node (p.node.site ≠ this Site), a cross-node data transmission is triggered. Since each participating node independently constructs a location-aware computation graph, an implicit consensus has been reached structurally regarding who the data owner and data consumer are. Therefore, the triggering of cross-node data transmission symmetrically derives the following two types of commands at both ends: First, the data receiver (the participating node that needs the data) issues a Pull instruction. That is, when the participating node initiates internal_evaluate on the remote variable RV input p and determines that p.node.site ≠ this Site ID, it generates a Pull instruction, instructing it to retrieve the data from the owner node of p.

[0055] Secondly, the data sender (the participating node providing the data) issues a Prepare instruction. That is, in the dual case, when the participating node performs evaluation proxy for a downstream graph node v' located at a remote target (the downstream graph node v' is projected as a symbolic node in the participating node's role, and still follows the evaluation process to drive the evaluation and transmission of its remote variable RV input, but does not execute the operation of the downstream graph node v' itself), the local remote variable RV input p' (p'.node.site=this Site ID, p'.site=remote) will automatically trigger a Prepare instruction in p''s internal_evaluate, materializing the local evaluation result of p' to the temporary storage area of ​​the participating node for later retrieval.

[0056] The Pull command of the data receiver and the Prepare command of the data sender constitute a symmetrical triggering closed loop. That is, both parties are driven by the topologically isomorphic PACG obtained by their respective local tracking. The transmission semantics can be aligned without relying on any central scheduler, and cross-node communication occurs strictly on demand. It is only triggered when a graph node enters the evaluation process of the participating node and its input is actually located at a remote end, without generating irrelevant forward push.

[0057] The triggering of cross-node data transmission involves two types of commands: the Prepare command corresponding to the data sender and the Pull command corresponding to the data receiver. Both commands employ a protocol where the data receiver actively pulls the data, while the data sender pre-materializes it to complete the cross-node data transmission. The specific process is as follows: (1) Materialized underlying delivery: The temporary storage form in the temporary storage area is as follows: It should be noted that the value of the same globally unique identifier (UID) is based on a single materialization; the corresponding allow_list can be incrementally merged with the subsequent accumulation of .at (other participating nodes) to support situations where .at_public() broadcasts publicly or the same data node is authorized multiple times by different participating nodes. If p' itself has not yet completed its local evaluation when materialization occurs, the prepare instruction will first chain the evaluation of p' before writing it to the temporary storage area.

[0058] (2) The underlying delivery of the pull: send the Pull command to the owner node of p in gRPC; the access token carried in the Pull command is derived from (UID, this Site ID) and is used for the owner's access control list ACL hit determination.

[0059] (3) Verification and return: The Pull instruction blocks asynchronously and waits until the entry is ready; after it is ready, it checks whether the access token matches the access control list acl. If it matches, the corresponding value Val is read and returned to the data receiver in gRPC response mode; if it does not match (i.e. the participating node is not in allow_list), it directly returns rejection, and the data receiver throws a cross-node authorization exception accordingly.

[0060] After the data receiver receives Val(), it uses it as the return value when initiating the Pull instruction. The evaluation proxy on the data receiver then unblocks, and its downstream graph nodes call operation to perform local computation once all remote variable inputs from the upstream graph nodes are in place. Thus, the process continues naturally after one round trip of the data receiver's Pull instruction and the data sender's Prepare instruction, without the need for an additional external scheduler.

[0061] It should be noted that the Pull instruction is only initiated when the local evaluation process of the data receiver actually consumes the input of a remote variable RV. When the host program executes an output or return statement, it determines whether the current input variable x is a remote variable RV (holding an actual value) or a dummy variable DV (holding only a symbolic placeholder) on the local participant node. If it is a remote variable RV, the output operation is performed locally on the local participant node (such as printing, writing to a file, or returning to the requester via an HTTP interface). If it is a dummy variable DV, the output operation is skipped. Unlike federated learning frameworks, this application does not have a unified result aggregation stage. Each participant node independently outputs or persists the computation results required by the host program to its local memory, without the need for centralized result collection. All participant nodes maintain equal status throughout the execution process, without the intervention of a central node, which can effectively protect the data privacy of each participant node.

[0062] In one exemplary embodiment, a computer device is provided. This computer device can be a local device of any participating node. The computer device can be a server or a terminal, and its internal structure diagram can be as follows: Figure 8As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores decentralized multi-party collaborative data processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a decentralized multi-party collaborative data processing method.

[0063] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0065] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0066] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0068] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0069] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A decentralized multi-party collaborative data processing method, characterized in that, The decentralized multi-party collaborative data processing method includes: All participating nodes in the collaboration process parse the computation request sent by the external client to obtain the target host program identifier; All participating nodes load the target host program based on the target host program identifier; the target host program contains embedded site identifiers of the participating nodes that indicate the executing program nodes; During the execution of the target host program, all participating nodes will decouple the execution flow into parallel symbolic plane and physical plane; The target host program is executed line by line in the symbolic plane to construct a location-aware computation graph; the location-aware computation graph includes a graph node set, a directed edge set, and a variable location mapping table; the information of each graph node in the graph node set includes a globally unique identifier, a function reference, the site identifier of the participating nodes of the executor node, and the graph node status; the information of each edge in the directed edge set includes an upstream graph node, a downstream graph node, the data variable identifier flowing through the corresponding edge, a data source node, a destination node, and an access control list; During the construction process, each participating node executes the location-aware computation graph using a hybrid evaluation strategy on the physical plane; wherein, if all input data of the current graph node of the location-aware computation graph is ready locally, early evaluation is triggered for asynchronous computation; if the value of the current graph node is explicitly dependent on the control flow or output statement, delayed evaluation is triggered for on-demand computation. During the early or delayed evaluation process, if the owner of the input data required by the current graph node is a remote participant node, the cross-node data transmission protocol is triggered. In response to the cross-node data transmission protocol, the participant node that needs the data actively initiates a data retrieval command, and at the same time, the participant node that provides the data pre-materializes the target data to the temporary storage area based on its local computing state to complete the cross-node data delivery. After the target host program is executed, all participating nodes obtain the exact same location-aware computation graph and their respective local cache values.

2. The decentralized multi-party collaborative data processing method according to claim 1, characterized in that, The step of executing the target host program line by line in the symbol plane to construct a location-aware computation graph includes: The target host program is parsed line by line in the symbol plane to generate an operator stream; When generating each operator in the operator flow, the operator type is determined. If the operator type is a computation operator, the input data of the computation operator is subjected to a common invariant check. If the check passes, a graph node and data dependency edge are added at the corresponding position of the initial position-aware computation graph to generate the current position-aware computation graph. If the operator type is a transport operator, update the variable position mapping table, update the owner node of the source variable corresponding to the transport operator to the destination node; add the destination node to the outgoing edge access control list of the upstream graph node corresponding to the source variable; If the operator type is a synchronization operator, the holder of the private data corresponding to the synchronization operator generates a criterion result and broadcasts the criterion result to other participating nodes; after receiving the criterion result, all participating nodes align their instruction pointers (IP) and enter the same branch to continue executing the target host program. After traversing the operator stream, the position-aware computation graph is generated.

3. The decentralized multi-party collaborative data processing method according to claim 2, characterized in that, The step of performing common-invariant verification on the input data of the computation operator includes: Verify whether the variable position mapping table Loc[x] of each input variable of the computation operator is equal to the site identifier of the target participant node of the computation operator; if it is not equal, it indicates that the operands corresponding to the input variables are distributed in the memory space of different participant nodes, throw a Fusion Gap exception, and prompt the developer to move the data to the same participant node through the δ_move operator.

4. The decentralized multi-party collaborative data processing method according to claim 3, characterized in that, If the verification passes, the process of adding graph nodes and data dependency edges at the corresponding positions in the initial location-aware computation graph to generate the current location-aware computation graph includes: After the co-occurrence check passes, the value of the current number of graph nodes plus 1 will be used as the globally unique identifier of the newly added graph node; Create a new graph node in the graph node set, and add a directed edge from the upstream graph node corresponding to the input variable to the new graph node in the directed edge set for each input variable; If the site identifier of the participant of the expected execution program node is equal to the site identifier of the newly added graph node, then the node state of the new graph node is set to pending state, and early evaluation is triggered in the physical plane; if the site identifier of the participant of the expected execution program node is not equal to the site identifier of the newly added graph node, then the node state of the new graph node is marked as symbolic state, retaining only the graph structure information without undertaking physical execution.

5. The decentralized multi-party collaborative data processing method according to claim 4, characterized in that, The step of triggering early evaluation for asynchronous computation if all input data for the current node of the location-aware computation graph is ready locally includes: If the upstream graph node of the current graph node has completed early evaluation, the local cached value of the upstream graph node is read directly; if the upstream graph node is still in the process of early evaluation, the early evaluation completion event of the upstream graph node is waited for; if the execution participant node of the upstream graph node is not the current participant node, the cross-node data transmission protocol is triggered to pull input data from the participant node that provides the data; after all the input data of the upstream graph node is in place, the current participant node performs local computation on the physical plane and writes the computation result into the local state of the current graph node.

6. The decentralized multi-party collaborative data processing method according to claim 5, characterized in that, The provision that if the value of the current graph node is explicitly dependent on the control flow or output statement, delayed evaluation is triggered for on-demand computation includes: Iterate through each input variable of the current graph node and recursively perform asynchronous evaluation operations on the input variables to make the input variables ready. The asynchronous evaluation operations include: if the input variable has been evaluated early, directly obtain and return the cached value of the input variable; if the input variable is in the early evaluation execution state, suspend the current thread and wait for the evaluation completion event of the input variable; if the input variable is in the suspended state, recursively trigger delayed evaluation of the input variable to backtrack up the data dependency chain to activate the ancestor node; if the target execution participant node of the input variable is not the current participant node, trigger the cross-node data transmission protocol to actively pull the value of the input variable from the participant node that owns the input variable. After detecting that all input variables of the current graph node are in place, the local operation method corresponding to the current graph node is executed on the physical plane, and the calculation result is written to the local state as input for the downstream synchronization contact.

7. A decentralized multi-party collaborative data processing system, characterized in that, The decentralized multi-party collaborative data processing system includes multiple participating node nodes with unique site identifiers; all participating node nodes execute the steps of the decentralized multi-party collaborative data processing method according to any one of claims 1-6.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the decentralized multi-party collaborative data processing method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the decentralized multi-party collaborative data processing method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the decentralized multi-party collaborative data processing method according to any one of claims 1-6.

Images