Method and electronic device for data federation

Abstract

The present disclosure provides a method and an electronic device for data federation, to improve the privacy security of each data participating in data federation on the basis of realizing data federation. It comprises: in response to the determination instruction of the user sending other nodes participating in data federation, determining other nodes participating in data federation according to the metadata information uploaded by each node in the blockchain; the metadata information includes the data attributes and data volume of the node local data; uploading the federated learning task and the initial training model to the blockchain, so that the other nodes participating in the data federation download the initial training model, and then perform federated learning according to the initial training model according to the federated learning task; and at the same time when the other nodes train the initial model according to the federated learning task, the initial training model is used to perform federated learning according to the federated learning task.

Description

Technical Field

[0001] This application relates to the field of data privacy and security technology, and in particular to a method and electronic device for data association. Background Technology

[0002] The rapid development of big data and artificial intelligence technologies relies heavily on massive amounts of data. While data from various industries continues to grow, the collaborative utilization of this data faces challenges related to data privacy and the protection of data rights.

[0003] Current technology involves transmitting data from various industries to a central server for data analysts to perform modeling and analysis, thus combining the data. However, aggregating data on a central server can lead to data leaks if the server is maliciously attacked or manipulated, resulting in inadequate protection of data privacy and security. Summary of the Invention

[0004] The purpose of this application is to provide a data federation method and electronic device for improving the privacy and security of the data being federated, based on the achievement of data federation.

[0005] According to one aspect of an exemplary embodiment, a data federation method is provided, applied to a target node in a blockchain, the method comprising:

[0006] In response to a user's instruction to determine other nodes that need to participate in the data federation, the other nodes that need to participate in the data federation are determined based on the metadata information uploaded to the blockchain by each node; the metadata information includes the data attributes and data volume of the node's local data;

[0007] The federated learning task and the initial training model are uploaded to the blockchain so that other nodes participating in the data federation can download the initial training model and then perform federated learning using the initial training model according to the federated learning task.

[0008] Federated learning is performed using the initial trained model according to the federated learning task.

[0009] The beneficial effects of this embodiment are as follows: By storing the data of each node locally and then uploading the metadata of the local data to the blockchain, the target node can determine other nodes that need to participate in data federation based on the metadata of each node in the blockchain, and achieve data federation among nodes through federated learning. Therefore, while achieving data federation, the local data of each node is not leaked, thereby improving the privacy and security of the data involved in the data federation.

[0010] In some exemplary implementations, the federated learning task includes metadata information of participants in the federated learning and termination conditions of the federated learning task.

[0011] The step of performing federated learning using the initial trained model according to the federated learning task includes:

[0012] The initial training model is trained using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, and the intermediate model parameters are uploaded to the blockchain.

[0013] Obtain the intermediate model parameters uploaded to the blockchain by the other nodes, and use a smart contract to securely aggregate the intermediate model parameters to obtain the updated model parameters;

[0014] The initial training model is updated using the updated model parameters. The process then returns to the step of training the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

[0015] The beneficial effects of this embodiment are as follows: Each node trains the initial training model using local data according to the federated learning task, and uses a smart contract to securely aggregate the intermediate parameters uploaded by each node to the blockchain to obtain new training parameters. The initial training model is then updated using the new training parameters, and this process continues until the termination condition is met. Thus, each node trains locally without having to disclose its own data, thereby improving data privacy and security.

[0016] In some exemplary embodiments, before performing federated learning using the initial training model according to the federated learning task after uploading the federated learning task and the initial training model to the blockchain so that other nodes participating in the data federation download the initial training model, the method further includes:

[0017] When it is detected that other nodes have uploaded their computational readiness status to the blockchain, a federated learning initiation command is sent to the blockchain to trigger the other nodes to perform federated learning using the initial training model according to the federated learning task.

[0018] The beneficial effects of this embodiment are: by sending a federated learning start command to trigger other nodes to start federated learning, the inaccurate results of federated learning are avoided due to nodes not performing federated learning.

[0019] In some exemplary implementations, the instruction for determining other nodes that need to participate in data federation includes metadata information of federated learning and the number of other nodes that need to participate in data federation;

[0020] The step of determining other nodes that need to participate in data collaboration based on the metadata information uploaded to the blockchain by each node includes:

[0021] Other nodes that need to participate in data federation are determined, wherein the other nodes that need to participate in data federation meet the following requirements: the other nodes are selected based on the metadata information of the federated learning, the number of other nodes that need to participate in data federation, and the metadata information uploaded to the blockchain by each node in the blockchain; the target node assigns the largest value to the other nodes.

[0022] The beneficial effects of this embodiment are: selecting other nodes through the metadata information of each node, and then determining whether a node can participate in this data federation based on the weight parameters assigned to other nodes by the target node. In this way, other nodes participating in the data federation are determined by assigning weight parameters, and each node can be incentivized to actively participate in the data federation.

[0023] In some exemplary embodiments, after performing federated learning using the initial trained model according to the federated learning task, the method further includes:

[0024] For any of the other nodes, an incentive value corresponding to the weight parameter is assigned to the other node using the preset correspondence between the weight parameter and the incentive value.

[0025] The beneficial effect of this embodiment is that by assigning incentive values ​​corresponding to the weight parameters to other nodes according to the weight parameters assigned to other nodes, each node can be encouraged to actively participate in data collaboration.

[0026] According to another aspect of an exemplary embodiment, an electronic device is provided, the electronic device being a target node in a blockchain, the electronic device including an input / output unit, a memory, and a processor:

[0027] The input / output unit is configured to receive metadata information and upload the metadata information to the blockchain; the metadata information includes the data attributes and data volume of the node's local data.

[0028] The memory is configured to store the initial training model;

[0029] The processor, connected to the input / output unit and the memory respectively, is configured as follows:

[0030] In response to a user's instruction to determine other nodes that need to participate in the data federation, the other nodes that need to participate in the data federation are determined based on the metadata information uploaded to the blockchain by each node in the blockchain.

[0031] The federated learning task and the initial training model are uploaded to the blockchain so that other nodes participating in the data federation can download the initial training model and then perform federated learning using the initial training model according to the federated learning task.

[0032] Federated learning is performed using the initial trained model according to the federated learning task.

[0033] In some exemplary implementations, the federated learning task includes metadata information of participants in the federated learning and termination conditions of the federated learning task.

[0034] The processor, when performing federated learning using the initially trained model according to the federated learning task, is specifically configured as follows:

[0035] The initial training model is trained using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, and the intermediate model parameters are uploaded to the blockchain.

[0036] Obtain the intermediate model parameters uploaded to the blockchain by the other nodes, and use a smart contract to securely aggregate the intermediate model parameters to obtain the updated model parameters;

[0037] The initial training model is updated using the updated model parameters. The process then returns to the step of training the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

[0038] In some exemplary embodiments, the processor is further configured to:

[0039] The federated learning task and the initial training model are uploaded to the blockchain so that other nodes participating in the data federation can download the initial training model and perform federated learning using the initial training model according to the federated learning task. Before the federated learning task is performed, if the federated learning task is performed, a federated learning start command is sent to the blockchain to trigger the other nodes to perform federated learning using the initial training model according to the federated learning task.

[0040] In some exemplary implementations, the instruction for determining other nodes that need to participate in data federation includes metadata information of federated learning and the number of other nodes that need to participate in data federation;

[0041] The processor executes the step of determining other nodes that need to participate in data federation based on the metadata information uploaded to the blockchain by each node in the blockchain, specifically configured as follows:

[0042] Other nodes that need to participate in data federation are determined, wherein the other nodes that need to participate in data federation meet the following requirements: the other nodes are selected based on the metadata information of the federated learning, the number of other nodes that need to participate in data federation, and the metadata information uploaded by each node in the blockchain to the blockchain; the target node assigns the largest weight parameter value to the other nodes.

[0043] In some exemplary embodiments, the processor is further configured to:

[0044] After performing federated learning using the initial training model according to the federated learning task, for any of the other nodes, an incentive value corresponding to the weight parameter is assigned to the other node using the preset correspondence between weight parameters and incentive values.

[0045] According to another aspect of the exemplary embodiments, a computer storage medium is provided, the computer storage medium storing computer program instructions that, when executed on a computer, cause the computer to perform the data association method as described above.

[0046] The data federation method described above in this application involves storing data locally on each node and then uploading the metadata of that local data to the blockchain. The target node can then determine which other nodes need to participate in the data federation based on the metadata of each node in the blockchain, and achieve data federation among the nodes through federated learning. Therefore, while achieving data federation, the local data of each node is not leaked, thus improving the privacy and security of the data involved in the data federation.

[0047] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the application embodiments, the drawings used in the application embodiments will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0050] Figure 2 This is one of the flowcharts illustrating a data joining method according to an embodiment of this application;

[0051] Figures 3A-3B This is a flowchart illustrating the process of determining other nodes participating in data federation in a data federation method according to an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of federated learning, a method for data federation according to an embodiment of this application;

[0053] Figure 5 This is a second schematic flowchart of a data association method according to an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of a data association apparatus according to one embodiment of the present application;

[0055] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0057] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0058] Hereinafter, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0059] Current technology involves transmitting data from various industry sectors to a central server for data analysts to perform modeling and analysis, thus enabling data aggregation. However, aggregating data onto a central server exposes it to the risk of data leakage if the server is maliciously attacked or manipulated. This compromises the privacy and security of the data.

[0060] In view of this, this application proposes a data federation method. This method involves storing the data of each node locally and then uploading the metadata of the local data to the blockchain. The target node can determine other nodes that need to participate in the data federation based on the metadata information in the blockchain, and achieve data federation among the nodes through federated learning. Therefore, while achieving data federation, the local data of each node is not leaked, thereby improving the privacy and security of the data involved in the data federation. Thus, this application proposes a data federation method and an electronic device. The following is a detailed description of this application with reference to the accompanying drawings.

[0061] like Figure 1 The diagram illustrates an application scenario of the data federation method of this disclosure. This application scenario diagram includes electronic device 110 and electronic devices 111 (including electronic devices 111A, 111B, and 111C). This disclosure uses one electronic device 110 and three electronic devices 111 as an example for illustration, but this disclosure does not limit the number of electronic devices 110 and 111. Electronic devices 110 and 111 are both nodes in the blockchain. The application scenario of the data federation method of this disclosure is introduced using electronic device 110 as the target node and electronic devices 111 as other nodes as an example.

[0062] In one possible scenario, electronic device 110, in response to a user's instruction to determine other nodes that need to participate in data federation, identifies the other electronic devices that need to participate in the data federation based on the metadata information uploaded to the blockchain by each electronic device in the blockchain. The metadata information includes the data attributes and data volume of the node's local data; for example, the identified other electronic devices that need to participate in the data federation include electronic device 111A and electronic device 111B. Then, electronic device 110 uploads the federated learning task and the initial training model to the blockchain, so that electronic devices 111A and 111B, which need to participate in the data federation, can download the initial training model and perform federated learning using the initial training model according to the federated learning task; and electronic device 110 performs federated learning using the initial training model according to the federated learning task.

[0063] It should be noted that the target node in this disclosure can be any node in the blockchain or a designated node in the blockchain, and this disclosure does not impose any restrictions on it.

[0064] The data association method in this disclosure will be described in detail below, such as... Figure 2 The diagram shown is a flowchart illustrating the method for combining data disclosed herein, which may include the following steps:

[0065] Step 201: In response to the user's instruction to determine other nodes that need to participate in the data federation, determine other nodes that need to participate in the data federation based on the metadata information uploaded to the blockchain by each node in the blockchain;

[0066] The metadata information includes data attributes and data volume of the node's local data; among which, data attributes include medical, financial, educational, and internet data.

[0067] It should be noted that each node must perform signature authentication according to the underlying blockchain technology requirements before uploading any data (metadata information, etc.) to the blockchain. This is to facilitate subsequent traceability and auditing.

[0068] In one embodiment, step 201 may be specifically implemented as: determining other nodes that need to participate in the data federation, wherein the other nodes that need to participate in the data federation meet the following requirements:

[0069] (1) Other nodes selected based on the metadata information of the federated learning, the number of other nodes that need to participate in the data federation, and the metadata information uploaded by each node in the blockchain to the blockchain;

[0070] (2) The target node assigns the largest weight parameter value to the other nodes.

[0071] The specific methods for determining this include the following two approaches:

[0072] Method 1: For example Figure 3A As shown, the specific steps can be implemented as follows:

[0073] Step 3A01: ​​In response to the user's instruction to determine other nodes that need to participate in the data federation, select other nodes to be determined based on the metadata information of the federated learning, the number of other nodes that need to participate in the data federation, and the metadata information of each node in the blockchain. The instruction to determine other nodes that need to participate in the data federation includes the metadata information of the federated learning and the number of other nodes that need to participate in the data federation.

[0074] For example, the metadata information for federated learning in the instruction to determine other nodes that need to participate in data federation includes financial data and 100GB (gigabyte). The number of other nodes that need to participate in data federation (i.e., the first preset number) is 3. Then, other nodes to be determined can be selected from the nodes in the block based on the metadata information of each node, which matches the metadata information of federated learning.

[0075] The number of other nodes to be determined is no less than 1 and no more than a first preset number. The specific number selected can be determined according to the actual situation, and the number of other nodes to be determined each time can be the same or different, which is not limited in this disclosure.

[0076] Step 3A02: Assign weight parameters to the other nodes to be determined;

[0077] Step 3A03: Within the specified time period, determine whether the weight parameter value assigned to the other nodes to be determined is the largest. If yes, proceed to step 3A04; otherwise, return to step 3A01.

[0078] Step 3A04: Determine the other nodes to be determined as other nodes to participate in the data federation;

[0079] Step 3A05: Determine whether the number of other nodes to participate in the data union meets the first preset number. If yes, proceed to step 3A06. If no, return to the specified step 3A01 until the number of other nodes participating in the data union meets the first preset number, then end.

[0080] Step 3A06: Determine the other nodes that are identified as potential participants in the data federation as the other nodes that need to participate in the data federation.

[0081] For example, the target node selects four other nodes to be determined based on the metadata information of each node: node 1, node 2, node 3, and node 4. The first preset number is 4. The target node then assigns weight parameters to each of the selected other nodes. If, within a specified time period, another target node also assigns a weight parameter to node 1, the target node compares whether the weight parameter assigned to node 1 by the other target node is higher than the weight parameter assigned to node 1 by the target node itself. If the target node determines that the weight parameter assigned to node 1 is the largest, then node 1 is determined as one of the other nodes to participate in the data union. Similarly, nodes 2, 3, and 4 are determined using the same method. If nodes 2 and 3 are determined as other nodes to participate in the data union, but node 4 is not, then only nodes 1, 2, and 3 are determined as other nodes to participate in the data union. Therefore, the number of other nodes to participate in the data union does not meet the first preset number. The target node then continues to select one node based on the metadata information of each node and assign a weight parameter until the number of determined other nodes to participate in the data union meets the first preset number, at which point the process ends.

[0082] Method 2: For example Figure 3B As shown, the specific steps include:

[0083] Step 3B01: In response to the user's instruction to determine other nodes that need to participate in the data federation, select other nodes to be determined based on the metadata information of the federated learning, the number of other nodes that need to participate in the data federation, and the metadata information uploaded to the blockchain by each node in the blockchain. The instruction to determine other nodes that need to participate in the data federation includes the metadata information of the federated learning and the number of other nodes that need to participate in the data federation.

[0084] The specific selection method is the same as the steps described above, and will not be repeated here.

[0085] The number of other nodes to be determined is greater than the second preset number (the number of other nodes that need to participate in the data union). The specific number of other nodes to be determined can be determined according to the specific actual situation, and this disclosure does not limit it. Moreover, the number of other nodes to be determined selected in each round can be the same or different.

[0086] Step 3B02: Assign weight parameters to the other nodes to be determined;

[0087] Step 3B03: Within a specified time period, the node with the largest weight parameter value assigned to the other nodes to be determined is identified as the other nodes to participate in the data federation.

[0088] Step 3B04: Determine whether the total number of other nodes to be included in the data union is equal to the second preset number. If yes, proceed to step 3B05; otherwise, proceed to step 3B06.

[0089] Step 3B05: Determine the other nodes identified as potential participants in the data federation as the other nodes required to participate in the data federation;

[0090] Step 3B06: Determine whether the total number of other nodes to be included in the data union is less than the second preset number. If yes, return to step 3B01; otherwise, proceed to step 3B07.

[0091] Step 3B07: Randomly select other nodes from the determined other nodes to participate in the data union, with the number equal to the second preset number, and determine the selected other nodes as the other nodes that need to participate in the data union.

[0092] For example, if the second preset quantity is 3, and the number of other nodes to be determined is 4, namely node 1, node 2, node 3, and node 4, then weight parameters are assigned to these four nodes. If node 1, node 2, and node 4 are determined to be other nodes to participate in the data union, and the number of other nodes to participate in the data union is equal to the second preset quantity, then node 1, node 2, and node 4 are determined to be the other nodes that need to participate in the data union.

[0093] If node 1, node 2, node 3, and node 4 are determined to be other nodes to participate in the data union, and the number of data unions is determined to be greater than the second preset number, then nodes equal to the second preset number are randomly selected from node 1, node 2, node 3, and node 4. If the randomly selected nodes include node 2, node 3, and node 4, then the other nodes that need to participate in the data union are determined to be: node 2, node 3, and node 4.

[0094] If node 1 and node 2 are identified as other nodes to participate in the data union, and the total number of nodes to be joined is less than the second preset number, then in response to the user's trigger selection operation on the nodes in the blockchain, other nodes to be selected are chosen based on the metadata information of each node in the blockchain. If the selected other nodes to be selected include node 6, node 7, and node 8, then weight coefficients are assigned to these three nodes. If node 7 is identified as another node to participate in the data union, and the total number of other nodes to participate in the data union is equal to the second preset number, then the other nodes to be joined in the data union include node 1, node 2, and node 7.

[0095] Step 202: Upload the federated learning task and the initial training model to the blockchain so that other nodes participating in the data federation can download the initial training model and then perform federated learning using the initial training model according to the federated learning task.

[0096] Step 203: While the other nodes are training using the initial model according to the federated learning task, federated learning is performed using the initial trained model according to the federated learning task.

[0097] The federated learning task includes metadata information of participants in the federated learning process and termination conditions for the federated learning task.

[0098] In one embodiment, step 203 may be implemented as follows: training the initial training model using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, and uploading the intermediate model parameters to the blockchain; obtaining the intermediate model parameters uploaded to the blockchain by other nodes, and using a smart contract to securely aggregate each intermediate model parameter to obtain updated model parameters; updating the initial training model using the updated model parameters, and returning to execute the step of training the initial training model using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

[0099] The termination condition may include model convergence at the number of training iterations or the target node. This can be determined based on the specific circumstances, and this disclosure does not impose any limitations.

[0100] For example, the metadata information participating in the federated learning task includes: financial data, 100GB. The termination condition is 1000 iterations. Figure 4 The diagram illustrates federated learning within a blockchain. After downloading the initial training model, each node trains the model using its local data (100GB of financial data) corresponding to the metadata information of the nodes participating in the federated learning, obtaining intermediate model parameters. These intermediate model parameters are then uploaded to the blockchain. Once all nodes (including the target node and other nodes) have uploaded their intermediate model parameters, each node securely aggregates these parameters using a smart contract, updates its own initial training model with the updated parameters, and then trains the model using its local data until it has been trained 1000 times.

[0101] It should be noted that the specific secure aggregation method in the smart contract is pre-set by the target node and then uploaded to the blockchain. Different federated learning tasks may correspond to different smart contracts. The specific methods are not limited here.

[0102] Therefore, this embodiment stores the data of each node locally and then uploads the metadata of the local data to the blockchain. The target node can determine other nodes that need to participate in data federation based on the metadata information in the blockchain, and achieve data federation among nodes through federated learning. Thus, while achieving data federation, the local data of each node is not leaked, thereby improving the privacy and security of the data involved in data federation.

[0103] To avoid the problem of inaccurate federated learning results due to other nodes not participating in federated learning, in one embodiment, the federated learning task and the initial training model are uploaded to the blockchain. After other nodes participating in the data federation download the initial training model, before performing federated learning using the initial training model according to the federated learning task, if it is detected that other nodes have uploaded their computational readiness status to the blockchain, a federated learning start command is sent to the blockchain to trigger the other nodes to perform federated learning using the initial training model according to the federated learning task.

[0104] For example, other nodes include node 1, node 2, and node 3. After each node has set up its local data and initial training model, it can send a computational readiness status to the blockchain. When the target node hears that nodes 1, 2, and 3 have all sent computational readiness statuses to the blockchain, it sends a federated learning start command to the blockchain. After hearing the federated learning start command, nodes 1, 2, and 3 perform federated learning using the initial training model according to the federated learning task.

[0105] Therefore, by sending a federated learning start command to trigger other nodes to begin federated learning, the inaccurate results of federated learning can be avoided if other nodes that need to participate in data federation do not perform federated learning.

[0106] To incentivize nodes in the blockchain to actively participate in data federation, in one embodiment, while other nodes are training using the initial model according to the federated learning task, after federated learning is performed using the initial training model according to the federated learning task, for any of the other nodes, an incentive value corresponding to the weight parameters is assigned to the other node using a preset correspondence between weight parameters and incentive values.

[0107] The corresponding incentive value can be set according to the specific actual situation, and this disclosure does not limit it.

[0108] To further understand the technical solutions provided in this disclosure, the following is in conjunction with... Figure 5 A detailed description of the methods for combining publicly available data may include the following steps:

[0109] Step 501: In response to the user's instruction to determine other nodes that need to participate in the data federation, determine other nodes that need to participate in the data federation based on the metadata information uploaded to the blockchain by each node; the metadata information includes the data attributes and data volume of the node's local data;

[0110] The instructions for determining other nodes that need to participate in data federation include metadata information for federated learning and the number of other nodes that need to participate in data federation.

[0111] The other nodes that need to participate in the data federation meet the following requirements: the other nodes are selected based on the metadata information of the federated learning, the number of other nodes that need to participate in the data federation, and the metadata information uploaded to the blockchain by each node in the blockchain; the target node assigns the largest value to the other nodes.

[0112] Step 502: Upload the federated learning task and the initial training model to the blockchain so that other nodes participating in the data federation can download the initial training model and then perform federated learning using the initial training model according to the federated learning task.

[0113] Step 503: Train the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, and upload the intermediate model parameters to the blockchain;

[0114] Step 504: Obtain the intermediate model parameters uploaded to the blockchain by the other nodes, and use a smart contract to securely aggregate the intermediate model parameters to obtain the updated model parameters;

[0115] Step 505: Update the initial training model using the updated model parameters, and return to the step of training the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

[0116] Step 506: For any of the other nodes, use the preset correspondence between weight parameters and incentive values ​​to assign incentive values ​​to the other nodes that correspond to the weight parameters.

[0117] The following section provides a detailed explanation of the data federation method disclosed herein, using a specific application scenario. Imagine a blockchain containing nodes 1, 2, 3, 4, 5, and 6. Nodes 1, 2, and 3 upload local data with medical attributes to the blockchain's metadata, and the data volume is sufficient. Node 4 uploads local data with financial attributes, and nodes 5 and 6 upload threat intelligence data. If node 1 is the target node, in response to a user's instruction to determine other nodes participating in the data federation, node 1 selects nodes 2 and 3 based on the federated learning metadata information, the number of other nodes required for data federation, and the metadata information of each node. Each node is then assigned a weight parameter. If, within a specified timeframe, node 1 assigns the node with the highest weight parameter value to nodes 2 and 3, then nodes 2 and 3 are determined to be other nodes participating in the data federation. Then, node 1 uploads the federated learning task and the initial training model to the blockchain, so that after other nodes participating in the data federation download the initial training model, they can perform federated learning using the initial training model according to the federated learning task. While nodes 2 and 3 are training using the initial model according to the federated learning task, node 1 is also performing federated learning using the initial training model according to the federated learning task.

[0118] Based on the same concept, such as Figure 6 As shown, this application also provides a data federation apparatus 600, which includes a module 610 for determining other nodes participating in data federation, an initial training model uploading module 620, and a federated learning module 630.

[0119] The other node determination module 610 is used to determine the other nodes that need to participate in the data union in response to the user's instruction to determine the other nodes that need to participate in the data union, based on the metadata information uploaded to the blockchain by each node in the blockchain; the metadata information includes the data attributes and data volume of the node's local data;

[0120] The initial training model upload module 620 is used to upload the federated learning task and the initial training model to the blockchain, so that other nodes participating in the data federation can download the initial training model and then use the initial training model to perform federated learning according to the federated learning task.

[0121] The federated learning module 630 is used to perform federated learning using the initial training model according to the federated learning task.

[0122] In one embodiment, the federated learning task includes metadata information of participants in the federated learning and termination conditions of the federated learning task.

[0123] The federated learning module 630 is specifically used for:

[0124] The initial training model is trained using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, and the intermediate model parameters are uploaded to the blockchain.

[0125] Obtain the intermediate model parameters uploaded to the blockchain by the other nodes, and use a smart contract to securely aggregate the intermediate model parameters to obtain the updated model parameters;

[0126] The initial training model is updated using the updated model parameters. The process then returns to the step of training the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

[0127] In one embodiment, the apparatus further includes:

[0128] The federated learning initiation command sending module 640 is used to send a federated learning initiation command to the blockchain when it detects that other nodes have uploaded the computational readiness status to the blockchain, so that other nodes participating in the data federation have downloaded the initial training model and before performing federated learning according to the federated learning task using the initial training model. This triggers the other nodes to perform federated learning according to the federated learning task using the initial training model.

[0129] In one embodiment, the instruction for determining other nodes that need to participate in data federation includes metadata information of federated learning and the number of other nodes that need to participate in data federation;

[0130] The module 610 for determining other nodes participating in the data concatenation is specifically used for:

[0131] Other nodes that need to participate in data federation are determined, wherein the other nodes that need to participate in data federation meet the following requirements: the other nodes are selected based on the metadata information of the federated learning, the number of other nodes that need to participate in data federation, and the metadata information uploaded to the blockchain by each node in the blockchain; the target node assigns the largest value to the other nodes.

[0132] In one embodiment, the apparatus further includes:

[0133] The incentive value allocation module 650 is used to allocate an incentive value corresponding to the weight parameters to any other node after performing federated learning using the initial training model according to the federated learning task, based on the preset correspondence between the weight parameters and incentive values.

[0134] The following is combined Figure 7 A detailed introduction to each component of the electronic device 700 is provided below:

[0135] The RF circuit 710 can be used for receiving and transmitting data during communication or a call. Specifically, after receiving downlink data from the base station, the RF circuit 710 sends it to the processor 730 for processing; additionally, it sends uplink data to be transmitted to the base station. Typically, the RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc.

[0136] Furthermore, the RF circuit 710 can also communicate wirelessly with networks and other terminals. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0137] WiFi technology is a short-range wireless transmission technology. The electronic device 700 can connect to an access point (AP) via the WiFi module 790 (i.e., the wireless network module described above in this disclosure), thereby enabling access to the data network. The WiFi module 790 can be used for receiving and sending data during communication.

[0138] The electronic device 700 can physically connect to other terminals through the communication interface 780. Optionally, the communication interface 780 can be connected to the communication interface of the other terminal via a cable to realize data transmission between the electronic device 700 and the other terminal.

[0139] The electronic device 700 is capable of performing communication services, and it needs to have data transmission capabilities, meaning it needs to include a communication module. Although Figure 7 The RF circuit 710, the WiFi module 790, and the communication interface 780 are shown, but it is understood that the electronic device 700 contains at least one of the above-mentioned components or other communication modules (such as a Bluetooth module) for data transmission.

[0140] The memory 740 can be used to store software programs and modules. The processor 730 executes various functional applications and data processing of the electronic device 700 by running the software programs and modules stored in the memory 740. Furthermore, when the processor 730 executes the program code in the memory 740, it can implement the embodiments of this disclosure. Figure 7 Part or all of the process.

[0141] Optionally, the memory 740 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications (such as communication applications), and modules for WLAN connection; the data storage area may store data created based on the use of the terminal.

[0142] In addition, the memory 740 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0143] The input unit 750 can be used to receive digital or character information input by the user, and to generate key signal inputs related to user settings and function control of the electronic device 700.

[0144] Optionally, the input unit 750 may include a touch panel 751 and other input terminals 752.

[0145] The touch panel 751, also known as a touchscreen, can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 751), and drive corresponding connection devices according to a pre-set program. Optionally, the touch panel 751 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 730, and can also receive and execute commands from the processor 730. Furthermore, the touch panel 751 can be implemented using various types of touch technologies, such as resistive, capacitive, infrared, and surface acoustic wave.

[0146] Optionally, the other input terminal 752 may include, but is not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0147] The display unit 760 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 700. The display unit 760 is the display system of the electronic device 700 used to present the interface and realize human-computer interaction.

[0148] The display unit 760 may include a display panel 761. Optionally, the display panel 761 may be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0149] Furthermore, the touch panel 751 may cover the display panel 761. When the touch panel 751 detects a touch operation on or near it, it transmits the information to the processor 730 to determine the type of touch event. Subsequently, the processor 730 provides corresponding visual output on the display panel 761 according to the type of touch event.

[0150] Although Figure 7 In this embodiment, the touch panel 751 and the display panel 761 are two independent components to realize the input and output functions of the electronic device 700. However, in some embodiments, the touch panel 751 and the display panel 761 can be integrated to realize the input and output functions of the electronic device 700.

[0151] The processor 730 is the control center of the electronic device 700. It connects various components through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 740, and calling data stored in the memory 740, it performs various functions of the electronic device 700 and processes data, thereby realizing various services based on the electronic device.

[0152] Optionally, the processor 730 may include one or more processing units. Optionally, the processor 730 may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 730.

[0153] The camera 770 is used to enable the electronic device 700 to take pictures or videos.

[0154] The electronic device 700 also includes a power supply 720 (such as a battery) for supplying power to various components. Optionally, the power supply 720 can be logically connected to the processor 730 through a power management system, thereby enabling the power management system to manage functions such as charging, discharging, and power consumption.

[0155] Although not shown, the electronic device 700 may also include at least one sensor, which will not be described further here.

[0156] In some possible implementations, various aspects of the methods provided in the embodiments of this application may also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the data combination method according to various exemplary embodiments of this application as described in this specification.

[0157] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0158] The program product for executing data association according to embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a server device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an information transmission device or apparatus.

[0159] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a periodic network operating system, apparatus, or device.

[0160] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0161] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computing device.

[0162] This application embodiment of the data processing method also provides a computing device readable storage medium, meaning that the content is not lost after power failure. This storage medium stores software programs, including program code. When the program code runs on a computing device, the software program, when read and executed by one or more processors, can implement any of the data processing schemes described in this application embodiment.

[0163] The present application has been described above with reference to block diagrams and / or flowcharts illustrating methods, apparatus (systems), and / or computer program products according to embodiments of the present application. It should be understood that a block of a block diagram and / or flowchart, as well as combinations of blocks of block diagrams and / or flowcharts, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, and / or other programmable data processing means to produce a machine, such that the instructions, executable via the computer processor and / or other programmable data processing means, create methods for implementing the functions / actions specified in the blocks of the block diagrams and / or flowcharts.

[0164] Accordingly, this application can also be implemented using hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, this application can take the form of a computer program product on a computer-usable or computer-readable storage medium, having computer-usable or computer-readable program code implemented in the medium for use by or in conjunction with an instruction execution system. In the context of this application, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, transmit, or deliver a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] Although this application has been described in conjunction with specific features and embodiments, it will be apparent that various modifications and combinations can be made therein without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of the application as defined by the appended claims, and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this application.

[0166] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for data concatenation, characterized in that, The method, applied to a target node in a blockchain, includes: In response to a user-sent instruction to determine other nodes that need to participate in the data federation, the instruction includes metadata information of the federated learning and the number of other nodes that need to participate in the data federation; Based on the metadata information uploaded to the blockchain by each node, other nodes that need to participate in the data federation are determined; the metadata information includes the data attributes and data volume of the node's local data. The step of determining other nodes that need to participate in data collaboration based on the metadata information uploaded to the blockchain by each node includes: Other nodes that need to participate in data federation are determined, wherein the other nodes that need to participate in data federation meet the following requirements: the other nodes are selected based on the metadata information of the federated learning, the number of other nodes that need to participate in data federation, and the metadata information uploaded to the blockchain by each node in the blockchain; the target node assigns the largest value to the other nodes; The federated learning task and the initial training model are uploaded to the blockchain, so that other nodes participating in the data federation can download the initial training model and then perform federated learning using the initial training model according to the federated learning task; and... Federated learning is performed using the initial training model according to the federated learning task; While the other nodes utilize the initial training model according to the federated learning task, federated learning is performed using the initial training model and the target node's local data according to the federated learning task.

2. The method according to claim 1, characterized in that, The federated learning task includes metadata information of the participants in the federated learning and the termination conditions of the federated learning task. The step of performing federated learning using the initial trained model according to the federated learning task includes: The initial training model is trained using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, and the intermediate model parameters are uploaded to the blockchain. Obtain the intermediate model parameters uploaded to the blockchain by the other nodes, and use a smart contract to securely aggregate the intermediate model parameters to obtain the updated model parameters; The initial training model is updated using the updated model parameters. The process then returns to the step of training the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

3. The method according to claim 1, characterized in that, The method further includes uploading the federated learning task and the initial training model to the blockchain so that other nodes participating in the data federation can download the initial training model, and before performing federated learning using the initial training model according to the federated learning task: When it is detected that other nodes have uploaded their computational readiness status to the blockchain, a federated learning initiation command is sent to the blockchain to trigger the other nodes to perform federated learning using the initial training model according to the federated learning task.

4. The method according to claim 1, characterized in that, After performing federated learning using the initial trained model according to the federated learning task, the method further includes: For any of the other nodes, an incentive value corresponding to the weight parameter is assigned to the other node using the preset correspondence between the weight parameter and the incentive value.

5. An electronic device, characterized in that, The electronic device is a target node in the blockchain, and the electronic device includes an input / output unit, a memory, and a processor. The input / output unit is configured to receive metadata information and upload the metadata information to the blockchain; the metadata information includes the data attributes and data volume of the node's local data. The memory is configured to store the initial training model; The processor, connected to the input / output unit and the memory respectively, is configured as follows: In response to a user-sent instruction to determine other nodes that need to participate in the data federation, the instruction includes metadata information of the federated learning and the number of other nodes that need to participate in the data federation; Based on the metadata information uploaded to the blockchain by each node, other nodes that need to participate in the data collaboration are identified. The step of determining other nodes that need to participate in data collaboration based on the metadata information uploaded to the blockchain by each node includes: Other nodes that need to participate in data federation are determined, wherein the other nodes that need to participate in data federation meet the following requirements: the other nodes are selected based on the metadata information of the federated learning, the number of other nodes that need to participate in data federation, and the metadata information uploaded to the blockchain by each node in the blockchain; the target node assigns the largest value to the other nodes; The federated learning task and the initial training model are uploaded to the blockchain, so that other nodes participating in the data federation can download the initial training model and then perform federated learning using the initial training model according to the federated learning task; and... Federated learning is performed using the initial training model according to the federated learning task; While the other nodes utilize the initial training model according to the federated learning task, federated learning is performed using the initial training model and the target node's local data according to the federated learning task.

6. The electronic device according to claim 5, characterized in that, The feature is that, The federated learning task includes metadata information of the participants in the federated learning and the termination conditions of the federated learning task. The processor, when performing federated learning using the initially trained model according to the federated learning task, is specifically configured as follows: The initial training model is trained using local data corresponding to the metadata information of the participants in the federated learning to obtain intermediate model parameters, and the intermediate model parameters are uploaded to the blockchain. Obtain the intermediate model parameters uploaded to the blockchain by the other nodes, and use a smart contract to securely aggregate the intermediate model parameters to obtain the updated model parameters; The initial training model is updated using the updated model parameters. The process then returns to the step of training the initial training model using local data corresponding to the metadata information of the federated learning participants to obtain intermediate model parameters, until the termination condition of the federated learning task is met.

7. The electronic device according to claim 5, characterized in that, The processor is also configured to: The federated learning task and the initial training model are uploaded to the blockchain so that other nodes participating in the data federation can download the initial training model and perform federated learning using the initial training model according to the federated learning task. Before the federated learning task is performed, if the federated learning task is performed, a federated learning start command is sent to the blockchain to trigger the other nodes to perform federated learning using the initial training model according to the federated learning task.

8. The electronic device according to claim 5, characterized in that, The processor is also configured to: After performing federated learning using the initial training model according to the federated learning task, for any of the other nodes, an incentive value corresponding to the weight parameter is assigned to the other node using the preset correspondence between weight parameters and incentive values.

Images