Data sharing and collaboration method and system in federated learning environment

Abstract

The present invention provides a data sharing and collaboration method and system in a federated learning environment, the method comprising: a data providing node extracts data from a local middle platform layer and publishes it in a first preset protocol format; a data evaluation node obtains data published by a data providing node, and performs data relative value evaluation on the data published by the data providing node based on a shared data security evaluation protocol; a data collaborative task initiating node obtains target data, obtains a target model based on target data training, and performs data dynamic value evaluation on the target data; a data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data; a data collaborative task initiating node calculates the incentive of the data providing node in the current data collaborative task, and pays and settles the data providing node according to the incentive. The present invention can reduce the difficulty of large-scale implementation of federated learning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data sharing and collaboration method and system in a federated learning environment. Background Art

[0002] Federated learning was first proposed by Google in 2016. It enables data to participate in multi-party collaborative modeling without leaving the local area, and can also ensure information security and personal privacy when exchanging data. Under the premise of ensuring compliance and security, it connects data from multiple different nodes to explore the collaborative economic value of big data. For example, in the medical field, the sample data of a single hospital is not enough to build a diagnostic model with sufficient accuracy. The problem of model accuracy can be solved by using federated learning technology to realize collaborative modeling of samples from multiple hospitals. At the same time, federated learning can also ensure that the data does not expose the privacy of patients.

[0003] The process of federated learning includes: first, the cloud server sends a unified model to the server nodes of multiple medical institutions. Each medical institution node is trained with local data, and then the trained model parameters are transmitted back to the cloud server in an encrypted state through the trusted computing network. The cloud server integrates and updates the model parameters obtained from multiple medical institution nodes, and then sends them back to each node for the next round of model training, and iterates step by step until the model converges to complete a complete federated learning process.

[0004] In general, federated learning is still in the early stage of pilot technology verification. To achieve truly large-scale application, at least the following problems need to be solved: How does the modeling initiator know what shared data is available for modeling in a multi-party joint modeling environment; how to evaluate the shared data of federated learning while protecting privacy; and how to incentivize the shared data behavior of federated learning data providers. Summary of the invention

[0005] The present invention provides a data sharing and collaboration method and system in a federated learning environment, which is used to at least solve the problems in the prior art of how a modeling initiator knows what shared data is currently available for modeling in a multi-party joint modeling environment; how to evaluate the shared data of federated learning while protecting privacy; and how to incentivize the shared data behavior of the data provider of federated learning.

[0006] In a first aspect, the present invention provides a data sharing and collaboration method in a federated learning environment, comprising:

[0007] The data providing node extracts data from the local middle platform layer and publishes it according to the first preset protocol format;

[0008] Other nodes in the trusted computing network act as data evaluation nodes, obtain data published by the data providing nodes, and perform data relative value evaluation on the data published by the data providing nodes based on the shared data security evaluation protocol;

[0009] The data collaborative task initiating node obtains target data, trains a target model based on the target data, and performs a dynamic data value assessment on the target data, wherein the data collaborative task initiating node is any node in the trusted computing network;

[0010] During the application of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data;

[0011] The data collaborative task initiating node conducts a comprehensive evaluation of the data released by the data providing node based on the data relative value evaluation results, the data dynamic value evaluation results and the data application value evaluation results, and calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol based on the comprehensive evaluation results, and pays and settles to the data providing node according to the incentive.

[0012] In one embodiment, the data providing node extracts data from the local middle platform layer and publishes it according to the first preset protocol format, including:

[0013] The data providing node extracts data from the local area and prepares the data before publishing;

[0014] The data providing node publishes target information according to a first preset protocol format, wherein the target information includes: information of the data and authentication information corresponding to the data;

[0015] The data providing node sends a data publishing message to other nodes in the trusted computing network.

[0016] In one embodiment, other nodes in the trusted computing network serve as data evaluation nodes, obtain data published by the data providing node, and perform data relative value evaluation on the data published by the data providing node based on the shared data security evaluation protocol, including:

[0017] Other nodes in the trusted computing network serve as data evaluation nodes. After receiving the data publishing message, they negotiate and reach an agreement with the data providing node on the evaluation method to form a shared data security evaluation protocol that all nodes participating in federated learning must comply with.

[0018] The data evaluation node obtains the data published by the data providing node, and based on the shared data security evaluation protocol, evaluates the relative value of the data published by the data providing node relative to the data evaluation node, and obtains the data relative value evaluation result corresponding to the data providing node;

[0019] The data evaluation node ranks the data providing nodes based on the data relative value evaluation result, and stores the ranking result and the data relative value evaluation result on the chain.

[0020] In one embodiment, the data collaborative task initiating node obtains target data, and based on the target data, trains to obtain a target model, including:

[0021] The data collaborative task initiating node sends a data sharing and collaboration request, receives the data in the trusted computing network that is allowed to be shared with the data collaborative task initiating node and is returned according to the score ranking, as well as the collaborative task matching degree data and participation willingness feedback from each data providing node;

[0022] The data collaborative task initiating node selects target data and establishes a connection channel with the data providing node corresponding to the target data;

[0023] The data collaborative task initiating node and the data providing node corresponding to the target data negotiate on the task and reach a consensus agreement;

[0024] The data collaborative task initiating node executes the data collaborative task according to the reached agreement, wherein the data collaborative task is to train a target model based on the target data.

[0025] In one embodiment, the step of performing dynamic data value assessment on the target data includes:

[0026] During the execution of the data collaborative task, the data collaborative task initiating node calculates the contribution of the data published by each data providing node in the target data to the collaborative task performance change according to the effective participation update cycle of the target data, and obtains the dynamic value of the data;

[0027] According to the dynamic value of the data, the relative value of the data of each data providing node is revised twice, and the dynamic value of the data is saved on the chain.

[0028] In one embodiment, during the application process of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data, including:

[0029] In the target model application phase, the data collaborative task initiating node calculates the data application value of each data providing node according to the number of calls of the target model and the number of times the data of each data providing node participates in collaborative calculation during each call;

[0030] The relative value of the data of each data providing node is revised secondary according to the data application value of each data providing node, and the data application value is stored on the chain.

[0031] In a second aspect, the present invention provides a data sharing and collaboration system in a federated learning environment, including: a plurality of nodes participating in federated learning that form a trusted computing network by means of a consortium chain;

[0032] Wherein, the node includes an agent layer, an interface layer and a middle platform layer;

[0033] The proxy layer and the interface layer are isolated by an external network firewall, and the interface layer and the middle platform layer are isolated by an internal network firewall;

[0034] The proxy layer includes: a multi-party collaborative computing management module, a data relative value assessment module, a trusted data traceability audit module, a data dynamic value assessment module, a data application value assessment module and a value incentive module;

[0035] The interface layer includes: a federated learning shared data management module and a federated learning shared data publishing module;

[0036] The middle platform layer includes: business middle platform and data middle platform.

[0037] In one embodiment, the multi-party collaborative computing management module includes:

[0038] The collaborative task management submodule is used to receive collaborative task requests, and issue and receive collaborative task summaries;

[0039] The participating node management submodule is used to send data collaboration requests to data providing nodes, receive collaborative task matching data and participation willingness feedback from each data providing node, comprehensively select participating nodes, record the process data of participating nodes participating in collaborative tasks, and perform life cycle management of participating nodes;

[0040] The protocol management submodule is used to generate and manage the protocol solution negotiated between the data collaborative task initiating node and the participating nodes on the collaborative task.

[0041] In a third aspect, the present invention provides an electronic device comprising a memory and a memory storing a computer program, wherein when the processor executes the program, the steps of the data sharing and collaboration method in the federated learning environment described in the first aspect are implemented.

[0042] In a fourth aspect, the present invention provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the steps of the data sharing and collaboration method in the federated learning environment described in the first aspect.

[0043] The data sharing and collaboration method and system in a federated learning environment provided by the present invention solve the problems of data sharing in a federated learning environment, evaluating shared data in federated learning while protecting privacy, and incentivizing the data sharing behavior of data providers in federated learning. It can promote data collaboration among various federated learning participants and collaboration among multiple federated learning frameworks, and reduce the difficulty of large-scale implementation of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A flowchart of a data sharing and collaboration method in a federated learning environment provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a data sharing and collaboration system in a federated learning environment provided by an embodiment of the present invention;

[0047] Figure 3 A functional architecture diagram of each node in the data sharing and collaboration system in the federated learning environment provided by an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] At present, to achieve real large-scale application, at least the following problems need to be solved:

[0051] 1) In a multi-party joint modeling environment, how does the modeling initiator know what shared data is available for modeling (i.e., what data problems are solved). Currently released federated learning frameworks include FATE, TFF, etc., and their focus is mainly on how the two or more parties involved in collaborative modeling can implement AI modeling technology solutions based on privacy data. For the shared data involved in modeling, it is required to synchronize offline in advance in terms of quantity, content, format, fields, etc. to maintain consistency. However, in the actual federated learning environment, especially when multiple entities are involved, this assumption does not hold.

[0052] 2) How to evaluate the shared data of federated learning while protecting privacy (i.e., solving the problem of which data is available). There are currently two main methods for evaluating data in federated learning: quality assessment, which focuses on the multi-dimensional characteristics of the data content itself, such as completeness, accuracy, precision, consistency, timeliness, etc., and value assessment, which further comprehensively considers the cost of data in the production process and the output in different tasks while evaluating data quality. At present, there are various challenges to data evaluation in practical applications, such as the difficulty in quantifying and evaluating data features, the low efficiency of data set quality assessment, the difficulty in calculating data costs, the subjective differences in value evaluation, the difficulty in predicting the value of data use, the dynamic changes in data value, etc.

[0053] 3) How to incentivize the sharing of data by federated learning data providers (i.e., how to reward them after data usage is exhausted). The core of federated learning is to aggregate data from multiple parties for collaborative modeling. If there is no sufficient economic incentive, it is impossible to ensure that federated participants provide sufficient high-quality data. Without high-quality data, it is impossible to obtain a high-quality model. In related technologies, data providers are incentivized based on data consumption behavior and satisfaction scores.

[0054] In order to solve or at least partially solve the above technical problems, the present invention provides a data sharing and collaboration method and system in a federated learning environment. The present invention designs and executes relevant mechanisms according to the complete life cycle of shared data in federated learning, from data publishing, data retrieval, data evaluation, model training to model application, and comprehensively considers the characteristics of each stage.

[0055] like Figure 1 As shown, it is a flow chart of a data sharing and collaboration method in a federated learning environment provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0056] Step 100: The data providing node extracts data from the local middle platform layer and publishes it according to the first preset protocol format;

[0057] Specifically, the data provider node extracts data from the local middle platform layer.

[0058] The local middle platform layer refers to the business middle platform and the data middle platform. The business middle platform includes the user center, entertainment center, billing center, and communication center, etc. The data middle platform includes the asset catalog, data algorithm, AI model, and data query module, etc.

[0059] After the data providing node extracts the data, it prepares for data publishing, and after the data is ready, it publishes it according to the first preset protocol format.

[0060] The first preset protocol format is a predetermined protocol format, and all nodes participating in the collaborative computing use this protocol format to publish data.

[0061] Step 101: other nodes in the trusted computing network serve as data evaluation nodes, obtain data published by the data providing node, and perform data relative value evaluation on the data published by the data providing node based on a shared data security evaluation protocol;

[0062] Among them, the shared data security assessment protocol is a protocol that all nodes participating in collaborative computing need to comply with after the data assessment node and the data provider node negotiate and reach an agreement on the assessment method. The content of the negotiation includes: assessment content, assessment fields, and security algorithms, etc.

[0063] The data evaluation node performs a data relative value evaluation on the data released by the data providing node based on the shared data security evaluation protocol. The data relative value evaluation refers to the evaluation of the value of the data released by the data providing node relative to the data evaluation node.

[0064] Step 102: The data collaborative task initiating node obtains target data, trains a target model based on the target data, and performs a dynamic data value assessment on the target data, wherein the data collaborative task initiating node is any node in the trusted computing network;

[0065] Optionally, the data collaboration task initiating node obtains target data by sending a data sharing and collaboration request, and then performs model training based on the target data to obtain a target model, thereby realizing the data collaboration task.

[0066] The data collaborative task initiating node performs dynamic value assessment on the target data during the model training process.

[0067] Step 103: During the application of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data;

[0068] After the data collaboration task is completed, the target model will be deployed in the trusted computing network in a secret sharing manner. During the application phase of the target model, the data collaboration task initiating node will evaluate the data application value of the data published by each data providing node in the target data.

[0069] Step 104: The data collaborative task initiating node conducts a comprehensive evaluation of the data released by the data providing node based on the data relative value evaluation results, the data dynamic value evaluation results and the data application value evaluation results, and calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol based on the comprehensive evaluation results, and pays and settles to the data providing node according to the incentive.

[0070] The data sharing and collaboration method in a federated learning environment provided by the present invention solves the problems of data sharing in a federated learning environment, evaluating the shared data of federated learning while protecting privacy, and incentivizing the shared data behavior of data providers of federated learning. It can promote data collaboration among various federated learning participants and collaboration among multiple federated learning frameworks, and reduce the difficulty of large-scale implementation of federated learning.

[0071] Optionally, the data providing node extracts data from the local middle platform layer and publishes it according to the first preset protocol format, including:

[0072] The data providing node extracts data from the local area and prepares the data before publishing;

[0073] The data providing node publishes target information according to a first preset protocol format, wherein the target information includes: information of the data and authentication information corresponding to the data;

[0074] The data providing node sends a data publishing message to other nodes in the trusted computing network.

[0075] The preparation before release includes: description, evaluation, update and maintenance of the overall situation. The overall situation of data includes: overall introduction of data, application field, data label, total data size, data integrity, data acquisition method and sampling method, data timeliness, basic statistical characteristics of data, key field definition, etc.

[0076] The data information may include: data label, general introduction, application field, field definition, sample data, CRC information, etc. However, the data information does not include any real information of the data and statistical information of any field, including data volume, variance, average value, etc., so as to protect the data privacy of the data node from leakage.

[0077] Authentication information is used to control which nodes are allowed to access which data in what form, and may include: a list of users or user levels allowed to access the information, the interaction protocol of secure multi-party computing, and the fields allowed to be accessed, etc.

[0078] Optionally, the data providing node sends data publishing messages to all other nodes in the trusted computing network that are interested in the target data according to the registration information of each node. It should be noted that if the number of other nodes in the trusted computing network is too large, the messages can be sent in batches according to the nodes or by sampling according to the threshold.

[0079] The present invention provides a mechanism for publishing and retrieving shared data under privacy protection. The published and retrieved shared data do not include any real information of the data and statistical information of any field, and adopts a multi-party secure computing protocol to realize data evaluation, training and application, thereby protecting data privacy while applying shared data.

[0080] Optionally, other nodes in the trusted computing network serve as data evaluation nodes, obtain data published by the data providing node, and perform data relative value evaluation on the data published by the data providing node based on the shared data security evaluation protocol, including:

[0081] Other nodes in the trusted computing network serve as data evaluation nodes. After receiving the data publishing message, they negotiate and reach an agreement with the data providing node on the evaluation method to form a shared data security evaluation protocol that all nodes participating in federated learning must comply with.

[0082] The data evaluation node obtains the data published by the data providing node, and based on the shared data security evaluation protocol, evaluates the relative value of the data published by the data providing node relative to the data evaluation node, and obtains the data relative value evaluation result corresponding to the data providing node;

[0083] The data evaluation node ranks the data providing nodes based on the data relative value evaluation result, and stores the ranking result and the data relative value evaluation result on the chain.

[0084] Wherein, evaluating the relative value of the data released by the data providing node relative to the data evaluating node includes:

[0085] Through the secure multi-party computing protocol, the number of public samples of both parties, the statistical characteristics of the eigenvalues ​​of the data providing nodes, and the importance coefficient of each eigenvalue relative to the label (such as WOE, IV value, etc.) are found;

[0086] Based on the number of public samples, the statistical characteristics of the eigenvalues ​​of the data providing nodes and the importance coefficient of each eigenvalue relative to the label, the relative value of the data released by the data providing node relative to the data evaluation node is comprehensively evaluated; wherein, the higher the sample matching degree, the higher the relative value score, the more important the eigenvalue, the higher the relative value score, and the more eigenvalues, the higher the relative value score.

[0087] The data evaluation node ranks the data providing nodes based on the data relative value evaluation results, and saves the ranking results and the data relative value evaluation results on the chain, thereby ensuring the credibility of the evaluation data quality to the greatest extent.

[0088] Optionally, the data collaborative task initiating node obtains target data, and based on the target data, trains to obtain a target model, including:

[0089] The data collaborative task initiating node sends a data sharing and collaboration request, receives the data in the trusted computing network that is allowed to be shared with the data collaborative task initiating node and is returned according to the score ranking, as well as the collaborative task matching degree data and participation willingness feedback from each data providing node;

[0090] The data collaborative task initiating node selects target data and establishes a connection channel with the data providing node corresponding to the target data;

[0091] The data collaborative task initiating node and the data providing node corresponding to the target data negotiate on the task and reach a consensus agreement;

[0092] The data collaborative task initiating node executes the data collaborative task according to the reached agreement, wherein the data collaborative task is to train a target model based on the target data.

[0093] It can be understood that the data collaborative task initiating node sends a data sharing and collaboration request, receives the data allowed to be shared with the data collaborative task initiating node in the trusted computing network returned according to the score ranking, as well as the collaborative task matching degree data and participation willingness feedback from each data providing node; the data collaborative task initiating node selects the target data based on the data score, the collaborative task matching degree data and participation willingness feedback from each data providing node, and establishes a connection channel with the data providing node corresponding to the target data.

[0094] After the connection channel is established, the data collaboration task initiating node and the data providing node corresponding to the target data negotiate on the task and reach a consensus agreement. The negotiation content may include: the data collaboration interaction mode of both parties, security encryption algorithm, interaction frequency, interaction termination / termination conditions, settlement price and strategy of both parties, model training method, and reach a consensus.

[0095] Then, the data collaborative task initiating node starts to execute the data collaborative task according to the reached agreement, wherein the data collaborative task is to train a target model based on the target data.

[0096] Optionally, the performing dynamic data value assessment on the target data includes:

[0097] During the execution of the data collaborative task, the data collaborative task initiating node calculates the contribution of the data published by each data providing node in the target data to the collaborative task performance change according to the effective participation update cycle of the target data, and obtains the dynamic value of the data;

[0098] According to the dynamic value of the data, the relative value of the data of each data providing node is revised twice, and the dynamic value of the data is saved on the chain.

[0099] Among them, calculating the contribution of the data published by each data providing node in the target data to the change in the performance of the collaborative task means calculating the contribution of the data published by each data providing node in the target data to the final training completion to obtain the target model, which is also called the dynamic contribution value.

[0100] The data collaborative task initiating node makes a secondary correction to the relative value of the data of each data providing node based on the contribution degree, obtains the dynamic value of the data, achieves a balance between the relative static value of the data and the dynamic training value, and saves the dynamic value of the data on the chain.

[0101] Optionally, during the application process of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data, including:

[0102] In the target model application phase, the data collaborative task initiating node calculates the data application value of each data providing node according to the number of calls of the target model and the number of times the data of each data providing node participates in collaborative calculation during each call;

[0103] The relative value of the data of each data providing node is revised secondary according to the data application value of each data providing node, and the data application value is stored on the chain.

[0104] Among them, data application value refers to the evaluation of the value of data during the model application stage.

[0105] Specifically, the data collaborative task initiating node calculates the data application value of each data providing node based on the number of calls to the target model and the number of times the data of each data providing node participates in collaborative calculation during each call. At the same time, the relative value of the data of each data providing node is revised secondary based on the data application value of each data providing node, and the data application value is saved on the chain to achieve a balance between the data model training value and the data application value.

[0106] The present invention provides a multi-dimensional dynamic evaluation method for data value based on the basic value of data in the data evaluation stage, the dynamic contribution value in the model training stage, and the data application value in the model application stage. At the same time, there is no need to introduce a third party in the evaluation process, thereby protecting the data privacy of the data provider.

[0107] The following describes the data sharing and collaboration system in the federated learning environment provided by the present invention.

[0108] Figure 2 A schematic diagram of a data sharing and collaboration system in a federated learning environment provided for an embodiment of the present invention includes: a plurality of nodes participating in federated learning that form a trusted computing network in the form of a consortium chain;

[0109] Each node participating in federated learning forms a trusted computing network through a consortium chain. Each data provider and each model owner serves as an independent node in the data collaboration network. Each node is responsible for the management and publication of local shared data through its local node, and interacts with the virtual agent nodes of other nodes in the network through its virtual agent node. It is mainly responsible for the evaluation, management, auditing and incentive functions of shared data.

[0110] Figure 3 The functional architecture diagram of each node in the data sharing and collaboration system in the federated learning environment provided by the embodiment of the present invention is as follows: Figure 3 As shown, the node includes an agent layer 310, an interface layer 320 and a middle platform layer 330;

[0111] The proxy layer 310 and the interface layer 320 are isolated by an external network firewall, and the interface layer 320 and the middle platform layer 330 are isolated by an internal network firewall;

[0112] The proxy layer 310 includes: a multi-party collaborative computing management module 3, a data relative value assessment module, a trusted data traceability audit module, a data dynamic value assessment module, a data application value assessment module and a value incentive module;

[0113] The interface layer includes: a federated learning shared data management module and a federated learning shared data publishing module;

[0114] The middle platform layer includes: business middle platform and data middle platform.

[0115] Among them, the federated learning shared data management module is used to extract data from the local middle station according to a unified format and field dimension and prepare the data before publishing.

[0116] The federated learning shared data publishing module is used to publish target information according to a first preset protocol format, wherein the target information includes: information about the data and authentication information corresponding to the data.

[0117] Among them, the data relative value assessment module is used to evaluate the relative value of data released by the data providing node relative to the data evaluation node based on the shared data security assessment protocol, and obtain the data relative value assessment result corresponding to the data providing node.

[0118] Optionally, the data relative value assessment module includes: an assessment method library submodule and an assessment calculation submodule, wherein:

[0119] The evaluation method library submodule is used to define methods or calculation formulas for evaluating, comparing, and calculating different types of data and information;

[0120] The evaluation calculation submodule is used for the specific quantification of the evaluation conditions of different dimensions (including but not limited to subjective scoring or objective calculation), as well as the comprehensive summary of multi-dimensional evaluation results (for example: weighted summation).

[0121] Optionally, the multi-party collaborative computing management module is used for accepting, initiating, matching and screening participants, reaching and maintaining collaborative agreements, etc.

[0122] The multi-party collaborative computing management module includes:

[0123] The collaborative task management submodule is used to receive collaborative task requests, and issue and receive collaborative task summaries;

[0124] The collaborative task summary includes: application field, total data volume, statistical characteristics, key fields, etc.; collaborative task life cycle management, etc.

[0125] The participating node management submodule is used to send data collaboration requests to data providing nodes, receive collaborative task matching data and participation willingness feedback from each data providing node, comprehensively select participating nodes, record the process data of participating nodes participating in collaborative tasks, and perform life cycle management of participating nodes;

[0126] The protocol management submodule is used to generate and manage the protocol solution negotiated between the data collaborative task initiating node and the participating nodes on the collaborative task.

[0127] The negotiation content may include (but not limited to) the data collaboration interaction mode, security encryption mechanism, interaction frequency, interaction termination / suspension conditions, settlement prices and strategies of both parties, etc., to form a data sharing and collaboration agreement that is complied with by all parties involved; the protocol management submodule is responsible for the generation, maintenance management and life cycle management of the protocol, and can generate a typical protocol library.

[0128] Optionally, the data dynamic value assessment module is used to dynamically track and evaluate the execution of collaborative tasks with the participation of all parties during the model training process, and estimate the changes and contributions brought by each participant (including the initiating node) to the collaborative task performance.

[0129] Optionally, the data application value assessment module is used to calculate the data application value of each data providing node during the target model application phase, when the data collaborative task initiating node is based on the number of calls of the target model and the number of times the data of each data providing node participates in collaborative calculation during each call.

[0130] Optionally, the value incentive module is used to distribute incentives and collect and manage relevant data records based on the relative value of data, the dynamic contribution value of the model training process, and the comprehensive data value obtained by the application value of the data model, combined with the data of the participant's participation process records and life cycle, and with reference to the agreement reached between the data collaborative task initiating node and each data providing node. The value incentive module may include sub-modules such as value data aggregation, incentive generation and execution, payment settlement, and data management.

[0131] Optionally, the trusted data traceability audit module is used to build a trusted data storage network using a consortium chain approach, which enables the on-chain storage of key information on the relative value of data in the data evaluation phase, the dynamic contribution value in the model training phase, and the long-term application value in the model application phase, for subsequent audits to ensure data authenticity.

[0132] In an optional embodiment, based on the data sharing and collaboration system in the above-mentioned federated learning environment, the data sharing and collaboration method in the federated learning environment provided by the present invention includes the following steps:

[0133] The federated learning shared data management module of the data providing node extracts data from the local middle platform layer and prepares the data before publishing;

[0134] The federated learning shared data publishing module of the data providing node publishes the target information according to the first preset protocol format, wherein the target information includes: information of the data and authentication information corresponding to the data;

[0135] The data providing node sends a data publishing message to the multi-party collaborative computing management module in the trusted computing network through the proxy layer, and the multi-party collaborative computing management module forwards the data publishing information to all other interested nodes according to the registration information of each node;

[0136] Other nodes in the trusted computing network serve as data evaluation nodes. After receiving the data publishing message, they negotiate and reach an agreement with the data providing node on the evaluation method to form a shared data security evaluation protocol that all nodes participating in federated learning must comply with.

[0137] The data relative value assessment module of the data assessment node obtains the data published by the data providing node, and based on the shared data security assessment protocol, assesses the relative value of the data published by the data providing node relative to the data assessment node, and obtains the data relative value assessment result corresponding to the data providing node;

[0138] The multi-party collaborative computing management module of the data evaluation node ranks the data providing nodes based on the data relative value evaluation results exceeding the specified threshold number, publishes the ranking results to the entire trusted computing network, and sends the ranking results and the data relative value evaluation results to the trusted data traceability audit module for on-chain storage;

[0139] The data collaborative task initiating node sends a data sharing and collaboration request to the multi-party collaborative computing management module. The multi-party collaborative computing management module returns the data that is currently allowed to be shared with the data collaborative task initiating node in the trusted computing network to the data collaborative task initiating node according to the score ranking, and returns the collaborative task matching degree data and participation willingness feedback from each data providing node. The data collaborative task initiating node receives the data that is allowed to be shared with the data collaborative task initiating node in the trusted computing network returned according to the score ranking, as well as the collaborative task matching degree data and participation willingness feedback from each data providing node;

[0140] The data collaborative task initiating node selects target data and establishes a connection channel with the data providing node corresponding to the target data;

[0141] The data collaborative task initiating node and the data providing node corresponding to the target data negotiate on the task and reach a consensus agreement;

[0142] The data collaborative task initiating node performs the data collaborative task according to the reached agreement, wherein the data collaborative task is to train a target model based on the target data;

[0143] During the execution of the data collaborative task, the data dynamic value assessment module of the data collaborative task initiating node calculates the contribution of the data published by each data providing node in the target data to the change in the collaborative task efficiency according to the effective participation update cycle of the target data, obtains the data dynamic value, and feeds the data dynamic value back to the multi-party collaborative computing management module; the multi-party collaborative computing management module makes a secondary correction to the relative value of the data of each data providing node according to the data dynamic value, and sends the data dynamic value to the trusted data traceability audit module for on-chain storage;

[0144] In the target model application phase, the data application value assessment module of the data collaborative task initiating node calculates the data application value of each data providing node according to the number of calls of the target model and the number of times the data of each data providing node participates in collaborative computing during each call, and feeds back the data application value to the multi-party collaborative computing management module;

[0145] The multi-party collaborative computing management module performs secondary correction on the relative value of the data of each data providing node according to the data application value of each data providing node, and sends the data application value to the trusted data traceability audit module for on-chain storage;

[0146] The multi-party collaborative computing management module conducts a comprehensive evaluation of the data published by the data providing node based on the data relative value evaluation results, the data dynamic value evaluation results and the data application value evaluation results, and then calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol based on the comprehensive evaluation results through the value incentive module, and the data collaborative task initiating node pays and settles to the data providing node according to the incentive.

[0147] In summary, the data sharing and collaboration method in the federated learning environment provided by the present invention has the following advantages: the release and retrieval of shared data are unified in a protocol manner, which is convenient for collaboration among various participants and frameworks and avoids new data island phenomena; the relative value of the data provider's data is evaluated by an evaluator, avoiding the tendency to absolutize the data value; the value of the data is evaluated based on the data contribution degree during the model training process, avoiding the tendency to staticize the data value, and combining the comprehensive performance of data evaluation, training and application of multiple links to obtain the final value of the data, which is closer to the true value of the data; the value that may be derived from data collaboration tasks and results is tracked, managed and allocated; the value of the data provider relative to the data application party is evaluated in a privacy-protected manner, which can promote data collaboration among various federated learning participants and collaboration among multiple federated learning frameworks, and effectively reduce the difficulty of large-scale implementation of federated learning.

[0148] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410, a communication interface (Communication Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the computer program in the memory 430 to execute the steps of the data sharing and collaboration method in the federated learning environment, for example, including: the data providing node extracts data from the local middle platform layer and publishes it in accordance with the first preset protocol format; other nodes in the trusted computing network act as data evaluation nodes, obtain the data published by the data providing node, and perform data relative value evaluation on the data published by the data providing node based on the shared data security evaluation protocol; the data collaborative task initiating node obtains the target data, trains the target model based on the target data, and performs data dynamic value evaluation on the target data, wherein the data collaborative task initiating node is any node in the trusted computing network; during the application of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data; the data collaborative task initiating node performs a comprehensive evaluation on the data published by the data providing node based on the data relative value evaluation result, the data dynamic value evaluation result and the data application value evaluation result, and calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol based on the comprehensive evaluation result, and pays and settles to the data providing node according to the incentive.

[0149] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0150] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data sharing and collaboration method in the federated learning environment provided by the above methods, and the method includes: a data providing node extracts data from a local middle platform layer and publishes it in accordance with a first preset protocol format; other nodes in the trusted computing network act as data evaluation nodes, obtain data published by the data providing node, and perform data relative value evaluation on the data published by the data providing node based on a shared data security evaluation protocol; a data collaborative task initiating node obtains target data, and based on the The target data is trained to obtain a target model, and a dynamic data value assessment is performed on the target data, wherein the data collaborative task initiating node is any node in the trusted computing network; during the application of the target model, the data collaborative task initiating node performs a data application value assessment on the data published by each data providing node in the target data; based on the data relative value assessment results, the data dynamic value assessment results and the data application value assessment results, the data collaborative task initiating node performs a comprehensive assessment on the data published by the data providing node, and based on the comprehensive assessment results, calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol, and pays and settles to the data providing node according to the incentive.

[0151] On the other hand, an embodiment of the present application also provides a processor-readable storage medium, which stores a computer program, and the computer program is used to enable the processor to execute the data sharing and collaboration methods in the federated learning environment provided by the above embodiments, for example, including: the data providing node extracts data from the local middle platform layer and publishes it in accordance with the first preset protocol format; other nodes in the trusted computing network act as data evaluation nodes, obtain the data published by the data providing node, and perform data relative value evaluation on the data published by the data providing node based on the shared data security evaluation protocol; the data collaborative task initiating node obtains the target data, and based on the target data, trains to obtain the target model , and performs a data dynamic value assessment on the target data, wherein the data collaborative task initiating node is any node in the trusted computing network; during the application of the target model, the data collaborative task initiating node performs a data application value assessment on the data released by each data providing node in the target data; based on the data relative value assessment results, the data dynamic value assessment results and the data application value assessment results, the data collaborative task initiating node performs a comprehensive assessment on the data released by the data providing node, and calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol based on the comprehensive assessment results, and pays and settles the data providing node according to the incentive.

[0152] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0153] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0154] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data sharing and collaboration method in a federated learning environment, characterized in that: include: The data providing node extracts data from the local middle platform layer and publishes it according to the first preset protocol format; Other nodes in the trusted computing network act as data evaluation nodes, obtain data published by the data providing nodes, and perform data relative value evaluation on the data published by the data providing nodes based on the shared data security evaluation protocol; The data collaborative task initiating node obtains target data, trains a target model based on the target data, and performs a dynamic data value assessment on the target data, wherein the data collaborative task initiating node is any node in the trusted computing network; During the application of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data; The data collaborative task initiating node conducts a comprehensive evaluation of the data released by the data providing node based on the data relative value evaluation results, the data dynamic value evaluation results and the data application value evaluation results, and calculates the incentive of the data providing node in the current data collaborative task according to the second preset protocol based on the comprehensive evaluation results, and pays and settles to the data providing node according to the incentive.

2. The data sharing and collaboration method in a federated learning environment according to claim 1, characterized in that: The data providing node extracts data from the local middle platform layer and publishes it according to the first preset protocol format, including: The data providing node extracts data from the local area and prepares the data before publishing; The data providing node publishes target information according to a first preset protocol format, wherein the target information includes: information of the data and authentication information corresponding to the data; The data providing node sends a data publishing message to other nodes in the trusted computing network.

3. The data sharing and collaboration method in a federated learning environment according to claim 1, characterized in that: Other nodes in the trusted computing network serve as data evaluation nodes, obtain data published by the data providing nodes, and perform data relative value evaluation on the data published by the data providing nodes based on the shared data security evaluation protocol, including: Other nodes in the trusted computing network serve as data evaluation nodes. After receiving the data publishing message, they negotiate and reach an agreement with the data providing node on the evaluation method to form a shared data security evaluation protocol that all nodes participating in federated learning must comply with. The data evaluation node obtains the data published by the data providing node, and based on the shared data security evaluation protocol, evaluates the relative value of the data published by the data providing node relative to the data evaluation node, and obtains the data relative value evaluation result corresponding to the data providing node; The data evaluation node ranks the data providing nodes based on the data relative value evaluation result, and stores the ranking result and the data relative value evaluation result on the chain.

4. The data sharing and collaboration method in a federated learning environment according to claim 1, characterized in that: The data collaborative task initiating node obtains target data, and based on the target data, trains to obtain a target model, including: The data collaborative task initiating node sends a data sharing and collaboration request, receives the data in the trusted computing network that is allowed to be shared with the data collaborative task initiating node and is returned according to the score ranking, as well as the collaborative task matching degree data and participation willingness feedback from each data providing node; The data collaborative task initiating node selects target data and establishes a connection channel with the data providing node corresponding to the target data; The data collaborative task initiating node and the data providing node corresponding to the target data negotiate on the task and reach a consensus agreement; The data collaborative task initiating node executes the data collaborative task according to the reached agreement, wherein the data collaborative task is to train a target model based on the target data.

5. The data sharing and collaboration method in a federated learning environment according to claim 1, characterized in that: The performing dynamic data value assessment on the target data includes: During the execution of the data collaborative task, the data collaborative task initiating node calculates the contribution of the data published by each data providing node in the target data to the change in the collaborative task efficiency according to the effective participation update cycle of the target data, and obtains the dynamic value of the data; According to the dynamic value of the data, the relative value of the data of each data providing node is revised twice, and the dynamic value of the data is saved on the chain.

6. The data sharing and collaboration method in a federated learning environment according to claim 1, characterized in that: In the application process of the target model, the data collaborative task initiating node performs data application value evaluation on the data published by each data providing node in the target data, including: In the target model application phase, the data collaborative task initiating node calculates the data application value of each data providing node according to the number of calls of the target model and the number of times the data of each data providing node participates in collaborative calculation during each call; The relative value of the data of each data providing node is revised secondary according to the data application value of each data providing node, and the data application value is stored on the chain.

7. A data sharing and collaboration system in a federated learning environment, characterized in that: include: Multiple nodes participating in federated learning form a trusted computing network through a consortium chain; Wherein, the node includes an agent layer, an interface layer and a middle platform layer; The proxy layer and the interface layer are isolated by an external network firewall, and the interface layer and the middle platform layer are isolated by an internal network firewall; The proxy layer includes: a multi-party collaborative computing management module, a data relative value assessment module, a trusted data traceability audit module, a data dynamic value assessment module, a data application value assessment module and a value incentive module; The interface layer includes: a federated learning shared data management module and a federated learning shared data publishing module; The middle platform layer includes: business middle platform and data middle platform.

8. The data sharing and collaboration system in a federated learning environment according to claim 7, characterized in that: The multi-party collaborative computing management module includes: The collaborative task management submodule is used to receive collaborative task requests, and issue and receive collaborative task summaries; The participating node management submodule is used to send data collaboration requests to data providing nodes, receive collaborative task matching data and participation willingness feedback from each data providing node, comprehensively select participating nodes, record the process data of participating nodes participating in collaborative tasks, and perform life cycle management of participating nodes; The protocol management submodule is used to generate and manage the protocol solution negotiated between the data collaborative task initiating node and the participating nodes on the collaborative task.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the data sharing and collaboration method in the federated learning environment described in any one of claims 1 to 6 are implemented.

10. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the steps of the data sharing and collaboration method in a federated learning environment as described in any one of claims 1 to 6.

Images