Blockchain-based federated reorganization method and system for deep neural network model

Abstract

A blockchain-based deep neural network model federal reorganization method comprises the following steps: constructing a consortium chain for model federal reorganization; setting an intelligent contract for model federal reorganization; storing data in a model federal reorganization process; training a source model of a model training party and submitting the source model to a model reorganization verification party for federal reorganization; model federal reorganization, the model reorganization verification party performs federal reorganization on the source model, and then obtains a target model; and calculating a contribution value of the model training party. The application also comprises a system for blockchain-based deep neural network model federal reorganization. The application considers the reusability of the deep neural network model, the limited computing power and data when a single-node user model is reorganized, and then performs multi-node model federal reorganization. Meanwhile, the application considers the data security and privacy in the federal process, and the mutual distrust between the multiple nodes in cooperation in the federal process, establishes a consortium chain and an intelligent contract based on Hyperledger Fabric, and provides a method for safe and trusted cooperation of multiple nodes.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of federal deep learning and blockchain, and relates to a deep neural network model federal reorganization method and system based on blockchain. BACKGROUND

[0002] The core of blockchain technology is decentralized data management, which is different from the traditional centralized data management method. The blockchain technology stores data on multiple nodes through a distributed network, ensuring the data's non-tamperability and security.

[0003] Deep neural network model reorganization is a deep neural model reuse task. With the rapid development of deep learning, the number of pre-trained deep neural network models on the network is huge, and reusing pre-models can reduce the cost of training new models from scratch. The goal of model reorganization is to extract the knowledge of multiple source models that handle different tasks and fuse them into a single compact target model under the condition that only the source model and its inference output results are available and the training data of the source model cannot be obtained, so as to obtain a lightweight target model that can handle all source tasks with high performance.

[0004] Model reorganization tasks are only implemented on a single node, but the limited computing power and data of single-node users limit the further expansion of model reorganization, and the setting of target tasks is relatively limited. The federal learning method is based on the mutual trust between multiple nodes, but this trust foundation cannot be achieved among multiple nodes in the entire network, and the security of data sharing between nodes cannot be guaranteed during the fusion process. At the same time, model federal reorganization needs to allocate a fair and reasonable contribution to the nodes participating in the model reorganization process. SUMMARY

[0005] To solve the above problems, the application provides a deep neural network model federal reorganization method and system based on blockchain.

[0006] The application considers the limited computing power and data of single-node users and proposes a method of multi-node model federal reorganization. Based on the blockchain technology, the cooperation agreement between multiple nodes is reached through a smart contract, ensuring the safety, credibility, and fairness of multi-node cooperation, and further proposing a deep neural network model federal reorganization method and system based on blockchain. The technical scheme of the application is as follows:

[0007] A deep neural network model federal reorganization method and system based on blockchain, comprising the following steps:

[0008] 1. Constructing a consortium chain for model federal reorganization;

[0009] The alliance chain is built based on the Hyperledger Fabric framework to meet the model federation reorganization cooperation. The alliance chain is composed of four roles, including alliance members, alliance operators responsible for the maintenance and update of the alliance chain, smart contract auditors, and blockchain developers. The nodes participating in the model federation reorganization cooperation are the alliance members. The Hyperledger Fabric framework is a basic framework of the alliance chain, which provides permission management, ledger maintenance, smart contract, and other blockchain technologies. Users only need to set the corresponding business logic. On the basis of the framework, the business between the alliance members is set as the model federation reorganization cooperation. The business is published by the customer side and received by the model training side and the model reorganization verification side. The three parties reach cooperation through the smart contract, thereby building the alliance chain of the model federation reorganization.

[0010] 2. Set the smart contract of the model federation reorganization;

[0011] The smart contract is signed between the customer side, the model training side, and the model reorganization verification side, and is automatically executed according to the set content to achieve the cooperation and constraints of the three parties. The basic smart contract of the model federation reorganization is as follows:

[0012] 1) Customer side:

[0013] a) Set the target task of the model federation reorganization and the final model receiving standard;

[0014] b) Provide the target task detection data (i.e., the subset of the sampled target task dataset) for the model reorganization verification side, set the task reward of the model reorganization verification side;

[0015] c) Specify the source model required by the model training side, set the minimum transferability score of the model training side that can participate in the reorganization process of step 5.1, set the training reward and follow the contribution calculation method in step 6 to allocate the corresponding reward to the model training side.

[0016] 2) Model training side:

[0017] a) After the first transferability evaluation of step 5.1 by the model reorganization verification side, the model training side node that fails to enter the selection can choose to withdraw from the task in advance, or continue to participate in the task and finally obtain the corresponding reward through the contribution calculation method in step 6;

[0018] b) Provide the source model of the target task for the model reorganization verification side;

[0019] 3) Model reorganization verification side:

[0020] a) Based on the detection data of the target task, the source model is federated and reorganized to obtain the target model, and the target model meeting the acceptance standard is submitted to the customer side for verification;

[0021] b) Calculate the contribution value for each model training party by the contribution method of step 6.

[0022] 3. Store data in the model federation reorganization process;

[0023] The probe data between the client party and the model reorganization verification party, the target model sharing, and the source model sharing between the model training party and the model reorganization verification party are realized through a distributed storage (DS) system to ensure the security of data and private machine learning model sharing. The DS system makes the access of these data and learning models more affordable, faster, safer, and better meets the communication needs in the federal learning process. After the node stores the data or private model in the DS system, a unique hash value address representing these network assets is generated and shared to ensure the integrity and authorization of the sharing process. Specifically, we will use the InterPlanetary File System (IPFS) for distributed storage in the subsequent process.

[0024] 4. Train the source model of the model training party;

[0025] The model federation reorganization is divided into two parts, one part is the source model training task, and the other part is the federation reorganization of the source model and the verification of the target model. The two parts are completed by the model training party and the model reorganization verification party, respectively, and the model training party provides the source model required by the model reorganization verification party. The source model training is completed by two steps of local pre-training and knowledge distillation.

[0026] The model training party trains the local pre-training model based on the local data, and uses M = {M1, M2, …, M N} to represent the local pre-training model of the N model training party nodes. To ensure privacy, knowledge distillation is performed on the local pre-training model to obtain the source model, which is provided to the model reorganization verification party for the federation reorganization and verification process.

[0027] The source model learns knowledge from the local pre-training model based on the feature-based knowledge distillation method. In the knowledge distillation process, due to the heterogeneity of the network structure, a translation module is used to align the feature output dimensions of the local pre-training model and the source model, and the source model output is converted to a predefined output length through the translation module. Let and represent the length-aligned features of the source model and the local pre-training model M n of the nth node, respectively. represent the source model and local pre-training model parameters of the nth node when communicating with the model reorganization verification party for the tth time. X is the training data of the model M n . At the same time, the intermediate output of the source model is made to imitate the pre-training model, and the feature knowledge distillation loss is calculated as follows:

[0028]

[0029] The source model parameter update adopts gradient descent as follows:

[0030]

[0031] After local knowledge refining, the source model parameters The source model uploaded to the interstellar file system as a model reorganization verification party obtains the model parameters through a unique hash address provided by the model training party.

[0032] 5. Model federation reorganization;

[0033] The source model obtained in step 4 is subjected to model federation reorganization by the model reorganization verification party. The source model provided by the model training party is trained based on different local data, and it is necessary to select a model more beneficial to the target model for federation reorganization, so the model federation reorganization has the following two steps: (1) source model selection; (2) federation reorganization and performance verification.

[0034] 5.1. Select a beneficial model in the source model set for federation reorganization in step 5.2. In order to measure whether the source model is beneficial to the target model, a saliency map is used to measure the transferability of the knowledge contained in the source model, thereby measuring the benefit. Specifically, based on the probe data and model requirements provided by the customer party, the model reorganization verification party first trains to obtain the original target model, and calculates the average saliency map of the target model on the probe data, and then based on the source model, calculates the average saliency map of each source model on the probe data, and the transferability score of the source model is calculated by pair calculation. The saliency map calculation formula is as follows:

[0035]

[0036] wherein Ψ k represents the hidden features of the kth layer of the source model, represents the saliency map of the kth layer of the source model of the nth node on the probe data D T for an input image x j in D

[0037] The saliency map of the kth layer of the source model of the nth node on all probe data D T is represented as The mean value of all is obtained as wherein S is the total number of images in D T The transferability γ n,T of the two models and the normalized selection probability p n of the source model can be estimated as follows, wherein is the saliency map of the kth layer of the target model,

[0038]

[0039]

[0040] Select p according to the source model selection probability n The top K source models with the minimum transferability score set by the customer party are selected to perform the reorganization update in step 5.2.

[0041] 5.2. The selected source models are reorganized, and the performance of the target model obtained after reorganization is verified. Based on the original target model parameters obtained in the first step 5.1, the source models are reorganized in a greedy reorganization manner, so that after each target model reorganization, the target model parameters are updated, and the target model performance is improved. In the greedy reorganization process, first, the source model to be added is verified and tested using the probe data provided by the customer party, then the source model parameters are sorted in descending order according to the verification test performance and averaged with the target model parameters, and the fused target model is again verified and tested based on the probe data. If the performance is improved, the updated target model parameters are retained, and the acc i For the i-th source model training party node, repeat the above operation until the end of the target model performance improvement value for the source model sorted in descending order. This is considered as one iteration of the federal reorganization. When the target model performance reaches the acceptance standard set by the customer party, the model reorganization verification party uploads the target model to the interstellar file system and provides the customer party with the corresponding hash address.

[0042] 6. Calculate the contribution value of the model training party;

[0043] According to the transferability score of the source model provided by each model training party in step 5 and the target model performance index improvement in the federal reorganization process, the contribution of each model training party to the target model is calculated by normalization. R t,i i represents the contribution value of the i-th model training party node of the model training party and the model reorganization verification party in the t-th federal communication iteration, and the contribution is calculated as follows:

[0044] R t,i = α * acc t,i + β * p t,i (4)

[0045] Where acc t,i is the number of target model performance improvements of the selected source model in step 5.2, p t,i is the transferability score of the selected source model in step 5.1, and if not selected, p t,i = 0, and α, β are variable parameters for adjusting acc t,i , p t,iThe weight proportion of the two parameters is unified and the magnitude of the two digital quantities is unified.

[0046] Considering that the growth of the target model performance is approximately logarithmic nonlinear growth, the average contribution value obtained each time cannot be directly accumulated, so an exponential moving average method with bias correction is adopted, with the number of iterations increasing, the weight of the contribution value of recent iterations is increased, and the weight of the contribution value of previous iterations is reduced, and finally the cumulative contribution value of the ith model training party after t rounds of communication iterations is The calculation is as follows:

[0047]

[0048] When t=0 The contribution value R of each time is defaulted t,i Adopted, so according to the formula The value of μ can be calculated. Through the following bias correction, the final contribution value E of the final ith model training party can be obtained i , and according to the final contribution value E i , the model training party can obtain the corresponding task reward. Wherein B is the total reward of the model training task given by the customer party, B i is the corresponding reward that the ith model training party can obtain:

[0049]

[0050]

[0051] The implementation of the deep neural network model federal reorganization method and system based on the blockchain like the present application includes a model federal reorganization alliance chain module, a model federal reorganization smart contract module, a distributed storage data sharing module, a source model training module, a model federal reorganization module, and a model reorganization contribution distribution module.

[0052] The application discloses a blockchain-based deep neural network model federal reorganization method and system, and first, an alliance chain conforming to a model federal reorganization cooperative task is constructed based on a Hyperledger Fabric framework. According to the characteristics of the model federal reorganization task, a corresponding smart contract is formulated to determine the business logic, and a safe and reliable cooperation method of a customer party, a model training party and a model reorganization verification party is established. Based on the fast and safe transmission and storage of the InterPlanetary File System, a distributed storage system, the safety of data sharing in the model federal reorganization process is further ensured, and the communication transmission requirement in the model federal reorganization process is met. Secondly, the model training party trains a local pre-training model based on local data, and knowledge distillation is performed, and the obtained model is used as a source model in the model federal reorganization process, so that the privacy of the pre-training model is ensured while the model knowledge is transmitted. The model reorganization verification party selects a source model with higher task migration based on a saliency map, the selected source model is fused into a target model in a greedy reorganization manner, and multiple communication iterations are performed until the target model reaches the customer party acceptance standard. Finally, according to a contribution calculation method, the migration score of the source model provided by the model training party and the target model performance index improvement number in the reorganization process are normalized to obtain a contribution value of each iteration, and all iteration contribution values are subjected to an exponential moving average calculation with bias correction, so that a final contribution value of each model training party is obtained, and the task reward distribution is performed according to the weight.

[0053] The application has the advantages that considering the reusability of the deep neural network model, the limited computing power and data in the single-node user model reorganization, multi-node model federal reorganization is performed. Meanwhile, the data safety and privacy in the federal process and the mutual distrust between the nodes in the federal cooperation are considered, an alliance chain and a smart contract based on the Hyperledger Fabric are established, and a multi-node safe and reliable cooperation method is provided. The InterPlanetary File System is introduced to ensure the data safety and privacy. Secondly, the parameter updating process of the federal reorganization uses the greedy reorganization, so that the target model performance is stably increased, and the model reorganization contribution calculation is facilitated. Finally, based on the source model migration and the target model performance improvement number in the model federal reorganization process, the normalized calculation and the exponential moving average calculation with bias correction are performed to perform a fair contribution distribution for the model training party. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 FIG. 1 is a deep neural network model federal reorganization flowchart based on a blockchain according to the application.

[0055] Figure 2 FIG. 3 is a source model training schematic diagram according to the application.

[0056] Figure 3 FIG. 5 is a model federal reorganization and performance verification process schematic diagram according to the application. DETAILED DESCRIPTION

[0057] The technical solutions of the present application will be clearly and completely explained and described below in combination with the drawings, and the exemplary embodiments of the present application will be described. The detailed description should not be considered as a limitation of the present application, but should be understood as a more detailed description of the application of certain aspects and features of the present application.

[0058] The present application provides a multi-party cooperation vehicle detection and classification method based on a blockchain-based deep neural network model federal reorganization method, comprising the following steps:

[0059] 1. Construct a consortium chain for model federal reorganization;

[0060] A consortium chain that meets the model federal reorganization cooperation is built based on the Hyperledger Fabric framework. The consortium chain is composed of four roles: consortium members, consortium operators responsible for the maintenance and update of the consortium chain, smart contract auditors, and blockchain developers. The nodes participating in the model federal reorganization cooperation serve as consortium members. The Hyperledger Fabric framework is a basic framework for consortium chains, providing permission management, ledger maintenance, smart contract, and other blockchain technologies. Users only need to set the corresponding business logic. On the basis of this framework, the business between consortium members is set as model federal reorganization cooperation, and the vehicle detection and classification business is published by the customer side and received by the model training side and the model reorganization verification side. The three parties reach cooperation through a smart contract, thereby constructing a consortium chain for model federal reorganization Figure 1 ).

[0061] 2. Set the smart contract for model federal reorganization;

[0062] The smart contract is signed between the customer side, the model training side, and the model reorganization verification side, and is automatically executed according to the set content to achieve cooperation and constraints among the three parties. The basic smart contract for model federal reorganization is as follows:

[0063] 1) Customer side:

[0064] a) Set the target task of model federal reorganization as vehicle detection and classification and the final model receiving standard;

[0065] b) Provide the target task detection data D T for the model reorganization verification side (i.e., a subset of the sampled vehicle detection and classification image dataset collected by the customer side), and set the task reward for the model reorganization verification side;

[0066] c) Specify the source model required by the model training side, and set the minimum transferability score p thres, set the training reward and follow the contribution calculation method in step 6 to assign the corresponding reward to the model training party.

[0067] 2) Model training party:

[0068] a) After the first migratability evaluation of step 5.1 by the model reorganization verification party, the model training party node that fails to be selected can choose to withdraw from the task in advance, or continue to participate in the task and finally obtain the corresponding reward through the contribution calculation method in step 6;

[0069] b) Provide the source model of the target task to the model reorganization verification party;

[0070] 3) Model reorganization verification party:

[0071] a) Based on the probe data of the target task, the source model is federated and reorganized to obtain the target model, and the target model that meets the acceptance standard is submitted to the customer party;

[0072] b) Calculate the contribution value of each model training party through the contribution calculation method in step 6.

[0073] 3. Store the data in the model federation reorganization process;

[0074] The sharing of probe data and target models between the customer party and the model reorganization verification party, and the sharing of source models between the model training party and the model reorganization verification party is realized through a distributed storage (DS) system to ensure the security of data and private machine learning model sharing. The DS system makes access to these data and learning models more affordable, faster, safer, and better meets the communication needs in the federal learning process. After the node stores data or private models in the DS system, it will generate a unique hash value address representing these network assets and share it to ensure the integrity and authorization of the sharing process. Specifically, we will use the InterPlanetary File System (IPFS) for distributed storage in the following process.

[0075] 4. Train the source model of the model training party;

[0076] Model federation reorganization is divided into two parts, one part is source model training task, and the other part is to reorganize the source model and verify to obtain the target model. The two parts are completed by the model training party and the model reorganization verification party respectively, and the model training party provides the source model required by the model reorganization verification party. The source model training is completed by two steps of local pre-training and knowledge distillation Figure 2 ).

[0077] The model training party trains a local pre-training model based on local data, and uses M = {M1, M2, …, M N} represents the local pre-training model of N model training party nodes. In order to protect the privacy of customer data, the model training party cannot obtain the vehicle detection and classification image data of the customer party, and can only train the local pre-training model based on the local data set with high correlation with vehicle detection and classification. At the same time, in order to ensure the data privacy of the model training party, the local pre-training model is distilled to obtain the source model, which is provided to the model reorganization verification party for federated reorganization and verification process.

[0078] The source model learns knowledge from the local pre-training model based on the feature-based knowledge distillation method. In the knowledge distillation process, due to the heterogeneity of the network structure, the translation module is used to align the feature output dimensions of the local pre-training model and the source model. The output of the source model is converted to a predefined output length through the translation module, for example, here our translation module uses 3 convolutions with 1x1 kernels. Let and respectively represent the source model and the local pre-training model M n of the n th node after length alignment. respectively represent the source model and the local pre-training model parameters of the n th node when communicating with the model reorganization verification party for the t th time. X is the training data of the model M n At the same time, the intermediate output of the source model is set to imitate the pre-training model, and the feature knowledge distillation loss is calculated as follows:

[0079]

[0080] The source model parameter update adopts gradient descent as follows:

[0081]

[0082] After local knowledge extraction, the source model parameter is uploaded to the interstellar file system as the source model of the model reorganization verification party, and the model reorganization verification party obtains the model parameters through the unique hash address provided by the model training party.

[0083] 5. Model federated reorganization;

[0084] The source model obtained in step 4 is subjected to model federated reorganization by the model reorganization verification party. The source model provided by the model training party is trained based on different local data, and it is necessary to select the source model more beneficial to the target model for federated reorganization, so the model federated reorganization has the following two steps: (1) source model selection; (2) federated reorganization and performance verification Figure 3 ).

[0085] 5.1. Select a favorable model from the source model set for the federated reorganization in step 5.2. To measure whether a source model is beneficial to the target model, a saliency map is used to measure the transferability of knowledge contained in the source model, thereby measuring its beneficialness. To ensure client data privacy, the model reorganization validation party only has the probe data D of the target task. T (This refers to a subset of the image dataset collected by the client for vehicle detection and classification.) Based on the client's model requirements, the model refactoring and validation team first trains the original target model and calculates the average saliency map of the target model against the probe data. Then, based on the source models, it calculates the average saliency map of each source model against the probe data. The two are then calculated in pairs to obtain the transferability score of the source models. The saliency map calculation formula is as follows:

[0086]

[0087] Among them Ψ k This represents the latent features of the k-th layer of the source model. This represents the k-th layer of the source model for the n-th node, corresponding to the probe data D. T In an input image x j The saliency map is calculated here through one forward and backward propagation.

[0088] The source model of the nth node at layer k has all probe data D. T The saliency plot is represented as Through all The average value is Where S is D T Total number of images. Transferability γ between the two models. n,T Normalized selection probability p of the source model n The following can be estimated, where This is the saliency map of the k-th layer of the target model.

[0089]

[0090]

[0091] To fully utilize the latent features extracted by the model, we use the latent features from the last layer before the model's output layer to calculate the saliency map and selection probability. Based on the selection probability of the source model, we select p... n Not lower than the minimum transferability score p set by the client thres The first K source models are reorganized and updated in step 5.2.

[0092] 5.2. Federated recombining the selected source models to verify the performance of the target model after recombination. Based on the original target model parameters obtained in the first time of step 5.1, federated recombine the source models in a greedy recombination manner, so that after each federated recombination of the target model, update the target model parameters, and the performance of the target model is improved. In the greedy recombination process, first, the source model to be added is tested and verified with the probe data provided by the customer, and then the source model parameters are sorted in descending order according to the performance of the verification test and averaged with the target model parameters. The fused target model is again tested and verified based on the probe data, and if the performance is improved, the updated target model parameters are retained, and the acc i is recorded. For the i-th source model training node, the value of the target model performance improvement this time is trained, and the above operation is repeated for the descendingly arranged source models to be added until the end, which is regarded as the completion of one iteration of federated recombination. When the performance of the target model of vehicle detection and classification reaches the acceptance standard set by the customer, the model recombination verification party uploads the target model to the interstellar file system and provides the customer with the corresponding hash address.

[0093] 6. Calculate the contribution value of the model training party;

[0094] According to the transferability score of the source model provided by each model training party in step 5 and the target model performance index improvement in the federated recombination process, the contribution value of each model training party to the target model is obtained. R t，i is used to represent the contribution value of the i-th model training node of the model training party and the model recombination verification party in the t-th federated communication iteration, and the contribution calculation is as follows:

[0095] R t，i = α * acc t，i + β * p t，i (6)

[0096] where acc t，i is the number of performance improvements of the selected source model to the target model in step 5.2, p t，i is the transferability score of the selected source model in step 5.1, and if not selected, p t，i = 0, and α, β are variable parameters for adjusting the weight proportion of acc t，i and p t，i and the numerical magnitude of the two parameters, for example, here, acc t，i and p t，i are both scores of 0-1, and we set α = 0.8 and β = 0.2, indicating that more attention is paid to the performance improvement value of the source model to the target model in each model federated recombination iteration process.

[0097] Considering that the growth of the target model performance is approximately logarithmic non-linear growth, the average contribution value per iteration cannot be directly accumulated, therefore, the exponential moving average method with bias correction is adopted, with the increase of the number of iterations, the weight of the contribution value of recent iterations is increased, and the weight of the contribution value of previous iterations is reduced, and finally the cumulative contribution value of the ith model training party after t rounds of communication iterations is The calculation is as follows:

[0098]

[0099] When t = 0 The contribution value R of each time is set as default t,i All rounds of contribution values are used for calculation, and the formula of the moving window size and the weight parameter is The corresponding mu value can be calculated according to the final iteration round. At the same time, through the following bias correction, the final contribution value E of the final ith model training party can be obtained i And the final contribution value E i The model training party can obtain the corresponding task reward. The reward reward:

[0100]

[0101]

[0102] Where B is the total reward value of the model training task given by the customer party, B i is the corresponding reward value that the ith model training party can obtain. According to the result, the smart contract in the alliance chain is automatically executed, and the model training party node participating in the vehicle detection and classification business published by the customer party automatically obtains the corresponding reward value, and the model reorganization verification party obtains the specified reward value in the contract, and the customer party obtains the final target model of vehicle detection and classification.

[0103] The implementation of the deep neural network model federal reorganization method and system based on the blockchain of the present application includes the alliance chain module of model federal reorganization, the smart contract module of model federal reorganization, the distributed storage data sharing module, the source model training module, the model federal reorganization module, and the model reorganization contribution distribution module.

[0104] The specific content of each module corresponds to steps 1-6 of the inventive method of the present embodiment, respectively.

[0105] Considering the reusability of deep neural network models, the limited computing power and data of single-node user models, and the limitedness of the reorganization of the models, multi-node model federal reorganization cooperation and corresponding contribution allocation are performed. Meanwhile, considering the data security and privacy in the federal process, and the mutual distrust of cooperation between multiple nodes in the federal process, based on the Hyperledger Fabric consortium chain and smart contract, a deep neural network model federal reorganization method and system based on a blockchain are proposed. First, based on the Hyperledger Fabric framework, a consortium chain that meets the model federal reorganization cooperation task is constructed, and a corresponding smart contract is formulated to determine the business logic, and a secure and reliable cooperation method of the customer party, the model training party and the model reorganization verification party is established. At the same time, the InterPlanetary File System, a distributed storage system, is used to ensure the security of data sharing in the model federal reorganization process, and to meet the communication transmission requirements in the model federal reorganization process. Then, the model training party trains a local pre-training model based on local data and knowledge distillation, and the obtained model is used as the source model in the model federal reorganization process. The model reorganization verification party selects a source model with higher transferability to the target task based on a saliency map, and the selected source model is gradually fused into the target model through greedy reorganization, and multiple communication iterations are performed until the target model meets the customer's acceptance standard. Finally, according to the contribution calculation method, the transferability score of the source model provided by the model training party and the performance index improvement of the target model in the reorganization process are normalized to obtain the contribution value of each iteration, and all iteration contribution values are calculated by the exponential moving average with bias correction, thereby obtaining the final contribution value of each model training party, and the task reward is allocated according to the weight. The present application combines the blockchain technology with the model reorganization method in the field of artificial intelligence, and makes a new attempt in the field of safe and reliable cooperation of model federal reorganization, and good results are obtained, which has high application value and development prospect.

[0106] The embodiments described in the specification are only enumerations of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as limited to the specific forms described in the embodiments, and the protection scope of the present application also includes equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A blockchain-based deep neural network model federated reorganization method, comprising the following steps: Step 1. Construct a consortium blockchain that conforms to the model of federated reorganization cooperation based on the Hyperledger Fabric framework; A consortium blockchain consists of four roles: consortium members, consortium operators responsible for maintaining and updating the blockchain, smart contract auditors, and blockchain developers. Nodes participating in the model federation reorganization collaboration are consortium members. The Hyperledger Fabric framework is a basic framework for a consortium blockchain, providing access control, ledger maintenance, and smart contract blockchain technology. Users only need to set the corresponding business logic. Based on this framework, the business between consortium members is set up as a model federation reorganization collaboration. The business is published by the client and received by the model trainer and the model reorganization verification provider. The three parties cooperate through smart contracts, thereby constructing a model federation reorganization consortium blockchain. Step 2. Set up the smart contract for the model's federated restructuring; The smart contract is signed between the client, the model trainer, and the model reorganization and verification party, and is automatically executed according to the settings, achieving tripartite cooperation and constraints, and ensuring the security of the model federation reorganization process; Step 3. Store data during the federal reorganization process of the storage model; The sharing of probe data and target models between the client and the model reorganization and validation parties, as well as the sharing of source models between the model trainer and the model reorganization and validation parties, are achieved through a distributed storage system to ensure the security of data and private machine learning model sharing. The distributed storage system makes access to this data and learning models more affordable, faster, and more secure, and better meets the communication needs in the federated learning process. After a node stores data or a private model in the distributed storage system, it generates a unique hash value address representing these network assets and shares it, ensuring the integrity and authorization of the sharing process. Step 4. The source model of the training party is handed over to the model reorganization and validation party for federated reorganization; The model federated reassembly is divided into two parts: one is the source model training task, and the other is the federated reassembly of the source model and its verification to obtain the target model. The two parts are completed by the model trainer and the model reassembly verification team, respectively. The model trainer provides the source model required by the model reassembly verification team. The source model training is completed by two steps: local pre-training and knowledge distillation. The model trainer uses local data to train a local pre-trained model, which is then used... This represents a local pre-trained model with N model training nodes; To ensure privacy, knowledge distillation is performed on the local pre-trained model to obtain the source model, which is then provided to the model reorganization and verification party for federated reorganization and verification processes. The source model learns knowledge from a locally pre-trained model using a feature-based knowledge distillation method; During knowledge distillation, due to the heterogeneity of network structures, a translation module is used to align the feature output dimensions of the local pre-trained model and the source model, converting the source model output into a predefined output length via the translation module; let... and These represent the source model and the locally pre-trained model of the nth node, respectively. The length-aligned features; They represent the first During the second communication with the model reorganization validation party, the first The parameters of the source model and the locally pre-trained model for each node; For the model The training data; simultaneously, to make the intermediate output of the source model mimic the pre-trained model, the feature knowledge distillation loss is calculated as follows: The source model parameters are updated using gradient descent as follows: After local knowledge extraction, the source model parameters The model is uploaded to the InterPlanetary File System as the source model for model reorganization and verification. The model reorganization and verification party obtains the model parameters through the unique hash address provided by the model trainer. Step 5. Model Federated Reorganization: The model reorganization validation party performs federated reorganization on the source model to obtain the target model; The source model obtained from step 4 is handed over to the model reorganization verification party for model federation reorganization; the source model provided by the model trainer is trained based on different local data, and it is necessary to select the model that is more favorable to the target model for federation reorganization. Therefore, the model federation reorganization has the following two steps: (1) Select the favorable model in the source model set for federation reorganization; (2) Validate the performance of the target model obtained after reorganization by verifying the source model selected for federation reorganization. Step 6: Calculate the contribution value of the model trainer; Based on the transferability scores of the source models provided by each model trainer in step 5 and the improvement in the target model's performance metrics during the federated reorganization process, normalized calculations are performed to obtain a contribution metric for each model trainer to the target model; using This represents the contribution value of the i-th model training node in the t-th federated communication iteration between the model trainer and the model reorganization validation parties. The contribution is calculated as follows: in The selected source model improves the target model's performance by a certain number in step 5.

2. The transferability score is the score of the source model selected in step 5.1; if not selected, then... , These are variable parameters used for adjustment The weighting ratio of the two parameters and the uniformity of their numerical magnitudes; Considering that the performance growth of the target model is approximately logarithmic and nonlinear, the contribution value cannot be directly obtained by averaging each iteration. Therefore, an exponential moving average method with bias correction is adopted. As the number of iterations increases, the weight of the contribution value of recent iterations is increased, while the weight of the contribution value of earlier iterations is decreased. Finally, the cumulative contribution value of the i-th model trainer after t rounds of communication iterations is obtained. The calculation is as follows: in hour By default, the contribution value will be calculated each time. All are adopted, so according to the formula It can be calculated value; By correcting the bias as follows, the final contribution value of the i-th model trainer can be obtained. And based on the final contribution value The calculation yields the corresponding task reward for the model trainer; where... The total reward for the model training task provided by the client. The corresponding reward that can be obtained for training the i-th model:

2. The method as described in claim 1, wherein the smart contract for setting up the model federation reorganization in step 2 includes: The smart contract described in step 2 is signed between the client, the model trainer, and the model reorganization and verification party, and is automatically executed according to the settings, achieving tripartite cooperation and constraints; the basic smart contract for model federation reorganization is as follows: 1) Client: a) Establish the objectives and final acceptance criteria for the federal reorganization model; b) Provide the model reorganization validation party with the target task probe data, which is a subset of the target task dataset after sampling, and set the task reward for the model reorganization validation party. c) Specify the source model that the model trainer needs to provide, set the minimum transferability score for the model trainer to participate in the reorganization process in step 5.1, set training rewards and allocate corresponding rewards to the model trainer in accordance with the contribution calculation method in step 6. 2) Model training method: a) After the first transferability assessment in step 5.1 is conducted in the model reorganization verification team, the model training nodes that are not selected can choose to withdraw from the task early or continue to participate in the task and finally obtain the corresponding reward through the contribution calculation method in step 6. b) Provide the source model for the target task to the model reorganization and validation party; 3) Model Reorganization Validation: a) Based on the probe data of the target mission, the federal reorganization source model is used to obtain the target model, and the target model that has been verified to meet the acceptance criteria is submitted to the client. b) Calculate the contribution value for each model trainer using the contribution calculation method in step 6.

3. The method as described in claim 1, wherein the model federation reorganization in step 5 comprises the following two steps: (1) selecting a favorable model from the source model set for the federation reorganization in step 5.2; (2) verifying the performance of the target model obtained after reorganization using the source model selected for the federation reorganization. Step 5.1 Select the advantageous models from the source model set for the federated reorganization in Step 5.

2. To measure whether the source models are beneficial to the target model, a saliency map is used to measure the transferability of the knowledge contained in the source models, thereby measuring the advantage. Specifically, based on the probe data and model requirements provided by the client, the model reorganization validation team first trains the original target model and calculates the average saliency map of the target model against the probe data. Then, based on the source models, it calculates the average saliency map of each source model against the probe data. The two are calculated in pairs to obtain the transferability score of the source models. The saliency map calculation formula is as follows: in This represents the latent features of the k-th layer of the source model. This represents the k-th layer of the source model for the n-th node, representing the probe data. One input image A saliency map; The source model of the nth node at layer k for all probe data The saliency plot is represented as Through all The average value is , where S is Total number of images; transferability between the two models Normalized selection probability of the source model The estimates are as follows, where This is the saliency map of the k-th layer of the target model. : Select the probability based on the source model. The top K source models with a transferability score no lower than the minimum set by the client are reorganized and updated in step 5.2; Step 5.2: Federated Reassembly of the Selected Source Models and Verification of the Performance of the Target Model After Reassembly. Based on the original target model parameters obtained in Step 5.1, the source models are federated using a greedy reassembly method. This ensures that the target model parameters are updated after each federated reassembly, resulting in improved target model performance. The greedy reassembly process involves first verifying the source models to be added using probe data provided by the client. Then, the source model parameters are averaged with the target model parameters in descending order of verification test performance. The merged target model is then subjected to another verification test based on probe data. If performance is improved, the updated target model parameters are retained and recorded. For the i-th source model training node, the value that improves the performance of the target model is repeated for the source models to be added in descending order until the end. This is considered as the completion of one iteration of federated reorganization. When the performance of the target model reaches the acceptance standard set by the client, the model reorganization verification party uploads the target model to the InterPlanetary File System and provides the client with the corresponding hash address.

4. The blockchain-based deep neural network model federated reorganization method as described in claim 1, characterized in that, Step 3 uses the InterPlanetary File System for distributed storage in subsequent processes.

5. A system for implementing the blockchain-based deep neural network model federated reorganization method as described in claim 1, characterized in that, include: The consortium blockchain module for model federation reorganization, the smart contract module for model federation reorganization, the distributed storage data sharing module, the source model training module, the model federation reorganization module, and the model reorganization contribution allocation module correspond to steps 1 to 6 of claim 1, respectively.

Images