A method for measuring the contribution of nodes in a collaborative learning process

Abstract

The application discloses a node contribution measurement method in a cooperative learning process. The method filters out nodes of original data with high quality and unique characteristics by comparing the similarity of models, and adopts a kernel density estimation method to calculate rewards, so as to encourage high-quality nodes to actively participate in the training process, and finally quickly calculate a high-precision global model. In addition, the method has low calculation complexity, can quickly calculate the contribution of the nodes participating in the calculation, improves the practicability of the application, and has wide application prospect and economic value.

Description

Technical Field

[0001] This invention relates to the technical field of collaborative learning, and specifically to a method for measuring node contribution in a collaborative learning process. Background Technology

[0002] Collaborative learning is a distributed learning method that allows multiple participants (such as devices or organizations) to jointly train a model while preserving their respective data privacy. Due to its advantages in data privacy protection, resource utilization optimization, and data security, it is widely used in various fields such as healthcare, financial services, smart devices and the Internet of Things, retail, and e-commerce.

[0003] In collaborative learning processes, such as federated learning, the incentive problem for participating nodes is undeniable. Given that model training inevitably consumes node resources, nodes are reluctant to actively participate in training without sufficient incentives. Incentive mechanisms aim to motivate nodes, especially those with ample computing resources and high-quality data, to actively participate in the training process, contributing to the final global model, thereby calculating an accurate global model and reducing communication overhead and convergence time.

[0004] An efficient incentive mechanism can fully utilize the high-quality data from nodes to quickly and accurately learn the global model for subsequent real-world applications. Taking intelligent transportation as an example, real-time, high-precision traffic information can be used to plan traffic flow distribution, reduce urban congestion, and thus improve the convenience of citizens' travel. In light of this, some classic incentive mechanisms, including Shapley's value, Stackelberg game, reputation theory, and auction mechanisms, are applied to increase the enthusiasm of nodes to actively participate.

[0005] However, most incentive mechanisms fail to adequately address the issues of high network latency and accurate measurement of node contributions. Furthermore, these methods are typically computationally complex, inevitably leading to high latency. To address the node incentive problem in collaborative learning, this invention designs a lightweight method that effectively measures node contributions. By comparing the local models of nodes and considering the quality and uniqueness of local data, it effectively measures node contributions, increases node participation in the global model computation training process, and ensures the timeliness and accuracy of the final calculated global model. Summary of the Invention

[0006] To address the shortcomings of reputation mechanisms, this invention improves upon them in the following ways:

[0007] Local model optimization. Each node participating in the training process independently computes a clustering model that effectively describes the characteristics of its local data, based on the characteristics of its local data.

[0008] Model similarity calculation. Directly comparing clustering models and calculating their similarity can reduce computational costs and speed up node convergence time.

[0009] Node contribution calculation. Based on the similarity between models, a kernel function estimation algorithm is used, taking into account the accuracy and uniqueness of the models, to calculate the contribution of each node.

[0010] The method of this invention is achieved through the following technical solution:

[0011] A method for measuring node contribution in a collaborative learning process includes:

[0012] S1: Optimize the local model;

[0013] S2: Calculate the similarity between different models;

[0014] S3: The contribution of the computing node.

[0015] Preferably, in step S1, the nodes participating in the training use a clustering algorithm based on the characteristics of their local data to calculate a clustering model that can describe their features.

[0016] Preferably, the calculation steps are as follows:

[0017] 1) The nodes participating in the training reduce the dimensionality of the original dataset into a low-dimensional matrix that is friendly to the input of the clustering algorithm;

[0018] 2) Perform clustering on the reduced-dimensional matrix;

[0019] 3) Optimize the local model through information sharing and updates.

[0020] Preferably, in S2, the similarity between different models is calculated using Earth Mover's Distnace.

[0021] More preferably, the calculation of the similarity between different models specifically includes the following steps:

[0022] 1) The nodes fit a histogram to each representative feature point and its corresponding number of members;

[0023] 2) Similarly, fit the local models of the other nodes received into a histogram;

[0024] 3) Optimize the corresponding ground distance and calculate the Earth Mover's Distance between the node and its neighboring node.

[0025] Preferably, in step S3, the contribution of a node is calculated by kernel density estimation.

[0026] Furthermore, the choice of bandwidth has a significant impact on the contribution of the computing nodes. The optimal bandwidth is found by using a bandwidth evaluation method based on cross-validation. Specifically, this involves searching within a given search space with a specific step size and comparing the mean squared errors between the found clusters. The bandwidth size corresponding to the cluster partition with the smallest mean squared error is the optimal bandwidth.

[0027] More preferably, the contribution of the computing node specifically includes the following steps:

[0028] 1) Optimize bandwidth and divide the model similarity of each node into different clusters based on the lowest point of the density curve;

[0029] 2) Calculate the contribution of each cluster based on its number;

[0030] 3) Further calculate the contribution of each node based on the number of nodes in the cluster model.

[0031] The advantages of collaborative learning, exemplified by federated learning, in areas such as data privacy protection, resource utilization optimization, and data security are predicated on nodes' willingness to participate in the training process. In collaborative learning, particularly federated learning, the incentive problem for participating nodes is unavoidable. Given that model training inevitably consumes node resources, nodes are unwilling to actively participate in training without sufficient incentives. Furthermore, most current incentive mechanisms suffer from high computational complexity and inaccurate contribution calculations, hindering the further development of distributed learning.

[0032] This invention presents an effective method for measuring node contribution. The principle is to filter nodes with high-quality and unique original data by comparing model similarity, and then use kernel density estimation to calculate rewards. This encourages high-quality nodes to actively participate in the training process, ultimately leading to a high-precision global model. Furthermore, this method has low computational complexity and can quickly calculate the contribution of participating nodes, improving its practicality and demonstrating broad application prospects and economic value. Attached Figure Description

[0033] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the implementation of a node contribution measurement method in a collaborative learning process according to the present invention.

[0035] Figure 2 This is a diagram illustrating the calculation process of the clustering model of this invention;

[0036] Figure 3 This is a flowchart of the similarity calculation based on EMD in this invention. Detailed Implementation

[0037] The following describes in further detail a method for measuring node contribution in a collaborative learning process with reference to specific embodiments. These embodiments are for comparison and explanation purposes only, and the present invention is not limited to these embodiments.

[0038] like Figure 1 As shown, a method for measuring node contribution in a collaborative learning process includes:

[0039] S1: Optimize the local model;

[0040] S2: Calculate the similarity between different models;

[0041] S3: The contribution of the computing node.

[0042] Preferably, in step S1, the nodes participating in the training use a clustering algorithm based on the characteristics of their local data to calculate a clustering model that can describe their features.

[0043] For high-dimensional datasets commonly found in networks (such as video and image data), this invention employs an autoencoder to reduce the dimensionality of the data, and then uses a clustering algorithm to derive a local model. Accurate dimensionality reduction of high-dimensional data not only reduces the computational cost of nodes but also ensures the correctness of subsequent model calculations. Based on the dimensionality-reduced data, a clustering algorithm is proposed to calculate the local model, and then the model's accuracy is improved through continuous model updates.

[0044] Preferred, such as Figure 2 As shown, the calculation steps are as follows:

[0045] 1) The nodes participating in the training reduce the dimensionality of the original dataset into a low-dimensional matrix that is friendly to the input of the clustering algorithm;

[0046] 2) Perform clustering on the reduced-dimensional matrix;

[0047] 3) Optimize the local model through information sharing and updates.

[0048] Preferably, in S2, the similarity between different models is calculated using Earth Mover's Distnace.

[0049] Because measuring similarity is an effective method for identifying heterogeneous data sources, this invention uses Earth Mover's Distance (EMD) to measure the similarity between different models. EMD has advantages such as robustness to noise, applicability to various network topologies, and shape awareness, and can be widely applied in IoT scenarios with heterogeneous data sources. However, due to the high computational complexity of EMD (O(n)... 3 Log n) is rarely used in research fields outside of computer vision. To reduce the computational cost of calculating EMD, we propose using cluster centers as features to calculate EMD. The specific steps for similarity calculation based on EMD are as follows: Figure 3 As shown.

[0050] Kernel density estimation (KDE) is a nonparametric statistical method used to estimate the probability density function of a continuous random variable. It can be used for data visualization, particularly to show the shape of univariate or multivariate distributions. KDE smoothly estimates the entire distribution based on observed data points. The basic idea is to place a kernel (e.g., a Gaussian kernel) around each observation point and then average these kernels to obtain a smooth estimate of the entire dataset. Kernel density estimation methods mainly include kernel function, bandwidth, and density estimation.

[0051] A kernel function K is a symmetric probability density function used to generate a smooth local shape around each data point. Commonly used kernel functions include the Gaussian kernel, the Epanechnikov kernel, and the trigonometric kernel.

[0052] Bandwidth h is a key parameter of KDE, determining the kernel width and thus affecting smoothness. Excessive bandwidth can lead to over-smoothing, potentially ignoring interesting structures in the data; insufficient bandwidth may result in excessive noise, failing to capture the overall trend of the data.

[0053] By summing the contributions of each kernel function around the data point and performing appropriate normalization, a density estimate across the entire data range can be obtained. Mathematically, this can be expressed as:

[0054]

[0055] Where n is the number of samples, x i Here are the sample data, and x is the point whose density is to be estimated.

[0056] Earth Mover's Distance (EMD), also known as Wasserstein distance, is a metric that measures the difference between two probability distributions. The core concept of EMD is the "minimum amount of work" required to transform one distribution into another. Imagine two piles of soil, each representing a distribution, and the goal is to make one pile similar to the other by moving the minimum amount of soil. Given two probability distributions P and Q, and a cost matrix C, where C... ij Let f represent the cost from the i-th part of distribution P to the j-th part of distribution Q. EMD aims to find a flow matrix f. ij This minimizes the total cost:

[0057]

[0058] More preferably, the calculation of the similarity between different models specifically includes the following steps:

[0059] 1) The nodes fit a histogram to each representative feature point and its corresponding number of members;

[0060] 2) Similarly, fit the local models of the other nodes received into a histogram;

[0061] 3) Optimize the corresponding ground distance and calculate the Earth Mover's Distance between the node and its neighboring node.

[0062] Preferably, in step S3, the contribution of a node is calculated by kernel density estimation.

[0063] Furthermore, the choice of bandwidth has a significant impact on the contribution of the computing nodes, and it has a crucial influence on the accuracy of the contribution calculation. If the bandwidth is too large, duplicate node information will be ignored, while if the bandwidth is too small, overfitting will occur. This invention finds the optimal bandwidth through a bandwidth evaluation method based on cross-validation. Specifically, it involves searching within a given search space with a specific step size and comparing the mean squared errors of the found clusters. The bandwidth corresponding to the cluster partition with the smallest mean squared error is the optimal bandwidth.

[0064] More preferably, the contribution of the computing node specifically includes the following steps:

[0065] 1) Optimize bandwidth and divide the model similarity of each node into different clusters based on the lowest point of the density curve;

[0066] 2) Calculate the contribution of each cluster based on its number;

[0067] 3) Further calculate the contribution of each node based on the number of nodes in the cluster model.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for measuring node contribution in a collaborative learning process, characterized in that, include: S1: Optimize the local model; S2: Calculate the similarity between different models; S3: Calculate the contribution of the node; In step S1, the nodes participating in the training use a clustering algorithm based on the characteristics of their local data to calculate a clustering model that can describe their features. The calculation steps are as follows: 1) The nodes participating in the training reduce the dimensionality of the original dataset into a low-dimensional matrix that is friendly to the input of the clustering algorithm; the original dataset includes image data; 2) Perform clustering on the reduced-dimensional matrix; 3) Optimize the local model through information sharing and updates; In S2, the similarity between different models is calculated using Earth Mover's Distnace; The calculation of similarity between different models specifically includes the following steps: 1) The nodes fit a histogram to each representative feature point and its corresponding number of members; 2) Similarly, fit the local models of the other nodes received into a histogram; 3) Optimize the corresponding ground distance and calculate the Earth Mover's Distance between the node and other nodes; In S3, the contribution of a node is calculated by kernel density estimation; The choice of bandwidth has a significant impact on the contribution of the computing nodes. The optimal bandwidth is found by a bandwidth evaluation method based on cross-validation. Specifically, it involves searching within a given search space with a specific step size and comparing the mean squared errors between the found clusters. The bandwidth size corresponding to the cluster partition with the smallest mean squared error is the optimal bandwidth. The contribution of the computing node specifically includes the following steps: 1) Optimize bandwidth and divide the model similarity of each node into different clusters based on the lowest point of the density curve; 2) Calculate the contribution of each cluster based on its number; 3) Further calculate the contribution of each node based on the number of nodes in the cluster model.

Images