Contribution degree assessment method and apparatus, and communication device and storage medium

Abstract

The present application relates to the technical field of federated learning, and relates to a contribution degree assessment method and apparatus, and a communication device and a storage medium. The method comprises: a server acquiring a feature probability distribution of a reference data set corresponding to a target task, and determining assessment indicators corresponding to the target task and calculation methods for the assessment indicators (S201); then, sending, to each participating end in federated learning, the feature probability distribution of the reference data set, and the assessment indicators and the calculation methods therefor, such that each participating end determines data quality parameters of a local data set under the assessment indicators (S202); furthermore, receiving the data quality parameters sent by each participating end, and assessing the contribution degree of each participating end (S203), wherein the data quality parameters comprise task-independent quality parameters and task-related quality parameters.

Description

Contribution evaluation method, device, communication device and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023112653440, filed on September 27, 2023, entitled “Contribution Assessment Method, Apparatus, Communication Equipment and Storage Medium,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of federated learning technology, and in particular to a contribution evaluation method, apparatus, communication device, and storage medium. Background Art

[0004] Federated learning is a new distributed machine learning framework, typically consisting of a server and participating clients. Multiple participating clients undergo multiple rounds of training to produce a global model. Evaluating the data contribution of participating clients in federated learning is a core issue. Fair, reliable, and efficient methods for evaluating data contribution can rationally assess the contributions of participating clients and thus incentivize them to participate.

[0005] In federated learning, the accuracy of the global model is closely related to the data quality of the training sample data provided by each participating client. Generally speaking, higher quality training sample data leads to higher global model accuracy. Therefore, the server evaluates the contribution of each participating client to federated learning based on the data quality of their training sample data. However, high-quality data that performs well in one practical task scenario may be poor quality data in another. Current methods for assessing contribution based on data quality only consider the data quality of each participating client's training sample data, making it difficult to comprehensively measure data quality. This, in turn, affects the fairness and rationality of assessing the contribution of participating clients.

[0006] Summary of the Invention

[0007] In a first aspect, the present application provides a contribution evaluation method, which is applied to a server and includes:

[0008] Obtain the characteristic probability distribution of the benchmark data set corresponding to the target task, and determine the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator;

[0009] Send the feature probability distribution of the benchmark dataset, each evaluation metric, and the calculation method of each evaluation metric to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local dataset under each evaluation metric; the data quality parameters include task-independent quality parameters and task-dependent quality parameters;

[0010] Receive data quality parameters sent by each participating terminal, and evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0011] In one embodiment, obtaining a feature probability distribution of a benchmark data set corresponding to a target task includes:

[0012] Build a benchmark dataset corresponding to the target task;

[0013] Extract data features of benchmark datasets;

[0014] According to the data characteristics of the benchmark data set, the characteristic probability distribution of the benchmark data set is determined.

[0015] In one embodiment, the characteristic probability distribution of the benchmark dataset is used by each participating terminal to determine task-related quality parameters of the local dataset under task-related indicators.

[0016] In one embodiment, each evaluation indicator includes a task-independent indicator and a task-related indicator; the task-independent indicator includes data accuracy, data integrity, and data uniqueness, and the task-related indicator includes data relevance.

[0017] In one embodiment, determining the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator includes: determining a task-independent indicator and a task-related indicator corresponding to the target task, and determining a calculation method for the task-independent indicator and a calculation method for the task-related indicator;

[0018] In one embodiment, the calculation method for determining the task-related indicator includes: determining the degree of correlation between the local data set of each participating terminal and the benchmark data set.

[0019] In one embodiment, the contribution of each participant is evaluated based on the data quality parameters sent by each participant, including:

[0020] Construct a federated learning contribution matrix based on the data quality parameters sent by each participating end;

[0021] The contribution of each participant is determined based on the contribution matrix and the weight of each evaluation indicator.

[0022] In one embodiment, the elements in the same row of the contribution matrix are the data quality parameters of the same participating terminal under each evaluation indicator, and the elements in the same column are the data quality parameters of each participating terminal under the same evaluation indicator.

[0023] In one embodiment, the contribution of each participating terminal is determined based on the contribution matrix and the weight of each evaluation indicator, including:

[0024] According to the maximum and minimum elements of each column in the contribution matrix, a maximum value vector and a minimum value vector are constructed;

[0025] According to the difference between each row element in the contribution matrix and the corresponding element in the maximum value vector, as well as the weight of each evaluation index, the distance between each row vector in the contribution matrix and the maximum value vector is determined;

[0026] According to the difference between each row element in the contribution matrix and the corresponding element in the minimum value vector, as well as the weight of each evaluation index, the distance between each row vector in the contribution matrix and the minimum value vector is determined;

[0027] The contribution of each participating terminal is determined based on the distance between each row vector and the maximum value vector, as well as the distance between each row vector and the minimum value vector.

[0028] In a second aspect, the present application provides a contribution evaluation method, which is applied to a participant terminal and includes:

[0029] Receive the feature probability distribution of the benchmark data set corresponding to the target task sent by the server, as well as the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator;

[0030] Based on the characteristic probability distribution of the benchmark dataset and the calculation method of each evaluation indicator, the data quality parameters of the local dataset under each evaluation indicator are determined; the data quality parameters include task-independent quality parameters and task-dependent quality parameters;

[0031] The data quality parameters are sent to the server so that the server can evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0032] In one embodiment, each evaluation indicator includes a task-independent indicator and a task-related indicator; the task-independent indicator includes data accuracy, data integrity, and data uniqueness, and the task-related indicator includes data relevance.

[0033] In one embodiment, determining the data quality parameters of the local dataset under each evaluation indicator based on the feature probability distribution of the benchmark dataset and the calculation method of each evaluation indicator includes:

[0034] According to the calculation method of task-independent indicators, determine the task-independent quality parameters of the local dataset under the task-independent indicators;

[0035] According to the calculation method of the task-related indicators and the characteristic probability distribution of the benchmark data set, the task-related quality parameters of the local data set under the task-related indicators are determined.

[0036] In one embodiment, determining the task-related quality parameters of the local dataset under the task-related indicators based on the calculation method of the task-related indicators and the characteristic probability distribution of the benchmark dataset includes:

[0037] Obtain the feature probability distribution of the local data set;

[0038] determining a bulldozer distance between a feature probability distribution of a local dataset and a feature probability distribution of a benchmark dataset according to a calculation method for the task-related indicator;

[0039] Determine a task-related quality parameter of the local data set under the task-related indicator according to the bulldozer distance.

[0040] In one embodiment, determining the task-related quality parameter of the local dataset under the task-related indicator based on the bulldozer distance includes: determining the reciprocal of the bulldozer distance as the task-related quality parameter of the local dataset under the task-related indicator.

[0041] In one embodiment, obtaining a feature probability distribution of a local data set includes:

[0042] Build a local dataset corresponding to the target task;

[0043] Extract data features of local datasets;

[0044] According to the data characteristics of the local data set, the characteristic probability distribution of the local data set is determined.

[0045] In a third aspect, the present application further provides a contribution evaluation device, which is configured on a server and includes:

[0046] A determination module is used to obtain the characteristic probability distribution of the benchmark data set corresponding to the target task, and determine the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator;

[0047] A sending module is configured to send the feature probability distribution of the benchmark dataset, each evaluation metric, and the calculation method of each evaluation metric to each participating end of the federated learning system, so that each participating end can determine the data quality parameters of the local dataset under each evaluation metric; the data quality parameters include task-independent quality parameters and task-dependent quality parameters;

[0048] The evaluation module is used to receive the data quality parameters sent by each participating terminal and evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0049] In a fourth aspect, the present application further provides a contribution evaluation device, which is configured at a participating terminal and includes:

[0050] A receiving module is used to receive the feature probability distribution of the benchmark data set corresponding to the target task, as well as the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator from the server;

[0051] A determination module is used to determine the data quality parameters of the local dataset under various evaluation indicators based on the characteristic probability distribution of the benchmark dataset and the calculation method of each evaluation indicator; the data quality parameters include task-independent quality parameters and task-dependent quality parameters;

[0052] The sending module is used to send the data quality parameters to the server so that the server can evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0053] In a fifth aspect, the present application further provides a communication device, comprising: a transmitter, a receiver, a processor, and a memory, wherein the memory stores a computer program;

[0054] The computer program is executed by the processor to obtain a feature probability distribution of a benchmark data set corresponding to the target task, and to determine evaluation indicators corresponding to the target task and calculation methods for the evaluation indicators;

[0055] The transmitter is used to send the feature probability distribution of the benchmark dataset, each evaluation indicator, and the calculation method of each evaluation indicator to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local dataset under each evaluation indicator; the data quality parameters include task-independent quality parameters and task-dependent quality parameters;

[0056] The receiver is used to receive the data quality parameters sent by each participating terminal, and evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0057] In a sixth aspect, the present application further provides a communication device, comprising: a transmitter, a receiver, a processor, and a memory, wherein the memory stores a computer program;

[0058] The receiver is used to receive the feature probability distribution of the benchmark data set corresponding to the target task, as well as the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator from the server;

[0059] The computer program is executed by the processor to determine data quality parameters of the local data set under each evaluation indicator based on the feature probability distribution of the benchmark data set and the calculation method of each evaluation indicator; the data quality parameters include task-independent quality parameters and task-dependent quality parameters;

[0060] The sender is used to send the data quality parameters to the server, so that the server can evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0061] In a seventh aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of the first aspect or the second aspect mentioned above are implemented.

[0062] In an eighth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the method of the first aspect or the second aspect when executed by a processor.

[0063] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. The drawings described below are merely some embodiments of the present application and do not constitute a limitation on the disclosure content and protection scope of the present application.

[0065] FIG1 is an application scenario diagram of a contribution evaluation method according to an embodiment;

[0066] FIG2 is a schematic diagram of a process flow of a contribution evaluation method according to an embodiment;

[0067] FIG3 is a schematic diagram of a process for determining a feature probability distribution of a reference data set in one embodiment;

[0068] FIG4 is a schematic diagram of a process for determining the contribution of each participating terminal in one embodiment;

[0069] FIG5 is a schematic diagram of a process for determining the contribution of each participating terminal in another embodiment;

[0070] FIG6 is a schematic flow chart of a contribution evaluation method according to another embodiment;

[0071] FIG7 is a schematic diagram of a process for determining task-related quality parameters in one embodiment;

[0072] FIG8 is a schematic diagram of a process for determining task-related quality parameters in another embodiment;

[0073] FIG9 is a schematic diagram of a process for determining a feature probability distribution of a local data set in one embodiment;

[0074] FIG10 is a schematic diagram of a flow chart of a contribution evaluation method according to another embodiment;

[0075] FIG11 is a schematic flow chart of a contribution evaluation method according to another embodiment;

[0076] FIG12 is a schematic flow chart of a contribution evaluation method according to yet another embodiment;

[0077] FIG13 is a structural block diagram of a contribution evaluation device according to an embodiment;

[0078] FIG14 is a structural block diagram of a contribution evaluation device according to another embodiment;

[0079] FIG15 is a structural block diagram of a contribution evaluation device in another embodiment;

[0080] FIG16 is a structural block diagram of a contribution evaluation device in yet another embodiment;

[0081] FIG17 is a structural block diagram of a contribution evaluation device in yet another embodiment. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0083] The contribution evaluation method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the federated learning framework includes a server 101 and several participating terminals, namely participating terminals 102 to 10n. These participating terminals are essentially clients installed on UE (User Equipment). The server 101 and each participating terminal are connected through network communication. Specifically, the federated learning process includes: the server 101 initializes the global model parameters, each participating terminal uses the local data set to train the local model, and then sends the trained model parameters to the server 101. Further, the server 101 weights and aggregates the model parameters received from each participating terminal, updates the global model, and sends the updated global model parameters to each participating terminal, so that each participating terminal updates the local model and starts the next round of training until the training is stopped when the conditions are met, and finally obtains the global model of federated learning.

[0084] Participating terminals 102 to 10n may be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 101 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0085] The current method of using data quality to evaluate the contribution of each participant in federated learning only considers the data quality of the participant's data itself, while ignoring the participant's data with higher data quality, which may not be suitable for specific task scenarios, resulting in unfair and unreasonable contribution evaluation of each participant in federated learning.

[0086] To solve the above technical problems, in one embodiment, as shown in FIG2 , a contribution evaluation method applied to a server is provided, comprising the following steps:

[0087] S201, obtaining a feature probability distribution of a benchmark data set corresponding to a target task, and determining evaluation indicators corresponding to the target task and calculation methods of the evaluation indicators.

[0088] The federated learning framework includes a server and several participating terminals. The target task is the task of the global model built by the current federated learning framework, such as image classification and object detection. The benchmark dataset is a high-quality training dataset that can express the task requirements. Different tasks correspond to different benchmark datasets.

[0089] The server obtains a benchmark dataset corresponding to the target task, extracts the data features from the benchmark dataset, and then calculates the probability distribution of the data features to obtain the feature probability distribution of the benchmark dataset corresponding to the target task. For example, if the target task is a target detection task, the benchmark dataset corresponding to the target detection task may include several frames of images with the target to be detected, as well as target labels corresponding to the target to be detected, such as vehicles, pedestrians, or buildings. The features of these targets to be detected have their own probability distributions during the sampling process, which is the feature probability distribution of the benchmark dataset corresponding to the target detection task.

[0090] A target task corresponds to multiple evaluation indicators, where each evaluation indicator includes task-independent indicators and task-related indicators. Specifically, the evaluation indicators are used to assess the contribution of each participant in the current federated learning framework based on the training data of each participant. Task-independent indicators refer to indicators that evaluate the quality of the training data itself, reflecting the inherent properties of the training data. Optional task-independent indicators include data accuracy, data completeness, and data uniqueness. Task-related indicators refer to indicators that evaluate the quality of training data in the context of the target task, meeting the requirements related to training data and tasks. Optional task-related indicators include data relevance.

[0091] The server determines the task-independent and task-dependent indicators corresponding to the target task, and determines a calculation method for the task-independent and task-dependent indicators. This embodiment does not limit the calculation method for the task-independent indicators. In one embodiment, the calculation method for the task-dependent indicators may include determining the degree of correlation between the local training data of each participating terminal and the benchmark dataset.

[0092] S202: Send the feature probability distribution of the benchmark dataset, each evaluation indicator, and the calculation method of each evaluation indicator to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local dataset under each evaluation indicator.

[0093] The server sends the acquired feature probability distribution of the benchmark dataset, along with the determined evaluation metrics and their calculation methods, to each participating terminal synchronously or asynchronously. Optionally, each participating terminal proactively downloads the feature probability distribution of the benchmark dataset from the server.

[0094] The local data set of each participating end is the local training data of each participating end. The data quality parameters of the local data set are used to measure the quality of the local data set in a digital way. The data quality parameters include task-independent quality parameters and task-related quality parameters. It can be understood that task-independent quality parameters correspond to task-independent indicators, and task-related quality parameters correspond to task-related indicators.

[0095] The characteristic probability distribution of the benchmark dataset, the various evaluation indicators, and the calculation methods for each evaluation indicator are used by each participating end to determine the data quality parameters of the local dataset under each evaluation indicator. Specifically, each participating end can calculate the task-independent quality parameters of the local dataset under each task-independent indicator based on the task-independent indicator and the calculation method for the task-independent indicator, and calculate the task-related quality parameters of the local dataset under the task-related indicator based on the characteristic probability distribution of the benchmark dataset, the task-related indicator, and the calculation method for the task-related indicator.

[0096] For example, each participating end calculates the task-independent quality parameters of the local dataset under the indicator of data accuracy based on the missing values, error values, outliers or sampling deviation results of the local dataset; each participating end calculates the task-independent quality parameters of the local dataset under the indicator of data completeness based on the coverage of the local dataset; each participating end calculates the task-independent quality parameters of the local dataset under the indicator of data uniqueness based on the repetition rate of the local dataset; each participating end calculates the task-related quality parameters of the local dataset under the indicator of data relevance based on the similarity between the feature probability distribution of the local dataset and the feature probability distribution of the benchmark dataset.

[0097] S203: Receive data quality parameters sent by each participating terminal, and evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

[0098] The server receives the task-independent quality parameters and task-related quality parameters sent by each participating terminal. Further, the server comprehensively evaluates the contribution of each participating terminal based on the task-independent quality parameters and task-related quality parameters of each participating terminal.

[0099] Specifically, the server can construct a contribution matrix that includes both task-independent and task-dependent quality parameters for all participating clients. Based on this contribution matrix, the server can then uniformly evaluate the contribution of each participating client along the same dimension, ensuring objectivity and fairness in the evaluation of each client's contribution. For example, the server assigns different contribution weights to different indicators. For each indicator, the server uniformly measures the contribution of all participating clients based on the data quality parameters of their local datasets under that indicator. Furthermore, the server uses the corresponding contribution weight for each indicator to comprehensively measure the contribution of all participating clients.

[0100] Optionally, the server can pre-set a threshold corresponding to a task-related quality parameter. When the task-related quality parameter of the participating terminal exceeds the threshold, the local dataset of the participating terminal is judged to meet the specific requirements of the target task scenario, which can better reflect the actual effect of using it to train the federated learning model, and then the contribution of the participating terminal is evaluated in combination with the task-independent quality parameters of the participating terminal. On the contrary, when the task-related quality parameter of the participating terminal does not exceed the threshold, the local dataset of the participating terminal is judged to not meet the specific requirements of the target task scenario. Then, no matter what the task-independent quality parameter of the participating terminal is, the problem of low contribution of the participating terminal cannot be changed, and the contribution of the participating terminal is directly evaluated based on the task-related quality parameters of the participating terminal.

[0101] In the above scheme, the server obtains the characteristic probability distribution of the benchmark data set corresponding to the target task, and determines the evaluation indicators and calculation methods of the evaluation indicators corresponding to the target task, and then sends the characteristic probability distribution of the benchmark data set, the evaluation indicators and the calculation methods of the evaluation indicators to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local data set under each evaluation indicator; wherein each evaluation indicator includes a task-independent indicator and a task-related indicator, and the data quality parameters include a task-independent quality parameter and a task-related quality parameter; further, the data quality parameters sent by each participating end are received, and the contribution of each participating end is evaluated based on the data quality parameters sent by each participating end. This embodiment comprehensively evaluates the contribution of each participating terminal from both task-independent and task-related dimensions. It not only considers the data quality of the local dataset of each participating terminal itself, but also considers the data quality of the local dataset of each participating terminal in a specific task scenario. This can improve the rationality of the contribution evaluation of each participating terminal in federated learning. Moreover, based on the benchmark dataset corresponding to the specific task scenario, the data quality of the local dataset of each participating terminal under task-related indicators is evaluated, which can ensure the objectivity and fairness of the evaluation results. Furthermore, based on the data quality evaluation results, a more accurate evaluation of the contribution of each participating terminal in federated learning can be achieved.

[0102] In order to obtain a benchmark dataset that is more closely aligned with the specific requirements of the target task scenario and more accurately calculate the feature probability distribution of the benchmark dataset, based on the above embodiment, in one embodiment, as shown in FIG3 , the above S201 may include:

[0103] S301, construct a benchmark dataset corresponding to the target task.

[0104] Obtain training data corresponding to the target task from various public databases on the internet platform, and select higher-quality training data from these to construct a benchmark dataset corresponding to the target task. It is understandable that the quality of the benchmark dataset will affect the task-related quality parameters of the participating end-users' local datasets. Therefore, when constructing the benchmark dataset, the selected training data should be as comprehensive and diverse as possible.

[0105] S302: Extract data features of the benchmark data set.

[0106] Data features of the training data in the benchmark dataset are extracted using methods such as RNN (Recurrent Neural Network), CNN (Convolutional Neural Networks), or Transformer. For example, if the target task is a target detection task, the static features of the target to be detected in each frame of the benchmark dataset, as well as the dynamic features of the target to be detected in adjacent frames, are extracted. The static and dynamic features of the target to be detected can represent the target type to be detected, such as a vehicle, pedestrian, or building. The dynamic features of the target to be detected can also represent the motion trajectory of the target to be detected, such as the motion trajectory of a vehicle or pedestrian.

[0107] S303: Determine the feature probability distribution of the benchmark data set according to the data features of the benchmark data set.

[0108] The probability distribution of the data features of the training data in the benchmark dataset is determined using methods such as KDE (Kernel Density Estimation), latent variable models, or GAN (Generative Adversarial Network). For example, if the target task is object detection, the probability distribution of the static and dynamic features of the target in the target task scenario is calculated based on the static features of the target in each frame and the dynamic features of the target in adjacent frames. The probability distribution of the data features of all the training data in the benchmark dataset is further analyzed to obtain the feature probability distribution of the benchmark dataset.

[0109] Among them, the characteristic probability distribution of the benchmark data set is used for each participating end to determine the task-related quality parameters of the local data set under task-related indicators.

[0110] In this embodiment, the server constructs a benchmark data set corresponding to the target task, extracts data features of the benchmark data set, and determines the feature probability distribution of the benchmark data set based on the data features of the benchmark data set, thereby ensuring that a benchmark data set that is more closely aligned with the specific requirements of the target task scenario is constructed, and ensuring the accuracy of the calculated feature probability distribution of the benchmark data set, thereby improving the rationality of the contribution evaluation of each participant in federated learning.

[0111] In order to reasonably evaluate the contribution of each participating terminal, based on the above embodiment, in one embodiment, a contribution matrix of federated learning can be constructed by combining mathematical methods. The overall performance of the data quality parameters of each participating terminal in the federated learning framework is then analyzed in the contribution matrix to determine the contribution of each participating terminal. As shown in FIG4 , the above S203 may include:

[0112] S401: Construct a contribution matrix for federated learning based on the data quality parameters sent by each participating terminal.

[0113] Based on the task-independent quality parameters and task-related quality parameters of each participating end, the contribution matrix of federated learning is jointly constructed and standardized.

[0114] For example, represents the task-independent quality parameter of the local dataset of participant k under the indicator of data accuracy, represents the task-independent quality parameter of the local dataset of participant k under the indicator of data completeness, represents the task-independent quality parameter of the local dataset of participant k under the indicator of data uniqueness, Represents the task-related quality parameter of the local dataset of participant k under the data relevance indicator, where k = 1, ..., n, where n is the number of participants in the federated learning framework.

[0115] The elements in the same row of the contribution matrix are the data quality parameters of the same participant under each evaluation indicator, and the elements in the same column are the data quality parameters of each participant under the same evaluation indicator. The constructed contribution matrix of federated learning is expressed as:

[0116] The contribution matrix X is normalized to eliminate the impact of the dimensions of different indicators, and the updated contribution matrix is ​​expressed as:

[0117] Furthermore, by simplifying the symbols of the elements in the contribution matrix Z, the final contribution matrix is ​​expressed as:

[0118] in, Represents the element in the i-th row and j-th column of the contribution matrix.

[0119] S402: Determine the contribution of each participating terminal based on the contribution matrix and the weight of each evaluation indicator.

[0120] In the final contribution matrix, the elements in the same column can represent the quality of the local datasets of different participating terminals under the same indicator. For each indicator, the contribution of all participating terminals under this indicator can be calculated by horizontally comparing the quality of the local datasets of all participating terminals under this indicator.

[0121] Optionally, TOPSIS (Technique for Order Preference by Similarity to an Ideal Solution) can be used to determine the contribution of each participant. Different indicators have different weights in the contribution evaluation. By combining the weight of each indicator, the overall contribution of all participants under all indicators can be calculated. For example, w j represents the weight of index j, j = 1, ..., 4, w j All are greater than 0, and w1+w2+w3+w4=1. The weight of each indicator is determined by the specific target task requirements.

[0122] It should be noted that the number of columns and indicators in the above contribution matrix is ​​only for example and does not mean that the indicators in this embodiment are limited to 4. In the scenario where the server determines any number of evaluation indicators, the steps in this embodiment can be taken to calculate the contribution of each participating terminal, and no further details will be given.

[0123] In this embodiment, the server constructs a contribution matrix for federated learning based on the data quality parameters sent by each participating terminal, and determines the contribution of each participating terminal based on the contribution matrix and the weights of each evaluation indicator. Different weights are set for different evaluation indicators, and the quality of the local data set of each participating terminal is comprehensively evaluated, thereby improving the accuracy of the quality evaluation results, and thereby improving the rationality and fairness of the quality evaluation of each participating terminal based on the local data set.

[0124] In order to accurately quantify the contribution of each participant in federated learning, based on the above embodiment, in one embodiment, the contribution of each participant in federated learning can be further determined based on a contribution matrix. As shown in FIG5 , the above S402 may include:

[0125] S501 : Construct a maximum value vector and a minimum value vector according to the maximum element and the minimum element of each column in the contribution matrix.

[0126] For each column in the contribution matrix, the maximum element and the minimum element of the column are determined, and then a maximum value vector is constructed based on the maximum elements of all columns, and a minimum value vector is constructed based on the minimum elements of all columns.

[0127] The maximum value vector is represented as: Z + =(Z1+ ,Z2 + ,Z3 + ,Z4 + ) =(max{z 11 ,…,z n1},max{z 12 ,…,z n2},max{z 13 ,…,z n3},max{z 14 ,…,z n4});

[0128] Among them, Z1 + is the maximum element in the first column of the contribution matrix, Z2 + is the maximum element in the second column of the contribution matrix, Z3 + is the maximum element in the third column of the contribution matrix, Z4 + is the maximum element in the fourth column of the contribution matrix.

[0129] The minimum vector is represented by: Z - =(Z1 - ,Z2 - ,Z3 - ,Z4 - ) =(min{z 11 ,…,z n1},min{z 12 ,…,z n2},min{z 13 ,…,z n3},min{z 14 ,…,z n4});

[0130] Among them, Z1 - is the minimum element in the first column of the contribution matrix, Z2 - is the minimum element in the second column of the contribution matrix, Z3 - is the minimum element in the third column of the contribution matrix, Z4 - is the minimum element in the fourth column of the contribution matrix.

[0131] For example, in the federated learning architecture, there are three participants A, B, and C. The data quality parameters of the local datasets of the three participants under various evaluation indicators are shown in the following table:

[0132] The contribution matrix of federated learning is expressed as:

[0133] The contribution matrix after standardization is expressed as:

[0134] Correspondingly, the maximum value vector Z + =(0.54,0.46,0.63,0.78);

[0135] Minimum vector Z - =(0.36,0.36,0.35,0.16).

[0136] S502 , determining the distance between each row vector in the contribution matrix and the maximum value vector according to the difference between each row element in the contribution matrix and the corresponding element in the maximum value vector, and the weight of each evaluation index.

[0137] Z j + is the maximum element in the jth column of the contribution matrix, z kj is the element in the kth row and jth column of the contribution matrix, which can be expressed as Z j + -z kj Represents the difference between each row element in the contribution matrix and the corresponding element in the maximum value vector.

[0138] Furthermore, combined with the weight w of index j j , the distance between each row vector and the maximum value vector in the contribution matrix is ​​constructed as:

[0139] S503 , determining the distance between each row vector in the contribution matrix and the minimum vector according to the difference between each row element in the contribution matrix and the corresponding element in the minimum vector, and the weight of each evaluation index.

[0140] Z j - is the minimum element in the jth column of the contribution matrix, z kj is the element in the kth row and jth column of the contribution matrix, which can be expressed as Z j - -z kj Represents the difference between each row element in the contribution matrix and the corresponding element in the minimum value vector.

[0141] Furthermore, combined with the weight w of index j j , the distance between each row vector and the minimum vector in the contribution matrix is ​​constructed as:

[0142] For example, w1 = 0.1, w2 = 0.15, w3 = 0.15, w4 = 0.6, the distances between each row vector and the maximum value vector and the minimum value vector are shown in the following table:

[0143] S504: Determine the contribution of each participating terminal according to the distance between each row vector and the maximum value vector, and the distance between each row vector and the minimum value vector.

[0144] Specifically, the evaluation results of the data quality parameters of the participating terminals are: Furthermore, the server evaluates the contribution of the participating end in federated learning based on the evaluation results of the data quality parameters of the participating end. The contribution of the participating end is

[0145] The evaluation results of the data quality parameters of the participating terminals and their contributions are shown in the following table:

[0146] It should be noted that the number of columns and indicators in the above contribution matrix is ​​only for example and does not mean that the indicators in this embodiment are limited to 4. In the scenario where the server determines any number of evaluation indicators, the steps in this embodiment can be taken to calculate the contribution of each participating terminal, and no further details will be given.

[0147] In this embodiment, TOPSIS is used to determine the positive and negative ideal solutions, that is, the maximum value vector and the minimum value vector, and then calculate the distance of each sample from the positive and negative ideal solutions, that is, the distance between each row vector and the maximum value vector, and the distance between each row vector and the minimum value vector, so as to quantify the degree of closeness of each participating end to the optimal solution, so as to obtain the contribution of each participating end in federated learning, thereby improving the accuracy and fairness of measuring the contribution.

[0148] In one embodiment, as shown in FIG6 , a contribution evaluation method applied to a participant terminal is provided, comprising the following steps:

[0149] S601: Receive, from a server, a feature probability distribution of a benchmark data set corresponding to a target task, and evaluation indicators corresponding to the target task and calculation methods of the evaluation indicators.

[0150] The federated learning framework includes a server and several participating terminals. This embodiment is applicable to any participating terminal. Specifically, a participating terminal receives the feature probability distribution of a benchmark dataset corresponding to the target task, as well as the evaluation metrics corresponding to the target task and their calculation methods, sent synchronously or asynchronously by the server. Optionally, a participating terminal proactively downloads the feature probability distribution of the benchmark dataset from the server.

[0151] The target task is the task of the global model constructed by the current federated learning framework, such as image classification or object detection. The benchmark dataset is a high-quality training dataset that can express the task requirements. Different tasks correspond to different benchmark datasets. For example, if the target task is an object detection task, the benchmark dataset corresponding to the object detection task may contain several frames of images with the target to be detected, as well as target labels corresponding to the target to be detected, such as vehicles, pedestrians, or buildings. The features of these targets to be detected have their own probability distribution during the sampling process, which is the feature probability distribution of the benchmark dataset corresponding to the object detection task.

[0152] A target task corresponds to multiple evaluation indicators, where each evaluation indicator includes task-independent indicators and task-related indicators. Specifically, the evaluation indicators are used to assess the contribution of the participating end in the current federated learning framework based on the training data of the participating end. Task-independent indicators refer to indicators that evaluate the quality of the training data itself, reflecting the inherent properties of the training data. Optional task-independent indicators include data accuracy, data completeness, and data uniqueness. Task-related indicators refer to indicators that evaluate the quality of the training data in the context of the target task, meeting the requirements related to the training data and the task. Optional task-related indicators include data relevance.

[0153] S602: Determine the data quality parameters of the local dataset under each evaluation indicator based on the feature probability distribution of the benchmark dataset and the calculation method of each evaluation indicator.

[0154] The local data set of the participating end is the local training data of the participating end. The data quality parameters of the local data set are used to measure the quality of the local data set in a digital way. The data quality parameters include task-independent quality parameters and task-related quality parameters. It can be understood that task-independent quality parameters correspond to task-independent indicators, and task-related quality parameters correspond to task-related indicators.

[0155] The participating end can calculate the task-independent quality parameters of the local dataset under various task-independent indicators based on the task-independent indicators and the calculation method of the task-independent indicators, and calculate the task-related quality parameters of the local dataset under the task-related indicators based on the feature probability distribution of the benchmark dataset, the task-related indicators and the calculation method of the task-related indicators.

[0156] Optionally, the participating end calculates the task-independent quality parameters of the local dataset under the indicator of data accuracy based on the missing values, error values, outliers or sampling deviation results of the local dataset; the participating end calculates the task-independent quality parameters of the local dataset under the indicator of data completeness based on the coverage of the local dataset; the participating end calculates the task-independent quality parameters of the local dataset under the indicator of data uniqueness based on the repetition rate of the local dataset; the participating end calculates the task-related quality parameters of the local dataset under the indicator of data relevance based on the similarity between the feature probability distribution of the local dataset and the feature probability distribution of the benchmark dataset.

[0157] This embodiment does not limit the calculation method of the task-independent indicators. The calculation method of the task-related indicators may include determining the degree of correlation between the local training data of the participating terminal and the benchmark data set.

[0158] S603: Send the data quality parameters to the server, so that the server can evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

[0159] The participating terminals send the task-independent quality parameters and the task-related quality parameters to the server, so that the server can comprehensively evaluate the contribution of all participating terminals based on the task-independent quality parameters and the task-related quality parameters of all participating terminals.

[0160] Optionally, the server can construct a contribution matrix that includes both task-independent and task-dependent quality parameters for all participating clients. Based on this contribution matrix, the server can then uniformly evaluate the contribution of each participating client along the same dimension, thereby ensuring objectivity and fairness in the evaluation of the contribution of each participating client. For example, the server can assign different contribution weights to different indicators. For each indicator, the server uniformly measures the contribution of all participating clients based on the data quality parameters of their local datasets under that indicator. Furthermore, the server can then use the corresponding contribution weights for each indicator to comprehensively measure the contribution of all participating clients.

[0161] In the above scheme, the participating end receives the characteristic probability distribution of the benchmark data set corresponding to the target task sent by the server, as well as the various evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator, and then determines the data quality parameters of the local data set under each evaluation indicator based on the characteristic probability distribution of the benchmark data set and the calculation method of each evaluation indicator. Furthermore, the data quality parameters are sent to the server so that the server can evaluate the contribution of each participating end based on the data quality parameters sent by each participating end; wherein, each evaluation indicator includes task-independent indicators and task-related indicators, and the data quality parameters include task-independent quality parameters and task-related quality parameters. This embodiment comprehensively evaluates the contribution of each participating terminal from both task-independent and task-related dimensions. It not only considers the data quality of the local dataset of each participating terminal itself, but also considers the data quality of the local dataset of each participating terminal in a specific task scenario. This can improve the rationality of the contribution evaluation of each participating terminal in federated learning. Moreover, based on the benchmark dataset corresponding to the specific task scenario, the data quality of the local dataset of each participating terminal under task-related indicators is evaluated, which can ensure the objectivity and fairness of the evaluation results. Furthermore, based on the data quality evaluation results, a more accurate evaluation of the contribution of each participating terminal in federated learning can be achieved.

[0162] In order to reasonably determine the data quality parameters of the local dataset under each evaluation indicator for each evaluation indicator, based on the above embodiment, in one embodiment, the data quality parameters of the local dataset under each evaluation indicator can be calculated respectively according to the calculation method corresponding to each evaluation indicator. As shown in FIG7 , the above S602 may include:

[0163] S701 , determining a task-independent quality parameter of a local data set under the task-independent indicator according to a calculation method of the task-independent indicator.

[0164] The participating end can calculate the task-independent quality parameters of the local data set under each task-independent indicator based on the task-independent indicator and the calculation method of the task-independent indicator.

[0165] For example, the participating end calculates the task-independent quality parameters of the local dataset under the indicator of data accuracy based on the missing values, erroneous values, outliers, or sampling deviation results of the local dataset; the participating end calculates the task-independent quality parameters of the local dataset under the indicator of data completeness based on the coverage of the local dataset; the participating end calculates the task-independent quality parameters of the local dataset under the indicator of data uniqueness based on the repetition rate of the local dataset; the participating end calculates the task-related quality parameters of the local dataset under the indicator of data relevance based on the similarity between the feature probability distribution of the local dataset and the feature probability distribution of the benchmark dataset. This embodiment does not limit the calculation method of the task-independent indicators.

[0166] S702 : Determine task-related quality parameters of the local dataset under the task-related indicators according to a calculation method of the task-related indicators and a characteristic probability distribution of the benchmark dataset.

[0167] The participating end can determine the task-related quality parameters of the local dataset under the task-related indicators based on the calculation method of the task-related indicators, the characteristic probability distribution of the benchmark dataset, and the characteristic probability distribution of the local dataset.

[0168] Specifically, the feature probability distribution of the benchmark dataset is recorded as P s , the characteristic probability distribution of the local data set is recorded as P k , by calculating P s and P k The correlation between the benchmark dataset and the local dataset is measured by Kullback-Leibler divergence, bulldozer distance, total variation distance, Hellinger distance, Jensen-Shannon divergence, or cosine similarity, and the numerical value representing the correlation is converted into a task-related quality parameter of the local dataset under task-related indicators.

[0169] In this embodiment, the participating end determines the task-independent quality parameters of the local dataset under the task-independent indicators based on the calculation method of the task-independent indicators, and determines the task-related quality parameters of the local dataset under the task-related indicators based on the calculation method of the task-related indicators and the characteristic probability distribution of the benchmark dataset. This not only takes into account the data quality of the local dataset itself, but also takes into account the fit of the local dataset in a specific target task scenario. Under the premise of ensuring data privacy and security, the accuracy of the data quality measurement results is improved.

[0170] To evaluate the similarity between the benchmark dataset and the local dataset, based on the above embodiment, in one embodiment, the bulldozer distance may be used to represent the distance between two feature probability distributions, thereby measuring the similarity between the two datasets. As shown in FIG8 , the above S702 may include:

[0171] S801, obtaining the feature probability distribution of the local data set.

[0172] The participating end obtains the local dataset corresponding to the target task, extracts the data features from the local dataset, and then calculates the probability distribution of the data features to obtain the feature probability distribution of the local dataset corresponding to the target task. For example, if the target task is a target detection task, the local dataset corresponding to the target detection task may contain several frames of images with the target to be detected, as well as target labels corresponding to the target to be detected, such as vehicles, pedestrians, or buildings. The features of these targets to be detected have their own probability distribution during the sampling process, which is the feature probability distribution of the local dataset corresponding to the target detection task.

[0173] S802 : Determine a bulldozer distance between a feature probability distribution of a local dataset and a feature probability distribution of a reference dataset according to the calculation method of the task-related indicator.

[0174] The method for calculating the task-related indicator includes determining the degree of correlation between the local dataset of each participating terminal and the reference dataset. In one embodiment, a bulldozer distance is calculated between the feature probability distribution of the reference dataset and the feature probability distribution of the local dataset, and the bulldozer distance is used as the degree of correlation between the reference dataset and the local dataset.

[0175] Specifically, the feature probability distribution P of the benchmark dataset s and the characteristic probability distribution P of the local dataset k The bulldozer distance between is expressed as:

[0176] Among them, inf represents the lower bound, Π(P k ,P s ) indicates P s and P k The set of all joint distributions γ(x,y). For any possible joint distribution γ(x,y), sample (x,y)~γ from it and get the distance ‖xy‖ between samples x and y. It represents the expected value of the distance between samples under the joint distribution γ(x,y).

[0177] S803: Determine, according to the bulldozer distance, a task-related quality parameter of the local data set under the task-related indicator.

[0178] In one embodiment, the reciprocal of the bulldozer distance may be determined as the task-related quality parameter of the local dataset under the task-related indicator.

[0179] The smaller the bulldozer distance, the higher the correlation between the benchmark dataset and the local dataset, and the higher the quality of the local dataset of the participating end under the task-related indicators; conversely, the larger the bulldozer distance, the lower the correlation between the benchmark dataset and the local dataset, and the lower the quality of the local dataset of the participating end under the task-related indicators.

[0180] Therefore, the inverse of the bulldozer distance is taken as the task-related quality parameter of the local dataset under the task-related indicators, i.e.

[0181] The bulldozer distance and task-related quality parameters are shown in the following table:

[0182] In this embodiment, the bulldozer distance is used to measure the similarity between the benchmark dataset and the local dataset, and then the task-related quality parameters of the local dataset are determined based on the bulldozer distance. At low latitudes and when there is no overlap between the feature probability distributions, the bulldozer distance can still accurately measure the similarity between the benchmark dataset and the local dataset, thereby ensuring the accuracy of the determined task-related quality parameters.

[0183] In order to more accurately calculate the feature probability distribution of the local data set, based on the above embodiment, in one embodiment, as shown in FIG9 , the above S801 may include:

[0184] S901: Build a local dataset corresponding to the target task.

[0185] The participating end constructs a local dataset corresponding to the target task based on local training data. For example, if the target task is object detection, the corresponding local dataset may include several frames of images with the target to be detected, as well as the target labels corresponding to the target to be detected, such as vehicles, pedestrians, or buildings.

[0186] S902: Extract data features of the local data set.

[0187] Use RNN, CNN, or Transformer methods to extract data features from the training data in the local dataset. For example, if the target task is target detection, for a set of images in the local dataset containing several frames of the target to be detected, extract the static features of the target to be detected in each frame, as well as the dynamic features of the target to be detected in adjacent frames. The static and dynamic features of the target to be detected can represent the target type to be detected, such as a vehicle, pedestrian, or building. The dynamic features of the target to be detected can also represent the motion trajectory of the target to be detected, such as the motion trajectory of a vehicle or pedestrian.

[0188] S903: Determine a feature probability distribution of the local data set according to the data features of the local data set.

[0189] The probability distribution of the data features of the training data in the local dataset is determined using methods such as KDE, latent variable models, or GANs. For example, if the target task is object detection, for a set of images in the local dataset containing several frames of the target to be detected, the probability distribution of the static and dynamic features of the target to be detected in the target task scenario is calculated based on the static features of the target to be detected in each frame and the dynamic features of the target to be detected in adjacent frames. The probability distribution of the data features of all the training data in the local dataset is further analyzed to obtain the feature probability distribution of the benchmark dataset.

[0190] Among them, the characteristic probability distribution of the local data set is used for the participating terminals to determine task-related quality parameters.

[0191] In this embodiment, the participating end constructs a local data set corresponding to the target task, extracts data features of the local data set, and determines the feature probability distribution of the local data set based on the data features of the local data set, thereby ensuring the accuracy of the calculated feature probability distribution of the local data set, and further improving the rationality of the contribution evaluation of the participating ends in federated learning.

[0192] In one embodiment, an optional example of a contribution evaluation method applied to a server is provided, as shown in FIG10 . The specific process is as follows:

[0193] S1001, construct a benchmark dataset corresponding to the target task.

[0194] S1002, extracting data features of the benchmark data set.

[0195] S1003: Determine a feature probability distribution of the benchmark data set according to the data features of the benchmark data set.

[0196] Among them, the characteristic probability distribution of the benchmark data set is used for each participating end to determine the task-related quality parameters of the local data set under task-related indicators.

[0197] S1004: Determine the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator.

[0198] S1005: Send the feature probability distribution of the benchmark dataset, each evaluation indicator, and the calculation method of each evaluation indicator to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local dataset under each evaluation indicator.

[0199] Among them, the evaluation indicators include task-independent indicators and task-related indicators, and the data quality parameters include task-independent quality parameters and task-related quality parameters; task-independent indicators include data accuracy, data integrity and data uniqueness, and task-related indicators include data relevance.

[0200] S1006: Receive data quality parameters sent by each participating terminal.

[0201] S1007: Construct a contribution matrix for federated learning based on the data quality parameters sent by each participating terminal.

[0202] S1008 , constructing a maximum value vector and a minimum value vector according to the maximum element and the minimum element of each column in the contribution matrix.

[0203] S1009 , determining the distance between each row vector in the contribution matrix and the maximum value vector according to the difference between each row element in the contribution matrix and the corresponding element in the maximum value vector, and the weight of each evaluation index.

[0204] S1010 , determining the distance between each row vector in the contribution matrix and the minimum vector according to the difference between each row element in the contribution matrix and the corresponding element in the minimum vector, and the weight of each evaluation index.

[0205] S1011 , determining the contribution of each participating terminal according to the distance between each row vector and the maximum value vector, and the distance between each row vector and the minimum value vector.

[0206] Among them, the elements in the same row of the contribution matrix are the data quality parameters of the same participating end under each evaluation indicator, and the elements in the same column are the data quality parameters of each participating end under the same evaluation indicator.

[0207] In one embodiment, an optional example of a contribution evaluation method applied to a participant is provided, as shown in FIG11 . The specific process is as follows:

[0208] S1101, construct a local dataset corresponding to the target task.

[0209] S1102: Extract data features of the local data set.

[0210] S1103: Determine a feature probability distribution of the local data set according to the data features of the local data set.

[0211] S1104: Receive the feature probability distribution of the benchmark data set corresponding to the target task, as well as the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator from the server.

[0212] S1105 , determining a task-independent quality parameter of the local data set under the task-independent indicator according to a calculation method of the task-independent indicator.

[0213] S1106 , determining a bulldozer distance between the feature probability distribution of the benchmark dataset and the feature probability distribution of the local dataset.

[0214] S1107 : Determine the reciprocal of the bulldozer distance as a task-related quality parameter of the local dataset under the task-related indicator.

[0215] Among them, the evaluation indicators include task-independent indicators and task-related indicators, and the data quality parameters include task-independent quality parameters and task-related quality parameters; task-independent indicators include data accuracy, data integrity and data uniqueness, and task-related indicators include data relevance.

[0216] S1108: Send the data quality parameters to the server, so that the server can evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0217] In one embodiment, an optional example of a contribution evaluation method is provided, as shown in FIG12 . The specific process is as follows:

[0218] S1201: The server constructs a benchmark data set, extracts data features of the benchmark data set, and calculates the probability distribution of the data features of the benchmark data set.

[0219] S1202: The server determines each evaluation indicator and a calculation method for each evaluation indicator.

[0220] Among them, each evaluation indicator includes task-independent indicators and task-related indicators.

[0221] S1203: The server sends the probability distribution of the data features of the benchmark data set, each evaluation index and the calculation method of each evaluation index to the participating terminal.

[0222] Among them, the participating end is any participating end in the federated learning framework.

[0223] S1204, the participating end calculates the task-independent quality parameters of the local data set under the task-independent indicators.

[0224] S1205, the participating end calculates the task-related quality parameters of the local data set under the task-related indicators.

[0225] S1206: The participating end sends the task-independent quality parameters and the task-related quality parameters to the server.

[0226] S1207, the server evaluates the overall quality of the data from the participating terminals.

[0227] S1208: The server evaluates the contribution of the participating terminals.

[0228] The specific process of the above steps can be found in the description of the above method embodiment. The implementation principles and technical effects are similar and will not be repeated here.

[0229] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0230] Based on the same inventive concept, embodiments of the present application also provide a contribution assessment device for implementing the aforementioned contribution assessment method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following contribution assessment device embodiments can be found in the limitations of the contribution assessment method described above and will not be further elaborated here.

[0231] In one embodiment, as shown in FIG13 , a contribution evaluation device 1 configured on a server is provided, comprising: a determination module 10 , a sending module 20 , and an evaluation module 30 , wherein:

[0232] The determination module 10 is used to obtain the characteristic probability distribution of the benchmark data set corresponding to the target task, and determine the evaluation indicators corresponding to the target task and the calculation method of each evaluation indicator.

[0233] The sending module 20 is used to send the feature probability distribution of the benchmark data set, each evaluation indicator and the calculation method of each evaluation indicator to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local data set under each evaluation indicator.

[0234] Among them, data quality parameters include task-independent quality parameters and task-related quality parameters.

[0235] The evaluation module 30 is configured to receive data quality parameters sent by each participating terminal and evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

[0236] In one embodiment, each evaluation indicator includes a task-independent indicator and a task-related indicator. The task-independent indicator includes data accuracy, data integrity, and data uniqueness, and the task-related indicator includes data relevance.

[0237] In one embodiment, the determination module 10 may also be used to determine task-independent indicators and task-related indicators corresponding to the target task, and determine a calculation method for the task-independent indicators and a calculation method for the task-related indicators.

[0238] In one embodiment, the determination module 10 may also be used to determine the degree of correlation between the local dataset of each participating terminal and the reference dataset.

[0239] In one embodiment, based on FIG. 13 , as shown in FIG. 14 , the determination module 10 may include:

[0240] The data set construction unit 11 is used to construct a benchmark data set corresponding to the target task.

[0241] The feature extraction unit 12 is used to extract data features of the benchmark data set.

[0242] The distribution determining unit 13 is configured to determine a feature probability distribution of the reference data set according to data features of the reference data set.

[0243] In one embodiment, the characteristic probability distribution of the benchmark dataset is used by each participating terminal to determine task-related quality parameters of the local dataset under task-related indicators.

[0244] In one embodiment, based on FIG. 13 , as shown in FIG. 15 , the evaluation module 30 may include:

[0245] The matrix construction unit 31 is used to construct a contribution matrix of federated learning based on the data quality parameters sent by each participating terminal.

[0246] The contribution determination unit 32 is configured to determine the contribution of each participating terminal according to the contribution matrix and the weights of the evaluation indicators.

[0247] In one embodiment, elements in the same row of the contribution matrix are data quality parameters of the same participating terminal under various evaluation indicators, and elements in the same column are data quality parameters of various participating terminals under the same evaluation indicator.

[0248] In one embodiment, the contribution determination unit may include:

[0249] The vector construction subunit is used to construct a maximum value vector and a minimum value vector according to the maximum element and the minimum element of each column in the contribution matrix.

[0250] The first determining subunit is used to determine the distance between each row vector in the contribution matrix and the maximum value vector according to the distance between each row element in the contribution matrix and the corresponding element in the maximum value vector, and the weight of each evaluation index.

[0251] The second determining subunit is used to determine the distance between each row vector in the contribution matrix and the minimum value vector according to the distance between each row element in the contribution matrix and the corresponding element in the minimum value vector, and the weight of each evaluation index.

[0252] The third determining subunit is configured to determine the contribution of each participating terminal according to the distance between each row vector and the maximum value vector, and the distance between each row vector and the minimum value vector.

[0253] In one embodiment, as shown in FIG16 , a contribution evaluation device 2 configured at a participant terminal is provided, comprising: a receiving module 40 , a determining module 50 , and a sending module 60 , wherein:

[0254] The receiving module 40 is configured to receive the feature probability distribution of the benchmark data set corresponding to the target task, as well as the evaluation indicators corresponding to the target task and the calculation methods of the evaluation indicators, sent by the server.

[0255] The determination module 50 is configured to determine the data quality parameters of the local dataset under various evaluation indicators based on the characteristic probability distribution of the benchmark dataset and the calculation method of various evaluation indicators.

[0256] Among them, data quality parameters include task-independent quality parameters and task-related quality parameters.

[0257] The sending module 60 is configured to send the data quality parameters to the server, so that the server can evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0258] In one embodiment, each evaluation indicator includes a task-independent indicator and a task-related indicator. The task-independent indicator includes data accuracy, data integrity, and data uniqueness, and the task-related indicator includes data relevance.

[0259] In one embodiment, based on FIG. 16 , as shown in FIG. 17 , the determination module 50 may include:

[0260] The first parameter determination unit 51 is configured to determine a task-independent quality parameter of a local data set under a task-independent indicator according to a calculation method of the task-independent indicator.

[0261] The second parameter determination unit 52 is configured to determine the task-related quality parameters of the local dataset under the task-related indicators according to the calculation method of the task-related indicators and the characteristic probability distribution of the benchmark dataset.

[0262] In one embodiment, the second parameter determination unit may include:

[0263] The distance determination subunit is configured to determine a bulldozer distance between a feature probability distribution of a reference data set and a feature probability distribution of a local data set according to a calculation method of the task-related indicator.

[0264] The parameter determination subunit is configured to determine, according to the bulldozer distance, a task-related quality parameter of the local data set under the task-related indicator.

[0265] In one embodiment, the parameter determination subunit is further configured to: determine the reciprocal of the bulldozer distance as a task-related quality parameter of the local dataset under the task-related indicator.

[0266] In one embodiment, the contribution evaluation device 2 may further include:

[0267] The construction module is used to build the local dataset corresponding to the target task.

[0268] The extraction module is used to extract data features of the local dataset.

[0269] The determination module is used to determine the feature probability distribution of the local data set according to the data features of the local data set.

[0270] Each module in the contribution evaluation device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0271] In one embodiment, a communication device is provided, comprising a transmitter, a receiver, a processor, and a memory, wherein the memory stores a computer program. The processor of the communication device is configured to provide computing and control capabilities, and when the processor executes the computer program, the steps of each of the above-described method embodiments are implemented. The memory of the communication device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the communication device is configured to communicate with an external terminal via a network connection.

[0272] In one embodiment, a communication device is provided, including a transmitter, a receiver, a processor and a memory, wherein the memory stores a computer program; the computer program is executed by the processor to obtain a characteristic probability distribution of a benchmark data set corresponding to a target task, and to determine each evaluation indicator corresponding to the target task and a calculation method for each evaluation indicator; the transmitter is used to send the characteristic probability distribution of the benchmark data set, each evaluation indicator and a calculation method for each evaluation indicator to each participating end of federated learning, so that each participating end can determine the data quality parameters of the local data set under each evaluation indicator; the data quality parameters include task-independent quality parameters and task-related quality parameters; the receiver is used to receive the data quality parameters sent by each participating end, and to evaluate the contribution of each participating end based on the data quality parameters sent by each participating end.

[0273] In one embodiment, a communication device is provided, including a transmitter, a receiver, a processor and a memory, wherein the memory stores a computer program; the receiver is used to receive a characteristic probability distribution of a benchmark data set corresponding to a target task, as well as various evaluation indicators corresponding to the target task and a calculation method for each evaluation indicator, sent by a server; the computer program is executed by the processor to determine data quality parameters of a local data set under various evaluation indicators based on the characteristic probability distribution of the benchmark data set and the calculation method for each evaluation indicator; the data quality parameters include task-independent quality parameters and task-related quality parameters; the transmitter is used to send the data quality parameters to the server, so that the server can evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

[0274] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the contribution evaluation method when executing the computer program.

[0275] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the contribution evaluation method described above are implemented.

[0276] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above contribution evaluation method when executed by a processor.

[0277] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0278] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0279] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A contribution evaluation method, applied to a server, comprising: Obtaining a characteristic probability distribution of a benchmark data set corresponding to a target task, and determining each evaluation index corresponding to the target task and a calculation method for each evaluation index; Sending the characteristic probability distribution of the benchmark data set, each evaluation index and the calculation method of each evaluation index to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local data set under each evaluation index; wherein the data quality parameters include task-independent quality parameters and task-related quality parameters; Receive data quality parameters sent by each participating terminal, and evaluate the contribution of each participating terminal based on the data quality parameters sent by each participating terminal.

2. The method according to claim 1, wherein: The step of obtaining a feature probability distribution of a benchmark data set corresponding to the target task includes: Build a benchmark dataset corresponding to the target task; Extracting data features of the benchmark data set; According to the data features of the benchmark data set, a feature probability distribution of the benchmark data set is determined.

3. The method according to claim 1, wherein: The evaluation indicators include task-independent indicators and task-related indicators; the task-independent indicators include data accuracy, data completeness and data uniqueness, and the task-related indicators include data relevance; the characteristic probability distribution of the benchmark data set is used for each participating end to determine the task-related quality parameters of the local data set under the task-related indicators.

4. The method according to claim 3, wherein the determining of each evaluation index corresponding to the target task and the calculation method of each evaluation index comprises: Determining task-independent indicators and task-related indicators corresponding to the target task, and determining a calculation method for the task-independent indicators and a calculation method for the task-related indicators; The calculation method for determining the task-related indicators includes: Determine the correlation between the local data set of each participating terminal and the reference data set.

5. The method according to claim 1, wherein: The step of evaluating the contribution of each participating terminal according to the data quality parameters sent by each participating terminal includes: Construct the contribution matrix of federated learning based on the data quality parameters sent by each participating end; The contribution of each participating terminal is determined based on the contribution matrix and the weight of each evaluation index.

6. The method according to claim 5, wherein: The elements in the same row of the contribution matrix are the data quality parameters of the same participating terminal under each evaluation index, and the elements in the same column are the data quality parameters of each participating terminal under the same evaluation index.

7. The method according to claim 5, wherein: Determining the contribution of each participating terminal according to the contribution matrix and the weight of each evaluation index includes: Constructing a maximum value vector and a minimum value vector according to the maximum element and the minimum element of each column in the contribution matrix; Determine the distance between each row vector in the contribution matrix and the maximum value vector according to the difference between each row element in the contribution matrix and the corresponding element in the maximum value vector, and the weight of each evaluation index; Determine the distance between each row vector in the contribution matrix and the minimum value vector according to the difference between each row element in the contribution matrix and the corresponding element in the minimum value vector, and the weight of each evaluation index; The contribution of each participating terminal is determined according to the distance between each row vector and the maximum value vector, and the distance between each row vector and the minimum value vector.

8. A contribution evaluation method, wherein: Applied to the participating end, including: Receive the feature probability distribution of the benchmark data set corresponding to the target task sent by the server, as well as the evaluation indicators corresponding to the target task and the calculation method of the evaluation indicators; Determine the data quality parameters of the local data set under each evaluation indicator according to the characteristic probability distribution of the benchmark data set and the calculation method of each evaluation indicator; the data quality parameters include task-independent quality parameters and task-related quality parameters; The data quality parameters are sent to the server so that the server can evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

9. The method according to claim 8, wherein: The evaluation indicators include task-independent indicators and task-related indicators; the task-independent indicators include data accuracy, data completeness and data uniqueness, and the task-related indicators include data relevance.

10. The method according to claim 9, wherein: Determining the data quality parameters of the local data set under each evaluation indicator according to the characteristic probability distribution of the benchmark data set and the calculation method of each evaluation indicator includes: Determine the task-independent quality parameter of the local data set under the task-independent indicator according to the calculation method of the task-independent indicator; According to the calculation method of the task-related indicator and the characteristic probability distribution of the benchmark data set, the task-related quality parameters of the local data set under the task-related indicator are determined.

11. The method according to claim 10, wherein: Determining the task-related quality parameters of the local data set under the task-related indicators according to the calculation method of the task-related indicators and the characteristic probability distribution of the benchmark data set includes: Get the feature probability distribution of the local data set; Determining a bulldozer distance between a feature probability distribution of the local data set and a feature probability distribution of the benchmark data set according to a calculation method of the task-related indicator; According to the bulldozer distance, a task-related quality parameter of the local data set under the task-related indicator is determined.

12. The method according to claim 11, wherein: Determining the task-related quality parameter of the local data set under the task-related indicator according to the bulldozer distance includes: The reciprocal of the bulldozer distance is determined as the task-related quality parameter of the local data set under the task-related indicator.

13. The method according to claim 12, wherein: The obtaining of the feature probability distribution of the local data set includes: Build a local dataset corresponding to the target task; Extracting data features of the local data set; According to the data features of the local data set, a feature probability distribution of the local data set is determined.

14. A contribution evaluation device, configured on a server, comprising: A determination module is used to obtain the characteristic probability distribution of the benchmark data set corresponding to the target task, and determine each evaluation index corresponding to the target task and the calculation method of each evaluation index; A sending module, used to send the feature probability distribution of the benchmark data set, each evaluation index and the calculation method of each evaluation index to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local data set under each evaluation index; the data quality parameters include task-independent quality parameters and task-related quality parameters; The evaluation module is used to receive the data quality parameters sent by each participating terminal, and evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

15. A contribution evaluation device, configured at a participant terminal, comprising: A receiving module, used to receive the characteristic probability distribution of the benchmark data set corresponding to the target task sent by the server, as well as each evaluation index corresponding to the target task and the calculation method of each evaluation index; A determination module, used to determine the data quality parameters of the local data set under each evaluation indicator according to the characteristic probability distribution of the benchmark data set and the calculation method of each evaluation indicator; the data quality parameters include task-independent quality parameters and task-related quality parameters; The sending module is used to send the data quality parameters to the server, so that the server can evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

16. A communication device comprising: a transmitter, a receiver, a processor and a memory, the memory storing a computer program; The computer program is executed by the processor to obtain a feature probability distribution of a benchmark data set corresponding to a target task, and determine each evaluation index corresponding to the target task and a calculation method for each evaluation index; The transmitter is used to send the feature probability distribution of the benchmark data set, each evaluation index and the calculation method of each evaluation index to each participating end of the federated learning, so that each participating end can determine the data quality parameters of the local data set under each evaluation index; the data quality parameters include task-independent quality parameters and task-related quality parameters; The receiver is used to receive the data quality parameters sent by each participating terminal, and evaluate the contribution of each participating terminal according to the data quality parameters sent by each participating terminal.

17. A communication device comprising: a transmitter, a receiver, a processor and a memory, the memory storing a computer program; The receiver is used to receive the feature probability distribution of the benchmark data set corresponding to the target task sent by the server, as well as each evaluation index corresponding to the target task and the calculation method of each evaluation index; The computer program is executed by the processor to determine data quality parameters of the local data set under each evaluation indicator according to the characteristic probability distribution of the benchmark data set and the calculation method of each evaluation indicator; the data quality parameters include task-independent quality parameters and task-related quality parameters; The transmitter is used to send the data quality parameter to the server, so that the server can evaluate the contribution of each participating terminal according to the data quality parameter sent by each participating terminal.

18. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images