Client selection method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a client selection method and device, electronic equipment, a storage medium and a program product. According to the embodiment of the invention, an initial model vector of a to-be-trained model is sent to a client; obtaining a training model vector of the client; based on the initial model vector and the training model vector, determining an update vector corresponding to each client and a mean update vector; determining a sample distribution difference, wherein the sample distribution difference reflects the difference between the multimedia sample corresponding to the update vector and the multimedia sample corresponding to the mean update vector; and determining a target client from the clients according to the sample distribution difference. In the embodiment of the invention, the difference of the distribution range of the multimedia samples in the client relative to the average distribution range of the multimedia samples can be obtained through the updating vector and the mean value updating vector, so that the target client which is not implanted with the backdoor can be screened out from the plurality of clients through the sample distribution difference. According to the scheme, the screening efficiency of the client can be improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a client selection method, device, electronic device, storage medium and program product. Background Art

[0002] Federated Learning is a distributed machine learning algorithm. A central model is set up in the central server. When training the central model, the central server distributes the model parameters of the central model to each client, so that the model parameters of the central model are updated through the client training the local model, and the trained model parameters sent by the client are received. The model parameters of the central model are adjusted by the trained model parameters, thereby achieving the purpose of training the central model.

[0003] However, the client may be implanted with a backdoor, which may cause the client to implant the backdoor into the local model when training the local model, so that the model parameters after the local model training may carry the backdoor, which is not conducive to adjusting the model parameters of the central model through the trained model parameters. Summary of the invention

[0004] The embodiments of the present application provide a client selection method, device, electronic device, storage medium and program product, which are convenient for screening clients that are not implanted with backdoors and improve the screening efficiency of clients.

[0005] The present application provides a method for selecting a client, including:

[0006] Send the initial model vector of the model to be trained to the client;

[0007] Obtaining a training model vector of the client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples;

[0008] Based on the initial model vector and the training model vector, determine the update vector corresponding to each client and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients;

[0009] determining a sample distribution difference reflecting a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector;

[0010] According to the sample distribution difference, the target client is determined from the clients.

[0011] The embodiment of the present application also provides a client selection device, including:

[0012] A sending unit, used for sending an initial model vector of a model to be trained to a client;

[0013] An acquisition unit, used to acquire a training model vector of a client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples;

[0014] A vector determination unit, used to determine an update vector corresponding to each client and a mean update vector based on the initial model vector and the training model vector, wherein the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients;

[0015] a difference determination unit, configured to determine a sample distribution difference, wherein the sample distribution difference reflects a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector;

[0016] The selection unit is used to determine a target client from the clients according to a sample distribution difference.

[0017] In some embodiments, based on the initial model vector and the training model vector, determining the update vector corresponding to each client and the mean update vector includes:

[0018] Subtract the initial model vector from the client's training model vector to obtain the update vector corresponding to the client;

[0019] Add the update vectors corresponding to all clients to get the total update vector;

[0020] Divide the total update vector by the number of clients to get the mean update vector.

[0021] In some embodiments, determining a target client from the clients according to the sample distribution difference includes:

[0022] Get the number of clients;

[0023] Determine the target quantity that meets the preset conditions from the quantity;

[0024] According to the sample distribution difference, a target number of target clients is determined from the clients.

[0025] In some embodiments, after determining a target number of target clients from the clients according to the sample distribution difference, the method further includes:

[0026] Using the update vector corresponding to the target client, update the initial model vector in the model to be trained to obtain the trained model;

[0027] Acquire multimedia content to be identified;

[0028] Use trained models to recognize multimedia content.

[0029] In some embodiments, determining a target number of target clients from the clients according to the sample distribution difference includes:

[0030] According to the sample distribution difference, a target number of target clients and other clients except the target clients are determined from the clients, and the sample distribution difference of the target clients is not greater than the sample distribution difference of the other clients;

[0031] The update vector corresponding to the target client is used to update the initial model vector in the model to be trained to obtain the trained model, including:

[0032] Using a first preset weight, weighting the update vector corresponding to the target client to obtain a first weighted vector;

[0033] Using the second preset weight, weighting the update vectors corresponding to other clients to obtain a second weighted vector;

[0034] Merging the first weighted vector and the second weighted vector to obtain a combined weight vector;

[0035] The combined weight vector is used to update the initial model vector in the model to be trained to obtain the trained model.

[0036] In some embodiments, using the update vector corresponding to the target client to update the initial model vector in the model to be trained to obtain the trained model includes:

[0037] Aggregate the update vectors corresponding to the target client to obtain an aggregate vector;

[0038] Merge the aggregated vector with the initial model vector to obtain the model update vector of the model to be trained;

[0039] The model to be trained is updated using the model update vector to obtain a trained model.

[0040] In some embodiments, the model to be trained includes model parameters and loss functions corresponding to the model parameters, the update vector includes the update parameters, the mean update vector includes the mean update parameters, and the model parameters, the update parameters and the mean update parameters correspond to each other;

[0041] Determine sample distribution differences, including:

[0042] Generate a test sample and a reference label of the test sample, where the test sample is a multimedia sample for testing the model to be trained;

[0043] Use the model to be trained to perform prediction processing on the test sample to obtain the predicted label of the test sample;

[0044] Based on the predicted label and the reference label, determine the loss value of the loss function corresponding to the model parameter;

[0045] According to the loss value of the loss function, determine the second-order derivative corresponding to the model parameter;

[0046] The sample distribution differences are determined based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters.

[0047] In some embodiments, determining the sample distribution difference based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters includes:

[0048] Subtract the mean update parameter from the update parameter to obtain the update offset parameter;

[0049] The sample distribution difference is determined based on the second-order derivatives corresponding to the model parameters and the updated offset parameters.

[0050] In some embodiments, determining the sample distribution difference based on the second-order derivatives corresponding to the model parameters and the updated offset parameters includes:

[0051] According to the second-order derivatives corresponding to the model parameters, the contribution of the updated offset parameters to the loss function is calculated;

[0052] All contributions are combined to obtain the sample distribution difference.

[0053] In some embodiments, calculating the contribution of the updated offset parameter to the loss function according to the second-order derivative corresponding to the model offset parameter includes:

[0054] Redistribute the second-order derivatives corresponding to the model parameters to obtain the weights corresponding to the model parameters. The weights are used to adjust the update effect of the model parameters.

[0055] The weight corresponding to the model parameter is multiplied by the square of the updated offset parameter to obtain the contribution of the updated offset parameter to the loss function.

[0056] In some embodiments, the second-order derivatives corresponding to the model parameters are redistributed to obtain weights corresponding to the model parameters, including:

[0057] Based on the update parameters of the same dimension in the update vector corresponding to each client, the absolute values ​​of the update parameters are summed to obtain the total absolute value;

[0058] Divide the total absolute value by the number of clients to get the mean update parameter corresponding to the update parameter;

[0059] Based on the correspondence between the update parameter and the model parameter, the redistribution weight of the second-order derivative corresponding to the model parameter is determined by using the mean update parameter corresponding to the update parameter and a preset threshold;

[0060] The redistribution weights of the second-order derivatives corresponding to all model parameters are summed to obtain the total distribution weight;

[0061] The redistributed weight of the second-order derivative corresponding to the model parameter is divided by the total distribution weight to obtain the normalized weight of the second-order derivative corresponding to the model parameter;

[0062] Sum the second-order derivatives corresponding to all model parameters to obtain the total second-order derivative;

[0063] The normalized weight of the second-order derivative corresponding to the model parameter is multiplied by the total second-order derivative to obtain the weight corresponding to the model parameter.

[0064] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in any client selection method provided in the embodiment of the present application.

[0065] An embodiment of the present application also provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps in any one of the client selection methods provided in the embodiments of the present application.

[0066] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in any client selection method provided in the embodiment of the present application.

[0067] The embodiment of the present application can send an initial model vector of a model to be trained to a client; obtain a training model vector of the client, where the training model vector is a vector obtained by the client updating the initial model vector through multimedia samples; based on the initial model vector and the training model vector, determine an update vector corresponding to each client and a mean update vector, where the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients; determine the sample distribution difference, where the sample distribution difference reflects the difference between the multimedia samples corresponding to the update vector and the multimedia samples corresponding to the mean update vector; and determine the target client from the clients based on the sample distribution difference.

[0068] In the present application, the mean update vector is the mean of the update vectors corresponding to all clients. Since the update vector can reflect the distribution range of the multimedia samples of the corresponding client, the mean update vector can reflect the average distribution range of the multimedia samples of all clients. Therefore, the difference between the distribution range of the multimedia samples in the client and the average distribution range of the multimedia samples can be obtained through the update vector and the mean update vector, that is, the sample distribution difference. Through the sample distribution difference, it is easy to screen out the target client that has not been implanted with a backdoor from multiple clients, thereby improving the client screening efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0070] Figure 1a is a scenario diagram of a client selection method provided in an embodiment of the present application;

[0071] Figure 1b It is a flowchart of a client selection method provided in an embodiment of the present application;

[0072] Figure 2a It is a schematic diagram of the client selection method provided in the embodiment of the present application applied in a federated learning scenario;

[0073] Figure 2b This is a table that reflects the average backdoor defense performance comparison of the model training method provided in the embodiments of the present application and other algorithms in multiple models, under attacks by multiple backdoors and multiple data sets.

[0074] Figure 2c This is a table that reflects the comparison of average backdoor defense performance of the model training method and other algorithms provided in the embodiments of the present application under different data sets and different backdoor attacks.

[0075] Figure 3 is a structural diagram of a client selection device provided in an embodiment of the present application;

[0076] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0078] Embodiments of the present application provide a client selection method, device, electronic device, storage medium, and program product.

[0079] The client selection device may be integrated in an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, or a personal computer (PC); the server may be a single server or a server cluster consisting of multiple servers.

[0080] In some embodiments, the client selection device may also be integrated into multiple electronic devices. For example, the client selection device may be integrated into multiple servers, and the client selection method of the present application may be implemented by multiple servers.

[0081] In some embodiments, the server may also be implemented in the form of a terminal.

[0082] For example, refer to Figure 1a , the electronic device 02 can send the initial model vector of the model to be trained to the client 01; obtain the training model vector of the client 01, the training model vector is the vector obtained by the client 01 updating the initial model vector through the multimedia sample; based on the initial model vector and the training model vector, determine the update vector corresponding to each client 01 and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client 01, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients 01; determine the sample distribution difference, the sample distribution difference reflects the difference between the multimedia sample corresponding to the update vector and the multimedia sample corresponding to the mean update vector; according to the sample distribution difference, determine the target client from the client 01.

[0083] Among them, the mean update vector is the mean of the update vectors corresponding to all clients. Since the update vector can reflect the distribution range of the multimedia samples of the corresponding client, the mean update vector can reflect the average distribution range of the multimedia samples of all clients. Therefore, the difference between the distribution range of multimedia samples in the client and the average distribution range of multimedia samples can be obtained through the update vector and the mean update vector, that is, the sample distribution difference. Through the sample distribution difference, it is easy to screen out the target client that has not been implanted with a backdoor from multiple clients, thereby improving the client screening efficiency.

[0084] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0085] Artificial Intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation, and other major directions.

[0086] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0087] In this embodiment, a client selection method based on machine learning involving artificial intelligence is provided, such as Figure 1b As shown, the specific process of the client selection method can be as follows:

[0088] 110. Send the initial model vector of the model to be trained to the client.

[0089] The client is a terminal that stores local user data and is used to train the model to be trained in the server. For example, the client can be a server, computer, smart phone, etc. that stores local user data. The user can be an individual user, enterprise, medical institution, financial institution, academic research institution, community organization, etc.

[0090] The model to be trained is the model in the server waiting to be trained in federated learning. For example, the model to be trained can be a neural network model, a decision tree model, a clustering model, etc.

[0091] The initial model vector is a vector composed of model parameters in the model to be trained.

[0092] For example, if the model to be trained is a neural network model, the model parameters in the model to be trained can be weights, biases, etc. The weight is the weight value of each connection in the neural network, and the user calculates the linear transformation between the input and hidden layers. The bias is the bias term of each hidden layer and output layer of the neural network, which is used to introduce an offset or translation of the activation function output.

[0093] For example, if the model to be trained is a decision tree model parameter, the model parameters in the model to be trained can be split features, split thresholds, etc. The split feature is the feature selected by the decision tree at each node for splitting data, and the split threshold is the feature value threshold used to divide data in the decision tree node.

[0094] For example, if the model to be trained is a clustering model, the model parameters in the model to be trained may be a center point, a distance measurement method, etc. The center point is used to represent the center point of each cluster in the clustering algorithm, and the distance measurement method is used to calculate the distance between the sample point and the cluster center.

[0095] 120. Obtain a training model vector of the client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples.

[0096] Among them, the training model vector is a vector used to update the initial model vector obtained after the client uses multimedia samples to train the local model, which reflects the learning results of the client's local model on the multimedia samples.

[0097] Multimedia samples are samples used by the client to train the local model when updating the initial model vector. For example, multimedia samples can be text, images, audio and video, etc.

[0098] 130. Based on the initial model vector and the training model vector, determine the update vector corresponding to each client and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients.

[0099] The update vector represents the change of the training model vector relative to the initial model vector, and is used to reflect the distribution range of the multimedia samples of the corresponding client.

[0100] Since the updating of the initial model vector is based on the gradient calculation of the multimedia samples in the client, the initial model vector is optimized through continuous iteration to obtain the training model vector, so that the local model in the client can better fit the multimedia samples, that is, the distribution range of the multimedia samples can affect the iteration of the local model in the client. Therefore, the training model vector can reflect the sample distribution of the multimedia samples in the client, and the update vector obtained by subtracting the initial model vector from the training model vector can reflect the changes in the distribution range of the multimedia samples in the corresponding client.

[0101] The mean update vector reflects the average distribution range of multimedia samples of all clients.

[0102] For example, since the update vector corresponding to each client can reflect the distribution range of the multimedia samples of the corresponding client, the mean update vector obtained through the update vector corresponding to each client can reflect the average distribution range of the multimedia samples of all clients.

[0103] In some embodiments, in order to calculate the update vector and the mean update vector, based on the initial model vector and the training model vector, the update vector and the mean update vector corresponding to each client are determined, including:

[0104] Subtract the initial model vector from the client's training model vector to obtain the update vector corresponding to the client;

[0105] Add the update vectors corresponding to all clients to get the total update vector;

[0106] Divide the total update vector by the number of clients to get the mean update vector.

[0107] The update vector is a vector obtained by subtracting the initial model vector from the client's training model vector.

[0108] For example, if the training model vector is [x, y, z] and the initial model vector is [a, b, c], the update vector u = [xa, yb, zc].

[0109] The total update vector is the sum of the update vectors corresponding to all clients.

[0110] The mean update vector is the mean of the update vectors corresponding to all clients.

[0111] For example, suppose there are n clients, and the update vectors of each client are u1, u2, ..., u n, then the total update vector = u1+u2+…+u n ,

[0112] 140. Determine a sample distribution difference, where the sample distribution difference reflects a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector.

[0113] The sample distribution difference reflects the difference between the multimedia samples corresponding to the update vector and the multimedia samples corresponding to the mean update vector. The sample distribution difference may be different for the update vectors corresponding to different clients. For example, the sample distribution difference includes but is not limited to F-divergence, Wasserstein distance, Kolmogorov-Smirnov test, etc.

[0114] Among them, F-divergence is an indicator used to measure the difference between two probability distributions. It is a measurement method introduced in information theory to compare the similarity or difference between two distributions. F-divergence includes Kullback-Leibler divergence (KL divergence), total variation divergence (Total Variation Divergence), Jensen-Shannon divergence, etc.

[0115] Since KL divergence measures the information loss of one distribution relative to another, total variation divergence measures the degree of difference between two distributions, and Jensen-Shannon divergence is the average of KL divergence and symmetric KL divergence, the F-divergence indicator can be used to estimate the difference between the distribution ranges of multimedia samples in two clients.

[0116] Among them, Wasserstein distance is an indicator that measures the difference between two sample distributions. It considers the minimum cost required to transform one distribution into another, and can handle distributions of different dimensions and structures.

[0117] Among them, the Kolmogorov-Smirnov test can be used to compare whether two sample distributions come from the same distribution. This test is based on the difference in the cumulative distribution function (CDF) and determines the degree of difference between the two distributions by calculating the maximum absolute difference.

[0118] In some embodiments, considering that the server cannot obtain the distribution range of multimedia samples in the client, in order to calculate the sample distribution difference, the model to be trained includes model parameters and loss functions corresponding to the model parameters, the update vector includes the update parameters, the mean update vector includes the mean update parameters, and the model parameters, the update parameters and the mean update parameters correspond to each other;

[0119] Determine sample distribution differences, including:

[0120] Generate a test sample and a reference label of the test sample, where the test sample is a multimedia sample for testing the model to be trained;

[0121] Use the model to be trained to perform prediction processing on the test sample to obtain the predicted label of the test sample;

[0122] Based on the predicted label and the reference label, determine the loss value of the loss function corresponding to the model parameter;

[0123] According to the loss value of the loss function, determine the second-order derivative corresponding to the model parameter;

[0124] The sample distribution differences are determined based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters.

[0125] The loss function is a function used to measure the difference between the predicted label and the true label in the model to be trained. For example, the loss function includes but is not limited to the KL divergence function (Kullback-Leibler Divergence), the mean square error function, the cross entropy loss function, and so on.

[0126] The model parameters are the parameters in the initial model vector that need to be updated. The parameters are used to control the behavior and prediction ability of the model to be trained. By adjusting and updating the model parameters, the model to be trained can fit the training samples and make accurate predictions. The update parameters are the parameters in the update vector that correspond to the model parameters. The update parameters are used to update the model parameters in the initial model vector corresponding to them.

[0127] The mean update parameter is a parameter in the mean update vector corresponding to the model parameter and the update parameter, and the mean update parameter is the mean of all update parameters corresponding to the model parameter.

[0128] The detection sample is a sample used to detect the model to be trained. The detection sample corresponds to the recognition task of the model to be trained, wherein the generated detection sample can be a detection sample set composed of multiple detection samples. For example, if the recognition task of the model to be trained is to recognize the semantics of text, the detection sample is text; if the recognition task of the model to be trained is to recognize the category of an image, the detection sample is an image; if the recognition task of the model to be trained is to recognize the type of audio and video, the detection sample is audio and video, and so on.

[0129] Methods for generating test samples include but are not limited to:

[0130] (1) The server uses unlabeled natural language texts such as Wikipedia and randomly labels them to generate a fake dataset. This method is called pseudo-labeling or semi-supervised learning method.

[0131] (2) Samples can be generated using a predefined list of keywords or phrases. These keywords or phrases are matched in unlabeled text and used as samples of the target category. For example, in sentiment analysis tasks, corresponding samples can be generated based on text fragments containing positive or negative sentiment words.

[0132] (3) Match the unlabeled text based on specific rules, patterns, regular expressions, etc., and generate corresponding samples. This method can be used to generate text fragments with specific structures, such as date, time, address, etc.

[0133] The reference label is the label carried by the sample of the model to be trained. For example, the reference label can be a manually annotated label or a randomly generated label. The randomly generated label can be a label generated using uniform distribution or Gaussian distribution, etc.

[0134] The method of generating labels using uniform distribution can be:

[0135] In probability theory and statistics, a uniform distribution (also called a rectangular distribution) is a probability distribution over a given interval in which each value has an equal probability density. It is one of the simplest probability distributions, characterized by the fact that the probability of each value appearing in the specified interval is equal.

[0136] In a uniform distribution, all values ​​have the same weight or probability density. This means that within a given interval, any value has the same probability of being generated.

[0137] The uniform distribution can be fully described by two parameters: the minimum value (a) and the maximum value (b). These two parameters define the range of the interval covered by the uniform distribution. Within this range, the probability density of each value is the same.

[0138] For example, if you want to generate a random number between 0 and 1 as a label, you can use a uniform distribution to generate these labels. In this case, the minimum value a is 0, the maximum value b is 1, and the generated labels (random numbers) will be uniformly distributed between 0 and 1.

[0139] The method of generating labels using Gaussian distribution can be:

[0140] In a Gaussian distribution, the values ​​of a random variable are concentrated around the mean, and the probability decreases as the distance from the mean increases. This means that most of the observed values ​​will be near the mean, and the probability of a value farther away from the mean decreases. A Gaussian distribution is fully described by two parameters: the mean and the variance. The mean determines where the center of the distribution is, while the variance determines how spread out the distribution is. The larger the variance, the more spread out the distribution is; the smaller the variance, the more concentrated the distribution is. Use the random number generator functions provided by your programming language to generate random numbers that follow a Gaussian distribution. These functions usually take the mean and variance as parameters. From the random numbers, you can generate Gaussian distribution labels that meet the specified mean and variance.

[0141] The predicted label is the label obtained by identifying the test sample through the model to be trained.

[0142] The loss value of the loss function is a measure of the difference between the predicted label and the reference label, which represents the error of the model to be trained during the training process.

[0143] The specific method for determining the difference in sample distribution based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters is:

[0144] If the sample distribution difference is F-divergence: Assume that q(z) represents the sample distribution of multimedia samples in the k-th client, and p(z) represents the sample distribution of multimedia samples in a clean client without a backdoor implanted. Through D_f(q(z)||p(z)), the sample distribution difference (F-divergence) can be estimated.

[0145] However, since the server cannot obtain the sample distribution of multimedia samples in the client, it cannot directly calculate the F-divergence through q(z) and p(z). When training a local model, the client usually uses multimedia samples to fit the parameters of the local model. If the multimedia samples can well represent the sample distribution in the actual application scenario, then the optimal parameter value should be close to the optimal parameter value of the sample distribution of the multimedia samples.

[0146] Thus, the update vector corresponding to the client in the present application can reflect the distribution range of the multimedia samples in the client, and by calculating the mean of the update vectors corresponding to all clients, that is, the mean update vector, it is used to reflect the average distribution range of the multimedia samples in all clients. Therefore, the mean update vector and the update vector corresponding to the client can be used to estimate the F-divergence (F F-div ).

[0147] From the F-Divergence Lower Bound theorem, we know that the lower bound of f-divergence is:

[0148]

[0149] When p(z) and f″(1)>0, the kth component in the second-order partial derivative matrix (Hessian matrix) of the loss function is It can be understood as the second-order derivative corresponding to the kth model parameter.

[0150] Where ο(1) represents a positive number not greater than 1, f″(1) represents the second-order derivative of the loss function at 1, and δ k is the update parameter by updating the kth dimension in the update vector, is the second-order derivative corresponding to the k-th dimension model parameter.

[0151] The second-order derivative can provide information about the curvature or convexity of the loss function around a given model parameter value. The value of is positive, indicating that the loss function corresponding to the k-th dimension model parameter is convex, indicating that increasing the model parameter may lead to a decrease in the loss function. The value of is negative, indicating that the loss function corresponding to the k-th dimension model parameter is concave, indicating that increasing the parameter value may lead to an increase in the loss function. The larger the absolute value of , the greater the impact of model parameter changes on the loss function, that is, The update direction of the k-th dimension model parameters can be adjusted.

[0152] According to the F-Divergence Lower Bound theorem, F can be calculated by the second-order derivatives corresponding to the model parameters, the update parameters, and the mean update parameters. F-div .

[0153] In some embodiments, considering that the sample distributions of multimedia samples in multiple clients may have overlapping parts, in order to eliminate the influence of the sample ranges of the multimedia samples in the overlapping parts on the sample distribution differences, the sample distribution differences are determined based on the second-order derivatives, update parameters and mean update parameters corresponding to the model parameters, including:

[0154] Subtract the mean update parameter from the update parameter to obtain the update offset parameter;

[0155] The sample distribution difference is determined based on the second-order derivatives corresponding to the model parameters and the updated offset parameters.

[0156] The update offset parameter is the difference between the k-th dimension update parameter in the update vector and the k-th mean update parameter in the mean update vector.

[0157] Among them, u i (k) is the k-th dimension update parameter in the update vector corresponding to the i-th client, i is any client among the n clients, is the mean of the k-th dimension update parameters in the update vector corresponding to all clients. In It can reflect the influence of the k-th dimension update parameter on the loss function corresponding to the model parameter relative to the change of the model parameter.

[0158] As can be seen from the above, the update vector can reflect the change in the distribution range of multimedia samples in the corresponding client. In this way, the update offset parameter (δ k ) can reflect the distribution range of the multimedia samples of the corresponding client, relative to the average distribution range of the multimedia samples in all clients. Specifically, it is the influence of the loss function corresponding to the model parameters, which is multiplied by the updated offset parameters to obtain the influence of the updated offset parameters on the loss function. This can reflect the sample distribution of the multimedia samples used to update the offset parameters when the client uses multimedia samples to iterate the local model.

[0159] It can reflect the influence of the updated offset parameter on the corresponding loss function in the training model, through the F-Divergence Lower Bound theorem The sample distribution of multimedia samples in the kth client can be obtained, and the distribution difference of the sample distribution of multimedia samples in the clean client can be obtained.

[0160] In some embodiments, considering that the server cannot directly obtain multimedia samples in the client, in order to indirectly calculate the sample distribution difference corresponding to the client, the sample distribution difference is determined based on the second-order derivative corresponding to the model parameter and the updated offset parameter, including:

[0161] According to the second-order derivatives corresponding to the model parameters, the contribution of the updated offset parameters to the loss function is calculated;

[0162] All contributions are combined to obtain the sample distribution difference.

[0163] The contribution is the degree of influence that the updated offset parameter can have on the updated model parameter. Contribution = H k *δ k 2 , which indicates the influence of the k-th dimension update offset parameter on the update of the k-th dimension model parameters.

[0164] The sample distribution difference is used to indirectly reflect the distribution range of multimedia samples in the client by updating the contribution of the offset parameter to the loss function.

[0165] Among them, k∈[1,d] is the dimension of the model parameters, the dimension of the model parameters corresponds to the dimension of the updated offset parameters, and k can be any dimension [1,d].

[0166] for example, The value of is positive, indicating that the loss function corresponding to the k-th dimension model parameter is convex, indicating that increasing the model parameter may lead to a decrease in the loss function. The value of is negative, indicating that the loss function corresponding to the k-th dimension model parameter is concave, indicating that increasing the parameter value may lead to an increase in the loss function. In this way, according to the second-order derivative corresponding to the model parameter, the model parameter is updated by updating the offset parameter, and the contribution of the updated offset parameter to the loss function can be obtained.

[0167] In some embodiments, because the backdoor of natural language processing can be applied to low-frequency model parameters, and the detection samples generated by the server do not necessarily contain these low-frequency words or some low-frequency semantics, in order to alleviate the problems caused by data sparsity and improve the performance of the model to be trained when processing low-frequency words and low-frequency semantics, the contribution of the updated offset parameter to the loss function is calculated according to the second-order derivative corresponding to the model offset parameter, including:

[0168] Redistribute the second-order derivatives corresponding to the model parameters to obtain the weights corresponding to the model parameters. The weights are used to adjust the update effect of the model parameters.

[0169] The weight corresponding to the model parameter is multiplied by the square of the updated offset parameter to obtain the contribution of the updated offset parameter to the loss function.

[0170] Among them, the weights corresponding to the model parameters are obtained by redistributing the values ​​of the second-order derivatives corresponding to the model parameters, which can change the learning rate or optimization direction of the model parameters, thereby affecting the training process of the model to be trained.

[0171] The update effect is the effect of adjusting the model parameters by the weights corresponding to the model parameters. For example, the update effect includes but is not limited to adjusting the learning rate, improving the optimization direction, etc.

[0172] The functions of the weights corresponding to the model parameters include but are not limited to:

[0173] 1. Adjust the learning rate: The second-order derivatives corresponding to the model parameters represent the curvature of the model parameters near the current point. A larger second-order derivative indicates a smaller curvature in the direction of the model parameters, and a smaller second-order derivative indicates a larger curvature in the direction of the parameters. By redistributing the second-order derivatives corresponding to the model parameters, the learning rates in different directions can be adjusted. For example, increasing the weights corresponding to larger second-order derivatives can make the parameters learn faster in these directions; increasing the weights corresponding to smaller second-order derivatives can make the parameters learn slower in these directions. This can improve the stability and efficiency of training.

[0174] 2. Improve optimization direction: The second-order derivatives corresponding to the model parameters represent different directions in the model parameter space. By redistributing the second-order derivatives corresponding to the model parameters, the update direction of the model parameters can be adjusted. For example, increasing the weights corresponding to larger second-order derivatives can make the model parameters pay more attention to these directions, thereby improving the model's optimization ability in these directions. This helps to improve the generalization performance and convergence speed of the model.

[0175] Methods for redistributing the second-order derivatives corresponding to model parameters include but are not limited to:

[0176] (1) The inverse of the second-order derivative corresponding to the model parameter: The inverse of the second-order derivative corresponding to the model parameter is used to adjust the learning rate of the model parameter. A classic method is to use Newton's method or conjugate gradient method, in which the inverse of the second-order derivative corresponding to the model parameter is used as a preconditioning matrix to accelerate convergence. In this way, the update step size of each model parameter can be adaptively adjusted during the training process according to the curvature information of the loss function corresponding to the model parameter.

[0177] (2) Random sampling: For large-scale data and complex models, in order to reduce the computational cost of the inverse of the second-order derivatives corresponding to the model parameters, random sampling is used to estimate the inverse of the second-order derivatives corresponding to the model parameters. A portion of the training samples is randomly selected from the training sample set and the inverse of the second-order derivatives corresponding to the model parameters is calculated, which is then applied to the model parameter update of the model to be trained.

[0178] (3) Dynamic adjustment: This method dynamically adjusts the distribution of second-order derivatives corresponding to model parameters during the training process according to specific rules or strategies. For example, the weight of the second-order derivative can be adjusted according to the importance or influence of the model parameters. In this way, the second-order derivatives can be adaptively distributed during the training process according to the actual situation.

[0179] In some embodiments, in order to make the contribution of the model parameters in calculating the corresponding weights relatively balanced and avoid the unbalanced impact caused by the excessive or too small weights of the model parameters, the weights corresponding to the model parameters are obtained by redistributing the second-order derivatives corresponding to the model parameters according to the weights corresponding to the model parameters, including:

[0180] Based on the update parameters of the same dimension in the update vector corresponding to each client, the absolute values ​​of the update parameters are summed to obtain the total absolute value;

[0181] Divide the total absolute value by the number of clients to get the mean update parameter corresponding to the update parameter;

[0182] Based on the correspondence between the update parameter and the model parameter, the redistribution weight of the second-order derivative corresponding to the model parameter is determined by using the mean update parameter corresponding to the update parameter and a preset threshold;

[0183] The redistribution weights of the second-order derivatives corresponding to all model parameters are summed to obtain the total distribution weight;

[0184] The redistributed weight of the second-order derivative corresponding to the model parameter is divided by the total distribution weight to obtain the normalized weight of the second-order derivative corresponding to the model parameter;

[0185] Sum the second-order derivatives corresponding to all model parameters to obtain the total second-order derivative;

[0186] The normalized weight of the second-order derivative corresponding to the model parameter is multiplied by the total second-order derivative to obtain the weight corresponding to the model parameter.

[0187] The total absolute value is the sum of the absolute values ​​of the update parameters of the same dimension in the update vector corresponding to each client. For example, if there are n clients, the update parameter of the kth dimension in the update vector corresponding to each client is obtained, and the absolute values ​​of the n kth dimension update parameters are added together to obtain the total absolute value.

[0188] Where n is the number of clients, i∈[1,n] is the i-th client among n clients, is the absolute value of the model parameter of the kth dimension in the update vector corresponding to the i-th client.

[0189] The mean update parameter is the sum of the absolute values ​​of the update parameters of the same dimension, divided by the number of clients.

[0190]

[0191] The preset threshold is a threshold set to prevent the mean update parameter from being zero. For example, the preset threshold (€) can be a very small positive number, such as € = 10 -8 , the value of € is not limited here.

[0192] The redistribution of weights is used to adjust the second-order derivatives corresponding to the model parameters to adapt to the characteristics and importance of different model parameters and improve the training effect and convergence speed.

[0193] Redistribute weights Among them, s k Reassign the weight of the second-order derivative corresponding to the k-th model parameter in the model to be trained.

[0194] For example, the distribution weight of the second-order derivatives corresponding to the model parameters can be adjusted by considering the magnitude of the gradient vector corresponding to the model parameters. Model parameters with larger gradient magnitudes are assigned more second-order derivatives, while model parameters with smaller gradient magnitudes are assigned smaller second-order derivatives, where the magnitude of the gradient magnitude reflects the slope and rate of change of the model parameters at the current point. A larger gradient magnitude means that the slope of the model parameters near the current point is steeper and the rate of change is faster, while a smaller gradient magnitude means that the slope of the model parameters near the current point is relatively gentle and the rate of change is slower.

[0195] The total allocation weight is the ratio of the redistributed weights of the second-order derivatives corresponding to the model parameters after the redistributed weights of the second-order derivatives corresponding to all model parameters.

[0196] Where E is the model parameter in the vector to be trained, j is the jth model parameter in the vector to be trained, j can be any model parameter in the vector to be trained, s j Reassign the weight of the second-order derivative corresponding to the j-th model parameter in the model to be trained.

[0197] The normalized weight can reflect the importance of the model parameter relative to other model parameters. A higher normalized weight means that the model parameter contributes more in the calculation of the second-order derivative, while a lower normalized weight means that the contribution of the model parameter is relatively small.

[0198]

[0199] The total second-order derivative is the sum of the second-order derivatives corresponding to all model parameters in the model to be trained.

[0200] Among them, E is the model parameter in the vector to be trained.

[0201] The normalized weights of the second-order derivatives corresponding to the model parameters represent the relative importance of each model parameter in calculating the total second-order derivatives. Multiplying the normalized weights of the second-order derivatives corresponding to the model parameters by the total second-order derivatives is equivalent to weighted averaging the total second-order derivatives according to the weights. This allows model parameters with larger normalized weights to contribute more to the total second-order derivatives, while model parameters with smaller normalized weights contribute less to the total second-order derivatives. The purpose of this is to better consider the importance and characteristics of different model parameters when redistributing the second-order derivatives corresponding to the model parameters, thereby affecting the update effect of the optimization algorithm on the model parameters. By reasonably weighting the average second-order derivatives, the effect of the optimization algorithm can be improved, the parameter update and adjustment process can be improved, and the model to be trained can be better optimized.

[0202] Diffusion theory points out the relationship between the second-order derivatives corresponding to model parameters and the updates during training. Based on diffusion theory and the updates on model parameters, the second-order derivatives corresponding to model parameters are derived and redistributed.

[0203] Diffusion theory is used to estimate the second-order derivatives corresponding to the model parameters. When I was very young, Established.

[0204] The redistribution process needs to keep the following two points unchanged:

[0205]

[0206] 150. Determine the target client from the clients based on the sample distribution difference.

[0207] The target clients are the clients corresponding to the smaller sample distribution differences.

[0208] According to the difference in sample distribution, the method of determining the target client from the clients can be:

[0209] (1) A clustering method can be used to determine a smaller sample distribution difference from the sample distribution differences corresponding to multiple clients, thereby obtaining the client corresponding to the smaller sample distribution difference.

[0210] (2) The sample distribution differences corresponding to multiple clients can be sorted by a sorting method to obtain a sample distribution difference sequence, and a smaller sample distribution difference can be selected from the sample distribution difference sequence to obtain the client corresponding to the smaller sample distribution difference.

[0211] (3) By calculating the difference between every two sample distribution differences, a smaller sample distribution difference can be determined from the sample distribution differences corresponding to multiple clients based on the difference, so as to obtain the client corresponding to the smaller sample distribution difference.

[0212] For example, if there are 4 clients, the sample distribution difference corresponding to client 1 is a, the sample distribution difference corresponding to client 2 is b, the sample distribution difference corresponding to client 3 is c, and the sample distribution difference corresponding to client 4 is d. If a, b, and c are close, and d is far away from a, b, and c, then the target clients include the client corresponding to a, the client corresponding to b, and the client corresponding to c.

[0213] In some embodiments, considering that there may be a client with a backdoor implanted in multiple clients, in order to reduce the impact of the client with the backdoor implanted on the training model, the target client is determined from the clients according to the sample distribution difference, including:

[0214] Get the number of clients;

[0215] Determine the target quantity that meets the preset conditions from the quantity;

[0216] According to the sample distribution difference, a target number of target clients is determined from the clients.

[0217] The number is the number of clients participating in updating the initial model vector. For example, the number of clients may be n, and the number of clients is not limited here.

[0218] The preset condition is a condition in the server for screening the number of clients, so as to reduce the influence of clients implanted with backdoors on updating the initial model parameters.

[0219] The target number is the number of clients required to update the initial model vector in the model to be trained.

[0220] For example, the target quantity can be or It can also be or It can also be or Etc. Among them, To round down, To round up.

[0221] For example, through the sample distribution differences corresponding to each client, a target number of target clients with the smallest sample distribution differences are selected from all clients to avoid large sample distribution differences among the target clients, so that the initial model vector in the model to be trained can be updated through the update vectors corresponding to all target clients.

[0222] In some embodiments, in order to reduce the impact of a client with a backdoor implanted on a model to be trained, determining a target number that meets a preset condition from the number includes:

[0223] Determine the quotient of the quantity divided by 2;

[0224] Add 1 to the quotient to get the number of candidates;

[0225] Round down the candidate quantity to get the target quantity.

[0226] Here, the quotient is half of the number of clients.

[0227] The number of candidates is half the number of clients plus 1.

[0228] For example, if n is 11, the quotient is 5.5, the candidate number is 6.5, and the target number is 6 after rounding down.

[0229] In some embodiments, in order to update the model to be trained, an update vector corresponding to the target client is used to update the initial model vector in the model to be trained to obtain a trained model, including:

[0230] Aggregate the update vectors corresponding to the target client to obtain an aggregate vector;

[0231] Merge the aggregated vector with the initial model vector to obtain the model update vector of the model to be trained;

[0232] The model to be trained is updated using the model update vector to obtain a trained model.

[0233] The aggregate vector is a vector obtained by merging the update vectors corresponding to all target clients.

[0234] Aggregation Vector in, is the update vector corresponding to the i-th client among n clients, w i is the weight corresponding to the i-th client, through w i , the target client can be determined from n clients, and the w corresponding to the target client i It can be 1, and the weight corresponding to the client other than the target client can be 0, so that The aggregation vector can be obtained.

[0235] The model update vector is a vector used to update the initial model vector in the model to be trained. When the model update vector updates the model to be trained, the trained model can be obtained.

[0236] The trained model is the model corresponding to the updated initial model parameters in the model to be trained.

[0237] In some embodiments, in order to reduce the impact of a client with a backdoor implanted therein on the model to be trained, a target number of target clients is determined from the clients according to the sample distribution difference, including:

[0238] According to the sample distribution difference, a target number of target clients and other clients except the target clients are determined from the clients, and the sample distribution difference of the target clients is not greater than the sample distribution difference of the other clients;

[0239] The update vector corresponding to the target client is used to update the initial model vector in the model to be trained to obtain the trained model, including:

[0240] Using a first preset weight, weighting the update vector corresponding to the target client to obtain a first weighted vector;

[0241] Using the second preset weight, weighting the update vectors corresponding to other clients to obtain a second weighted vector;

[0242] Merging the first weighted vector and the second weighted vector to obtain a combined weight vector;

[0243] The combined weight vector is used to update the initial model vector in the model to be trained to obtain the trained model.

[0244] Among them, other clients are clients other than the target client among all clients participating in updating the model to be trained, and the sample distribution difference of the target client is no greater than the sample distribution difference of other clients.

[0245] The first preset weight is used to increase the contribution of the update vector corresponding to the target client in updating the initial model vector.

[0246] The first weighted vector is used to increase the contribution made by the target client corresponding update vector when updating the initial model vector.

[0247] The second preset weight is used to reduce the contribution of the update vectors corresponding to other clients in updating the initial model vector.

[0248] The second weighted vector is used to reduce the contribution made by the target client corresponding update vector when updating the initial model vector.

[0249] The combined weight vector is a vector obtained by combining the first weight vector and the second weight vector, and is used to update the initial model vector in the model to be trained.

[0250] Merge weight vector in, is the update vector corresponding to the i-th client among n clients, w i is the weight corresponding to the i-th client. If the i-th client is the target client, then w i is the first preset weight, is the first weighted vector. If the i-th client is another client, then wi is the second preset weight, is the second weight vector.

[0251] Among them, if the first preset weight is 1, the second preset weight can be 0, and the merged weight vector is the vector obtained by merging all the first weighted vectors. If the first preset weight is a value greater than 1, the second preset weight can be a value close to 0, and the merged weight vector is the vector obtained by merging all the first weighted vectors and the second weighted vectors.

[0252] In some embodiments, in order to reduce the influence of the client with the backdoor implanted on the model to be trained when training the model to be trained, after determining the target number of target clients from the clients according to the sample distribution difference, the method further includes:

[0253] Using the update vector corresponding to the target client, update the initial model vector in the model to be trained to obtain the trained model;

[0254] Acquire multimedia content to be identified;

[0255] Use trained models to recognize multimedia content.

[0256] The multimedia content is the content that needs to be recognized by the trained model. For example, if the multimedia sample used by the client to train the local model is text, the multimedia content is text; if the multimedia sample used by the client to train the local model is image, the multimedia content is image; if the multimedia sample used by the client to train the local model is audio and video, the multimedia content is audio and video, and so on.

[0257] In some embodiments, after the update vector corresponding to the target client is used to update the initial model vector in the model to be trained to obtain the trained model, the method further includes:

[0258] The current model parameters of the trained model are sent to the client so that the client can update the local model with the current model parameters.

[0259] For example, if the client is an individual user, the client can use the updated local model to recognize the voice, text, etc. input by the user. If the client is an enterprise, the client can use the updated local model to recognize the voice, text, images, etc. input by the enterprise employees. If the client is a medical institution, the client can use the updated local model to recognize and detect medical images, and so on.

[0260] As can be seen from the above, the embodiments of the present application can send the initial model vector of the model to be trained to the client; obtain the training model vector of the client, the training model vector is the vector obtained by the client updating the initial model vector through multimedia samples; based on the initial model vector and the training model vector, determine the update vector corresponding to each client and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients; determine the sample distribution difference, the sample distribution difference reflects the difference between the multimedia samples corresponding to the update vector and the multimedia samples corresponding to the mean update vector; according to the sample distribution difference, determine the target client from the clients.

[0261] The mean update vector in this scheme is the mean of the update vectors corresponding to all clients. Since the update vector can reflect the distribution range of the multimedia samples of the corresponding client, the mean update vector can reflect the average distribution range of the multimedia samples in all clients. Therefore, the difference between the distribution range of the multimedia samples in the client and the average distribution range of the multimedia samples can be obtained through the update vector and the mean update vector, that is, the sample distribution difference. Through the sample distribution difference, it is easy to screen out the target client that has not been implanted with a backdoor from multiple clients, thereby improving the client screening efficiency.

[0262] The method described in the above embodiment will be further described in detail below.

[0263] In this embodiment, if Figure 2a As shown, the method of the embodiment of the present application will be described in detail by taking the federated model training as an example.

[0264] The specific process of a joint training method is as follows:

[0265] 210. Send the initial model vector of the model to be trained to n clients.

[0266] In some embodiments, sending an initial model vector of a model to be trained to n clients includes:

[0267] If the model to be trained in the server is in the first training, the server initializes the model parameters in the model to be trained, and all model parameters in the model to be trained constitute an initial model vector.

[0268] 220. Obtain a training model vector of the client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples.

[0269] 230. Based on the initial model vector and the training model vector, determine the update vector corresponding to each client and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients.

[0270] In some embodiments, in order to calculate the update vector and the mean update vector, based on the initial model vector and the training model vector, the update vector and the mean update vector corresponding to each client are determined, including:

[0271] Subtract the initial model vector from the client's training model vector to obtain the update vector corresponding to the client;

[0272] Add the update vectors corresponding to all clients to get the total update vector;

[0273] Divide the total update vector by the number of clients to get the mean update vector.

[0274] 240. Determine a sample distribution difference, where the sample distribution difference reflects a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector.

[0275] In some embodiments, the model to be trained includes model parameters and loss functions corresponding to the model parameters, the update vector includes the update parameters, the mean update vector includes the mean update parameters, and the model parameters, the update parameters, and the mean update parameters correspond to each other;

[0276] Determine sample distribution differences, including:

[0277] Generate a test sample and a reference label of the test sample, where the test sample is a multimedia sample for testing the model to be trained;

[0278] Use the model to be trained to perform prediction processing on the test sample to obtain the predicted label of the test sample;

[0279] Based on the predicted label and the reference label, determine the loss value of the loss function corresponding to the model parameter;

[0280] According to the loss value of the loss function, determine the second-order derivative corresponding to the model parameter;

[0281] The sample distribution differences are determined based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters.

[0282] In some embodiments, determining the sample distribution difference based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters includes:

[0283] Subtract the mean update parameter from the update parameter to obtain the update offset parameter;

[0284] Redistribute the second-order derivatives corresponding to the model parameters to obtain the weights corresponding to the model parameters. The weights are used to adjust the update effect of the model parameters.

[0285] The weight corresponding to the model parameter is multiplied by the square of the updated offset parameter to obtain the contribution of the updated offset parameter to the loss function.

[0286] In some embodiments, the second-order derivatives corresponding to the model parameters are redistributed to obtain weights corresponding to the model parameters, including:

[0287] Based on the update parameters of the same dimension in the update vector corresponding to each client, the absolute values ​​of the update parameters are summed to obtain the total absolute value;

[0288] Divide the total absolute value by the number of clients to get the mean update parameter corresponding to the update parameter;

[0289] Based on the correspondence between the update parameter and the model parameter, the redistribution weight of the second-order derivative corresponding to the model parameter is determined by using the mean update parameter corresponding to the update parameter and a preset threshold;

[0290] The redistribution weights of the second-order derivatives corresponding to all model parameters are summed to obtain the total distribution weight;

[0291] The redistributed weight of the second-order derivative corresponding to the model parameter is divided by the total distribution weight to obtain the normalized weight of the second-order derivative corresponding to the model parameter;

[0292] Sum the second-order derivatives corresponding to all model parameters to obtain the total second-order derivative;

[0293] The normalized weight of the second-order derivative corresponding to the model parameter is multiplied by the total second-order derivative to obtain the weight corresponding to the model parameter.

[0294] 250. Determine a target client and other clients except the target client from the clients according to the sample distribution difference, and the sample distribution difference corresponding to the target client is not greater than the sample distribution difference corresponding to the other clients.

[0295] In some embodiments, determining a target client from the clients according to the sample distribution difference includes:

[0296] Get the number of clients;

[0297] Determine the quotient of the quantity divided by 2;

[0298] Add 1 to the quotient to get the number of candidates;

[0299] Round down the candidate quantity to get the target quantity.

[0300] According to the sample distribution difference, a target number of target clients and other clients except the target clients are determined from the clients.

[0301] 260. Using the update vector corresponding to the target client and the update vectors corresponding to other clients, the initial model vector in the model to be trained is updated to obtain a trained model.

[0302] In some embodiments, using the update vector corresponding to the target client and the update vectors corresponding to other clients to update the initial model vector in the model to be trained to obtain the trained model includes:

[0303] Using a first preset weight, weighting the update vector corresponding to the target client to obtain a first weighted vector;

[0304] Using the second preset weight, weighting the update vectors corresponding to other clients to obtain a second weighted vector;

[0305] Merging the first weighted vector and the second weighted vector to obtain a combined weight vector;

[0306] The merged weight vector is used to update the initial model vector in the model to be trained to obtain the trained model. In some embodiments, after the update vector corresponding to the target client and the update vector corresponding to other clients are used to update the initial model vector in the model to be trained to obtain the trained model, the method further includes:

[0307] According to the preset rounds, steps 210-260 are adopted to train the model to be trained, and the model to be trained is the trained model of the previous round.

[0308] For example, the preset round number is t=1, 2, ..., T. When training the model to be trained, the aforementioned steps 210-260 are executed until T times, and the model training is terminated.

[0309] This application aims to derive a second-order inverse of the update parameters corresponding to the client and the model parameters according to the theoretical framework, estimate the F-divergence of the distribution of a certain client data and clean data, and propose an F-divergence index based on this. According to the F-divergence index, suspicious backdoor clients (i.e., some clients with higher F-divergence index) are detected, and the parameter updates of these clients (i.e., other clients) are discarded, and the parameter updates of the remaining clients (target clients) are aggregated.

[0310] This application is applicable to backdoor defense scenarios in general federated learning scenarios, especially backdoor defense scenarios in natural language processing tasks, that is, in order not to expose the client's privacy or due to policy requirements, the server cannot obtain the client's local data, but only the client's trained update vector. The server needs to detect and discard update vectors that may be suspected of containing backdoors, so as to achieve the effect of backdoor defense. Specific application scenarios are as follows:

[0311] Scenario for Federated Learning Backdoor Defense of Cloud Model Upload without Exposing User Privacy: For example, on a mobile phone, a local recommendation system is trained based on the user's habits. The user agrees to join the experience improvement plan and can use the local model trained by the user to improve the company's recommendation system algorithm. However, in order to protect the user's privacy or in accordance with relevant laws and regulations, the user's privacy cannot be uploaded for training. In this case, multiple users are regarded as clients, and the company uses a central server to train the recommendation system. In order to prevent malicious clients from uploading backdoor parameter updates, this solution can be used for Federated Learning Backdoor Defense.

[0312] Federated learning backdoor defense scenarios in specific privacy-sensitive fields: For example, in the medical field, the data itself contains patient personal information or genetic information, which is very sensitive and cannot be handed over to the company for training. At this time, a federated learning framework is built between the company and many hospitals or medical company customers. The company's central server can also use this application to defend against backdoors in updates from hospitals or medical companies that may not be trusted.

[0313] This application improves the backdoor defense performance on multiple models, multiple backdoor attack scenarios and multiple data sets, and reduces the backdoor attack success rate (Attack Success Rate, ASR) in the case of backdoor attacks. Experiments were conducted under the framework of three classic natural language processing models (Model), namely, Convolutional Neural Network (CNN), Recurrent Neural Network (GRU), and Long Short-Term Memory (LSTM), and the following four sentence classification tasks were used: Internet Movie Database (IMDB), News Topic Classification Dataset (AgNews), User Comment Dataset (Amazon-5w) and Sentiment Classification Dataset (SST-2); the attack methods include word-based attacks (phishing attacks that only operate word vectors (Evil Twin Phishing, EP) attacks) and improper sentence attacks (BadWord attacks) that train the entire model), sentence-based attacks (BadSent attacks) and attacks based on grammar changes (Hidden attacks).

[0314] like Figure 2b and Figure 2c As shown, in a variety of model architectures, a variety of attack methods or scenarios, the federated model training method of the present application can surpass other existing backdoor defense algorithms, including the median algorithm (Median), a method for detecting malicious nodes in a distributed system (FoolsGold), a mechanism for model aggregation in federated learning (Robust Federated Aggregation, RFA), a robust training method for federated learning (Certified Robust Federated Learning, CRFL), a method for detecting and filtering out malicious nodes that produce erroneous results in a distributed system (Residual), a method for detecting and filtering out malicious nodes in distributed learning (Krum), a method for enhanced detection and filtering out malicious nodes in distributed learning (M-Krum), a federated learning aggregation algorithm for resisting malicious nodes (Bulyan), and a method for detecting and filtering out multiple malicious nodes (Dim-Krum) algorithm. That is, the present application can obtain a lower ASR when training a federated model, and this conclusion is consistent across different models, data sets, and attack methods. Among them Figure 2c Only some representative defense algorithms are selected.

[0315] in, Figure 2bMetric in the table is the metric standard, including ACC (Accuracy, accuracy on a clean dataset) and ASR. "FedAvg" is a model aggregation algorithm for federated learning. "Non-discarding Aggregations" means that no data is discarded during the aggregation process. "Discarding Aggregations" refers to the strategy or method of discarding part of the data during the aggregation process. Fed-FA: ​​Federated F-Divergence-Based Aggregation, federated aggregation based on sample distribution difference (F-divergence), ACC: Accuracy, accuracy (on a clean dataset).

[0316] in, Figure 2c The Dataset in it is the dataset, and Attack is the attack.

[0317] As can be seen from the above, when training the model to be trained, the present application tries not to be affected by the update vector containing the backdoor, thereby reducing the success rate of the backdoor attack on the trained model after training. At the same time, the performance degradation of the model on clean data caused by the backdoor defense algorithm is minimized, thereby improving the accuracy of the trained model after training.

[0318] In order to better implement the above method, the embodiment of the present application also provides a client selection device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0319] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the client selection device being specifically integrated in the server as an example.

[0320] For example, Figure 3 As shown, the client selection device may include a sending unit 310, an acquiring unit 320, a vector determining unit 330, a difference determining unit 340, and a selecting unit 350, as follows:

[0321] (a) Sending unit 310.

[0322] The sending unit 310 is used to send the initial model vector of the model to be trained to the client.

[0323] (ii) Acquisition unit 320.

[0324] The acquisition unit 320 is used to acquire a training model vector of the client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples.

[0325] (iii) Vector determination unit 330.

[0326] The vector determination unit 330 is used to determine the update vector corresponding to each client and the mean update vector based on the initial model vector and the training model vector. The update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients.

[0327] In some embodiments, based on the initial model vector and the training model vector, determining the update vector corresponding to each client and the mean update vector includes:

[0328] Subtract the initial model vector from the client's training model vector to obtain the update vector corresponding to the client;

[0329] Add the update vectors corresponding to all clients to get the total update vector;

[0330] Divide the total update vector by the number of clients to get the mean update vector.

[0331] (iv) Difference determination unit 340.

[0332] The difference determination unit 340 is used to determine the sample distribution difference, where the sample distribution difference reflects the difference between the multimedia sample corresponding to the update vector and the multimedia sample corresponding to the mean update vector.

[0333] In some embodiments, the model to be trained includes model parameters and loss functions corresponding to the model parameters, the update vector includes the update parameters, the mean update vector includes the mean update parameters, and the model parameters, the update parameters and the mean update parameters correspond to each other;

[0334] Determine sample distribution differences, including:

[0335] Generate a test sample and a reference label of the test sample, where the test sample is a multimedia sample for testing the model to be trained;

[0336] Use the model to be trained to perform prediction processing on the test sample to obtain the predicted label of the test sample;

[0337] Based on the predicted label and the reference label, determine the loss value of the loss function corresponding to the model parameter;

[0338] According to the loss value of the loss function, determine the second-order derivative corresponding to the model parameter;

[0339] The sample distribution differences are determined based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters.

[0340] In some embodiments, determining the sample distribution difference based on the second-order derivatives, update parameters, and mean update parameters corresponding to the model parameters includes:

[0341] Subtract the mean update parameter from the update parameter to obtain the update offset parameter;

[0342] The sample distribution difference is determined based on the second-order derivatives corresponding to the model parameters and the updated offset parameters.

[0343] In some embodiments, determining the sample distribution difference based on the second-order derivatives corresponding to the model parameters and the updated offset parameters includes:

[0344] According to the second-order derivatives corresponding to the model parameters, the contribution of the updated offset parameters to the loss function is calculated;

[0345] All contributions are combined to obtain the sample distribution difference.

[0346] In some embodiments, calculating the contribution of the updated offset parameter to the loss function according to the second-order derivative corresponding to the model offset parameter includes:

[0347] Redistribute the second-order derivatives corresponding to the model parameters to obtain the weights corresponding to the model parameters. The weights are used to adjust the update effect of the model parameters.

[0348] The weight corresponding to the model parameter is multiplied by the square of the updated offset parameter to obtain the contribution of the updated offset parameter to the loss function.

[0349] In some embodiments, the second-order derivatives corresponding to the model parameters are redistributed to obtain weights corresponding to the model parameters, including:

[0350] Based on the update parameters of the same dimension in the update vector corresponding to each client, the absolute values ​​of the update parameters are summed to obtain a total absolute value;

[0351] Divide the total absolute value by the number of clients to get the mean update parameter corresponding to the update parameter;

[0352] Based on the correspondence between the update parameter and the model parameter, the redistribution weight of the second-order derivative corresponding to the model parameter is determined by using the mean update parameter corresponding to the update parameter and a preset threshold;

[0353] The redistribution weights of the second-order derivatives corresponding to all model parameters are summed to obtain the total distribution weight;

[0354] The redistributed weight of the second-order derivative corresponding to the model parameter is divided by the total distribution weight to obtain the normalized weight of the second-order derivative corresponding to the model parameter;

[0355] Sum the second-order derivatives corresponding to all model parameters to obtain the total second-order derivative;

[0356] The normalized weight of the second-order derivative corresponding to the model parameter is multiplied by the total second-order derivative to obtain the weight corresponding to the model parameter.

[0357] (V) Selection unit 350.

[0358] The selection unit 350 is used to determine a target client from the clients according to the sample distribution difference.

[0359] In some embodiments, determining a target client from the clients according to the sample distribution difference includes:

[0360] Get the number of clients;

[0361] Determine the target quantity that meets the preset conditions from the quantity;

[0362] According to the sample distribution difference, a target number of target clients is determined from the clients.

[0363] In some embodiments, after determining a target number of target clients from the clients according to the sample distribution difference, the method further includes:

[0364] Using the update vector corresponding to the target client, update the initial model vector in the model to be trained to obtain the trained model;

[0365] Acquire multimedia content to be identified;

[0366] Use trained models to recognize multimedia content.

[0367] In some embodiments, determining a target number of target clients from the clients according to the sample distribution difference includes:

[0368] According to the sample distribution difference, a target number of target clients and other clients except the target clients are determined from the clients, and the sample distribution difference of the target clients is not greater than the sample distribution difference of the other clients;

[0369] The update vector corresponding to the target client is used to update the initial model vector in the model to be trained to obtain the trained model, including:

[0370] Using a first preset weight, weighting the update vector corresponding to the target client to obtain a first weighted vector;

[0371] Using the second preset weight, weighting the update vectors corresponding to other clients to obtain a second weighted vector;

[0372] Merging the first weighted vector and the second weighted vector to obtain a combined weight vector;

[0373] The combined weight vector is used to update the initial model vector in the model to be trained to obtain the trained model.

[0374] In some embodiments, using the update vector corresponding to the target client to update the initial model vector in the model to be trained to obtain the trained model includes:

[0375] Aggregate the update vectors corresponding to the target client to obtain an aggregate vector;

[0376] Merge the aggregated vector with the initial model vector to obtain the model update vector of the model to be trained;

[0377] The model to be trained is updated using the model update vector to obtain a trained model.

[0378] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0379] As can be seen from the above, the client selection device of this embodiment sends the initial model vector of the model to be trained to the client by the sending unit; the acquisition unit acquires the training model vector of the client, and the training model vector is a vector obtained by the client updating the initial model vector through multimedia samples; the vector determination unit determines the update vector corresponding to each client and the mean update vector based on the initial model vector and the training model vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients; the difference determination unit determines the sample distribution difference, and the sample distribution difference reflects the difference between the multimedia sample corresponding to the update vector and the multimedia sample corresponding to the mean update vector; the selection unit determines the target client from the clients according to the sample distribution difference.

[0380] Therefore, the embodiment of the present application facilitates the screening of clients that are not implanted with backdoors, and can improve the efficiency of client screening.

[0381] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.

[0382] In some embodiments, the client selection device may also be integrated into multiple electronic devices. For example, the client selection device may be integrated into multiple servers, and the client selection method of the present application may be implemented by multiple servers.

[0383] In this embodiment, the electronic device of this embodiment is a server as an example for detailed description, for example, Figure 4 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present application, specifically:

[0384] The server may include one or more processing core processors 410, one or more computer-readable storage media memories 420, a power supply 430, an input module 440, and a communication module 450. Those skilled in the art will appreciate that Figure 4 The server structure shown in the figure does not constitute a limitation on the server, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0385] The processor 410 is the control center of the server, and uses various interfaces and lines to connect various parts of the entire server. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 420, and calling data stored in the memory 420. In some embodiments, the processor 410 may include one or more processing cores; in some embodiments, the processor 410 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 410.

[0386] The memory 420 can be used to store software programs and modules. The processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0387] The server also includes a power supply 430 for supplying power to various components. In some embodiments, the power supply 430 can be logically connected to the processor 410 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 430 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0388] The server may further include an input module 440, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0389] The server may further include a communication module 450. In some embodiments, the communication module 450 may include a wireless module. The server may perform short-range wireless transmission through the wireless module of the communication module 450, thereby providing wireless broadband Internet access to the user. For example, the communication module 450 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0390] Although not shown, the server may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 410 in the server will load the executable files corresponding to the processes of one or more application programs into the memory 420 according to the following instructions, and the processor 410 will run the application programs stored in the memory 420, thereby realizing various functions, as follows:

[0391] Send the initial model vector of the model to be trained to the client;

[0392] Obtaining a training model vector of the client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples;

[0393] Based on the initial model vector and the training model vector, determine the update vector corresponding to each client and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients;

[0394] determining a sample distribution difference reflecting a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector;

[0395] According to the sample distribution difference, the target client is determined from the clients.

[0396] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0397] As can be seen from the above, it is easy to filter out target clients that are not implanted with backdoors from multiple clients, thereby improving the efficiency of client screening.

[0398] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0399] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the client selection methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0400] Send the initial model vector of the model to be trained to the client;

[0401] Obtaining a training model vector of the client, where the training model vector is a vector obtained by the client updating an initial model vector through multimedia samples;

[0402] Based on the initial model vector and the training model vector, determine the update vector corresponding to each client and the mean update vector, the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all clients;

[0403] determining a sample distribution difference reflecting a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector;

[0404] According to the sample distribution difference, the target client is determined from the clients.

[0405] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0406] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including a computer program / instruction, the computer program / instruction being stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the electronic device executes the method provided in various optional implementations of the client selection aspect provided in the above embodiments.

[0407] Since the instructions stored in the storage medium can execute the steps in any client selection method provided in the embodiments of the present application, the beneficial effects that can be achieved by any client selection method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0408] The above is a detailed introduction to a client selection method, device, electronic device, storage medium and program product provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A client selection method, characterized in that: include: Send the initial model vector of the model to be trained to the client; Acquire a training model vector of the client, where the training model vector is a vector obtained by the client updating the initial model vector through multimedia samples; Based on the initial model vector and the training model vector, determine an update vector corresponding to each of the clients and a mean update vector, wherein the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all the clients; determining a sample distribution difference reflecting a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector; According to the sample distribution difference, a target client is determined from the clients.

2. The client selection method according to claim 1, characterized in that: The determining, based on the initial model vector and the training model vector, an update vector corresponding to each of the clients and a mean update vector, comprises: Subtract the initial model vector from the training model vector of the client to obtain an update vector corresponding to the client; Adding the update vectors corresponding to all the clients to obtain a total update vector; The total update vector is divided by the number of the clients to obtain a mean update vector.

3. The client selection method according to claim 1, characterized in that: Determining a target client from the clients according to the sample distribution difference includes: Obtain the number of the clients; determining a target quantity satisfying a preset condition from the quantity; According to the sample distribution difference, the target number of target clients is determined from the clients.

4. The client selection method according to claim 3, characterized in that: After determining the target number of target clients from the clients according to the sample distribution difference, the method further includes: Using the update vector corresponding to the target client, updating the initial model vector in the model to be trained to obtain a trained model; Acquire multimedia content to be identified; The multimedia content is recognized using the trained model.

5. The client selection method according to claim 4, characterized in that: The step of determining the target number of target clients from the clients according to the sample distribution difference includes: Determine the target number of target clients and other clients except the target clients from the clients according to the sample distribution difference, wherein the sample distribution difference of the target clients is not greater than the sample distribution difference of the other clients; The adopting the update vector corresponding to the target client to update the initial model vector in the model to be trained to obtain the trained model includes: Using a first preset weight, weighting the update vector corresponding to the target client to obtain a first weighted vector; Using a second preset weight, weighting the update vectors corresponding to the other clients to obtain a second weighted vector; Merging the first weighted vector and the second weighted vector to obtain a combined weight vector; The combined weight vector is used to update the initial model vector in the model to be trained to obtain a trained model.

6. The client selection method according to claim 4, characterized in that: The adopting the update vector corresponding to the target client to update the initial model vector in the model to be trained to obtain the trained model includes: Aggregating the update vectors corresponding to the target client to obtain an aggregated vector; Merging the aggregated vector with the initial model vector to obtain a model update vector of the model to be trained; The model to be trained is updated using the model update vector to obtain the trained model.

7. The client selection method according to claim 1, characterized in that: The model to be trained includes model parameters and loss functions corresponding to the model parameters, the update vector includes update parameters, the mean update vector includes mean update parameters, and the model parameters, the update parameters and the mean update parameters correspond to each other; The determining of the sample distribution difference comprises: Generate a detection sample and a reference label of the detection sample, wherein the detection sample is a multimedia sample for detecting the model to be trained; Using the model to be trained, performing prediction processing on the test sample to obtain a prediction label of the test sample; Determining a loss value of a loss function corresponding to the model parameter based on the predicted label and the reference label; Determining the second-order derivative corresponding to the model parameter according to the loss value of the loss function; The sample distribution difference is determined based on the second-order derivatives corresponding to the model parameters, the update parameters and the mean update parameters.

8. The client selection method according to claim 7, characterized in that: The determining of the sample distribution difference based on the second-order derivative corresponding to the model parameter, the update parameter and the mean update parameter comprises: Subtract the mean update parameter from the update parameter to obtain an update offset parameter; Based on the second-order derivatives corresponding to the model parameters and the updated offset parameters, a sample distribution difference is determined.

9. The client selection method according to claim 8, characterized in that: The determining of the sample distribution difference based on the second-order derivative corresponding to the model parameter and the updated offset parameter comprises: Calculating the contribution of the updated offset parameter to the loss function according to the second-order derivatives corresponding to the model parameters; All the contributions are combined to obtain the sample distribution difference.

10. The client selection method according to claim 9, characterized in that: The calculating, according to the second-order derivative corresponding to the model offset parameter, the contribution of the updated offset parameter to the loss function comprises: Redistributing the second-order derivatives corresponding to the model parameters to obtain weights corresponding to the model parameters, wherein the weights are used to adjust the update effect of the model parameters; The weight corresponding to the model parameter is multiplied by the square of the updated offset parameter to obtain the contribution of the updated offset parameter to the loss function.

11. The client selection method according to claim 10, characterized in that: The redistribution of the second-order derivatives corresponding to the model parameters to obtain weights corresponding to the model parameters includes: Based on the update parameters of the same dimension in the update vector corresponding to each of the clients, the absolute values ​​of the update parameters are summed to obtain a total absolute value; Dividing the total absolute value by the number of the clients to obtain a mean update parameter corresponding to the update parameter; Based on the correspondence between the update parameter and the model parameter, a redistribution weight of the second-order derivative corresponding to the model parameter is determined by using a mean update parameter corresponding to the update parameter and a preset threshold; summing the redistribution weights of the second-order derivatives corresponding to all the model parameters to obtain a total distribution weight; Dividing the redistributed weight of the second-order derivative corresponding to the model parameter by the total allocated weight to obtain a normalized weight of the second-order derivative corresponding to the model parameter; Summing the second-order derivatives corresponding to all the model parameters to obtain a total second-order derivative; The normalized weight of the second-order derivative corresponding to the model parameter is multiplied by the total second-order derivative to obtain the weight corresponding to the model parameter.

12. A client selection device, characterized in that: include: A sending unit, used for sending an initial model vector of a model to be trained to a client; An acquisition unit, configured to acquire a training model vector of the client, wherein the training model vector is a vector obtained by the client updating the initial model vector through multimedia samples; A vector determination unit, configured to determine, based on the initial model vector and the training model vector, an update vector corresponding to each of the clients and a mean update vector, wherein the update vector is used to reflect the distribution range of the multimedia samples of the corresponding client, and the mean update vector is used to reflect the average distribution range of the multimedia samples of all the clients; a difference determination unit, configured to determine a sample distribution difference, wherein the sample distribution difference reflects a difference between a multimedia sample corresponding to the update vector and a multimedia sample corresponding to the mean update vector; A selection unit is used to determine a target client from the clients according to the sample distribution difference.

13. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the client selection method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the client selection method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps in the client selection method according to any one of claims 1 to 11 are implemented.

Images