Efficient task inference method, device and equipment based on federated learning model

Abstract

The application provides an efficient task inference method and device based on a federal learning model, and relates to the technical field of federal learning. The method is applied to a central processor and comprises the following steps: obtaining to-be-inferred data corresponding to a to-be-inferred task; inputting the to-be-inferred data into a target global model to obtain an inference result corresponding to the to-be-inferred data output by the target global model; the target global model is an efficient global model obtained by training a federal learning model based on low-dimensional feature data, the low-dimensional feature data is generated based on a key feature set and a key sample set, the key feature set is determined based on initial dynamic sparse coefficients corresponding to each dimension feature in a sample, and the key sample set is determined based on initial loss weights of the initial model for each sample. The application realizes systematic and collaborative improvement of data privacy protection, communication cost reduction and model convergence acceleration in a vertical federal learning scenario.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to an efficient task reasoning method, apparatus, and device based on a federated learning model. Background Technology

[0002] Federated learning, as a distributed machine learning paradigm, enables multi-party collaborative modeling while maintaining data locality, demonstrating significant value in scenarios such as cross-institutional fraud prevention in finance and multi-center joint diagnosis in medicine. However, with the increasing complexity of application scenarios and the expansion of data scale, existing technologies face problems in actual deployment, such as excessive communication load, limited overall performance improvement, and a vicious cycle of decreased model accuracy and increased communication rounds.

[0003] To address the aforementioned issues, existing optimization schemes only employ single-stage or specific scenario optimizations (such as optimizing only gradient compression, device partitioning, or wireless resource allocation) to improve model accuracy and communication efficiency. In complex and ever-changing vertical federated learning scenarios, they cannot achieve a systematic and synergistic improvement in communication overhead reduction, model convergence acceleration, and privacy protection strength during efficient task inference based on federated learning models. Summary of the Invention

[0004] This invention provides an efficient task inference method, apparatus, and device based on a federated learning model, which addresses the shortcomings of existing technologies in complex and ever-changing vertical federated learning scenarios, which cannot achieve a systematic and synergistic improvement in communication overhead reduction, model convergence acceleration, and privacy protection strength. It achieves a systematic and synergistic improvement in data privacy protection, communication overhead reduction, and model convergence acceleration in vertical federated learning scenarios.

[0005] This invention provides an efficient task inference method based on a federated learning model, applied to a central processing unit, comprising the following steps:

[0006] Obtain the data to be reasoned for the task to be reasoned;

[0007] The data to be inferred is input into the target global model to obtain the inference result corresponding to the data to be inferred output by the target global model. The target global model is an efficient global model trained on a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample.

[0008] According to the present invention, an efficient task inference method based on a federated learning model is provided. The target global model is trained based on the following steps: determining the key feature set of each sample based on the initial dynamic sparse coefficients corresponding to the features of each dimension in multiple samples; determining the key sample set from the multiple samples based on the initial weights and Gaussian noise of the initial model for each sample; sending the key feature set and the key sample set to at least two clients and receiving the target model parameters sent by each client; wherein, the clients are used to perform low-dimensional processing on the key feature set and the key sample set based on a federated contrastive learning loss function to obtain low-dimensional feature data; training the initial model based on the low-dimensional feature data to obtain a first model; compressing the model parameters of the first model based on lightweight knowledge distillation to obtain the target model parameters; determining the first global model based on the state information of each client and the corresponding target model parameters; optimizing the first global model based on the loss change and the communication change of the first global model to obtain the target global model.

[0009] According to the present invention, an efficient task inference method based on a federated learning model is provided. The method for determining the key feature set of each sample based on the initial dynamic sparse coefficients corresponding to each dimension of features in multiple samples includes: determining the gradient corresponding to each dimension of features of the sample; determining the gradient importance score corresponding to each dimension of features based on the initial dynamic sparse coefficients and the gradients; determining the second dynamic sparse coefficients corresponding to each dimension of features based on the gradient importance scores and the initial dynamic sparse coefficients; and determining the key feature set of each sample based on the second dynamic sparse coefficients.

[0010] According to the present invention, an efficient task inference method based on a federated learning model is provided, wherein determining a key sample set from the plurality of samples based on the initial weights and Gaussian noise of each sample in the initial model includes: determining the privacy protection weights corresponding to each sample based on the initial weights and Gaussian noise of each sample; and determining the key sample set from the plurality of samples based on the privacy protection weights.

[0011] According to the present invention, an efficient task reasoning method based on a federated learning model is provided. The step of determining a first global model based on the state information of each client and the corresponding target model parameters includes: determining the policy distribution of the client selection probability based on the state information of each client; determining key clients from among the clients based on the policy distribution; and aggregating the target model parameters corresponding to the key clients to obtain the first global model.

[0012] According to the present invention, an efficient task inference method based on a federated learning model is provided. The method optimizes the first global model based on the change in loss and the change in communication of the first global model to obtain a target global model. The method includes: substituting the change in loss and the change in communication into a multi-objective function to obtain first parameters to be optimized; the multi-objective function is constructed based on the change in loss and the change in communication of the first global model; continuously adjusting the model parameters of the first global model based on the first parameters to be optimized until the first parameters to be optimized corresponding to the model parameters reach a convergence condition to obtain a first candidate global model; adjusting the weight coefficients corresponding to the change in loss and the change in communication in the multi-objective function respectively to obtain an updated multi-objective function; performing a next round of iterative training on the first candidate global model based on the second dynamic sparsity coefficient and the second weights of each sample to obtain a second global model; optimizing the second global model based on the updated multi-objective function to obtain a second candidate global model; and determining the target global model based on the first candidate global model and the second candidate global model.

[0013] According to the present invention, an efficient task inference method based on a federated learning model is provided. The step of determining the target global model based on a first candidate global model and a second candidate global model includes: performing multiple iterations of training on the second candidate global model based on the dynamic sparsity coefficients and sample loss weights corresponding to the second candidate global model to obtain multiple third candidate global models; constructing a Pareto optimization objective function based on the loss change and communication change corresponding to each of the candidate global models, wherein all candidate global models include the first candidate global model, the second candidate global model, and the third candidate global models; and determining the target global model from all candidate global models based on the Pareto optimization objective function.

[0014] This invention also provides an efficient task inference device based on a federated learning model, comprising the following modules:

[0015] The acquisition module is used to acquire the data to be reasoned for the task to be reasoned.

[0016] The inference module is used to input the data to be inferred into the target global model and obtain the inference result corresponding to the data to be inferred output by the target global model. The target global model is an efficient global model trained on a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to the features of each dimension in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the efficient task inference method based on the federated learning model as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an efficient task inference method based on a federated learning model as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an efficient task inference method based on a federated learning model as described above.

[0020] The present invention provides an efficient task inference method, apparatus, and device based on a federated learning model. This method obtains inference results by inferring from data to be inferred using a trained target global model. The target global model is an efficient global model trained on a federated learning model using low-dimensional feature data. This low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise applied to each sample by the initial model. Thus, through a phased multi-objective optimization strategy, including selecting the feature set and sample set, reducing model parameters, and optimizing the global model using communication and loss variations, the target global model is obtained. This achieves a systematic and synergistic improvement in data privacy protection, communication overhead reduction, and model convergence acceleration in a vertical federated learning scenario, further enhancing the accuracy, communication efficiency, and privacy of efficient task inference based on a federated learning model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the efficient task reasoning method based on a federated learning model provided by the present invention.

[0023] Figure 2 This is a flowchart illustrating the training method for the target global model provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of the efficient task inference device based on the federated learning model provided by the present invention.

[0025] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The existing technologies have the following problems: (1) Traditional federated learning methods generate high communication loads during full data exchange and frequent global model aggregation, especially in scenarios with heterogeneous clients (different computing capabilities) and non-independent and identically distributed data, where communication overhead grows exponentially, causing communication efficiency bottlenecks; (2) Existing optimization schemes mostly focus on a single stage (such as gradient compression or client selection), lacking a collaborative optimization mechanism for the entire process of data intersection, local pre-training, and global training, and the fragmentation of multi-stage optimization leads to limited overall performance improvement; (3) When introducing differential privacy protection, existing data screening methods often suffer from excessive noise injection, resulting in the loss of key features, causing a vicious cycle of decreased model accuracy and increased communication rounds, leading to a trade-off between privacy and performance. However, existing technologies cannot collaboratively optimize the above problems, thus failing to achieve a systematic synergistic improvement in communication overhead reduction, model convergence acceleration, and privacy protection strength.

[0028] To address the aforementioned problems, this invention provides an efficient task inference method based on a federated learning model. Through a phased, multi-objective optimization strategy, including selecting feature sets and sample sets, reducing model parameters, and optimizing the global model using communication and loss variations, this method achieves a systematic and synergistic improvement in data privacy protection, communication overhead reduction, and model convergence acceleration in vertical federated learning scenarios.

[0029] The following is combined Figures 1-2 This invention describes an efficient task inference method based on a federated learning model. This efficient task inference method based on a federated learning model is applicable to any task, such as image recognition and image classification. The execution subject of this method can be an efficient task inference system based on a federated learning model, or an efficient task inference method based on a federated learning model set in the system. The efficient task inference device based on a federated learning model can be implemented by software, hardware, or a combination of both.

[0030] Figure 1 This is one of the flowcharts illustrating the efficient task inference method based on a federated learning model provided by this invention, such as... Figure 1 As shown, this method is applied to a central server and includes the following:

[0031] Step 101: Obtain the data to be reasoned for the task to be reasoned.

[0032] Here, the inference task includes, but is not limited to, classification, recognition, and object detection, and the inference data can be text or images.

[0033] Step 102: Input the data to be reasoned into the target global model to obtain the reasoning result corresponding to the data to be reasoned output by the target global model.

[0034] The target global model is obtained by training a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample.

[0035] Here, the reasoning result can be a classification result, a recognition result, an object detection result, etc.

[0036] In this embodiment of the invention, the training target global model is used to infer the reasoning data to obtain the reasoning result. The target global model is an efficient global model trained on a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to the features of each dimension in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample. Thus, the target global model is obtained through a phased multi-objective optimization strategy, such as screening the feature set and sample set, reducing model parameters, and optimizing the global model using communication changes and loss changes. This achieves a systematic and synergistic improvement in data privacy protection, communication overhead reduction, and model convergence acceleration in the vertical federated learning scenario, further improving the accuracy, communication efficiency, and privacy of efficient task reasoning based on the federated learning model.

[0037] Furthermore, the target global model is trained based on the following steps: determining the key feature set of each sample based on the initial dynamic sparse coefficients corresponding to the features of each dimension in multiple samples; determining the key sample set from the multiple samples based on the initial loss weights and Gaussian noise of the initial model for each sample; sending the key feature set and the key sample set to at least two clients and receiving the target model parameters sent by each client; wherein, the client is used to perform low-dimensional processing on the key feature set and the key sample set based on the federated contrastive learning loss function to obtain low-dimensional feature data; training the initial model based on the low-dimensional feature data to obtain a first model; compressing the model parameters of the first model based on lightweight knowledge distillation to obtain the target model parameters; determining the first global model based on the state information of each client and the corresponding target model parameters; optimizing the first global model based on the loss change and the communication change of the first global model to obtain the target global model.

[0038] Here, the dynamic sparsity coefficient refers to the sparsity constraint parameter that is adaptively adjusted for each dimension of features during model training or inference, and is used to adjust the weight of feature importance. A sample may have multiple dimensions, and each dimension has a corresponding dynamic sparsity coefficient.

[0039] Here, the initial dynamic sparsity coefficients can be the dynamic sparsity coefficients of the features in the previous round of training iterations.

[0040] Here, the method for determining the key feature set can be any suitable method. For example, the initial dynamic sparse coefficients can be input into the feature model to directly output the key feature set; or the key feature set can be calculated using a calculation formula.

[0041] Optionally, determining the key feature set of each sample based on the initial dynamic sparse coefficients corresponding to each dimension of features in multiple samples includes: determining the gradient corresponding to each dimension of features of the sample; determining the gradient importance score corresponding to each dimension of features based on the initial dynamic sparse coefficients corresponding to each dimension of features and the gradients corresponding to each dimension of features; determining the second dynamic sparse coefficients corresponding to each dimension of features based on the gradient importance scores corresponding to each dimension of features and the initial dynamic sparse coefficients corresponding to each dimension of features; and determining the key feature set of each sample based on the second dynamic sparse coefficients corresponding to each dimension of features.

[0042] Here, the features corresponding to each dimension reflect the impact of that feature on the current model performance.

[0043] Here, the second dynamic sparsity coefficient refers to the dynamic sparsity coefficient of the feature in the current round of training iterations.

[0044] Here, the gradient importance score corresponding to each feature dimension can be the product of the initial dynamic sparse coefficient and the gradient, or it can be the weighted product of the initial dynamic sparse coefficient and the gradient. The second dynamic sparse coefficient can be the sum of the gradient importance score and the initial dynamic sparse coefficient, or it can be obtained by weighted summation.

[0045] For example, the gradient importance score is calculated using the following formula (1):

[0046] (1)

[0047] in, This represents the gradient importance score of feature j in the t-th round of training. This represents the dynamic sparsity coefficient of feature j in round t, which is the initial dynamic sparsity coefficient mentioned above. This represents the total number of samples currently being processed. Greater than 1, Represents the model loss function For the sample The Gradient of dimensional features Indicates a label, Indicates model parameters, Indicates sample The Dimensional features, This represents the model loss function.

[0048] Furthermore, the second dynamic sparsity coefficient is calculated using the following formula (2):

[0049] (2)

[0050] in, This represents the second dynamic sparsity coefficient, i.e., the dynamic sparsity coefficient of feature j in the (t+1)th round. This represents the sparse coefficient update step size (i.e., the learning rate), which controls the sparse coefficients. The adjustment range, This represents the total number of iterations, indicating the total number of optimization iterations performed during training. This indicates the indicator variable used when traversing all rounds.

[0051] Here, from all feature sets Determining the key feature set of the sample The method may include: after obtaining the second dynamic sparsity coefficients corresponding to each dimension of features, applying the dynamic sparsity coefficients of all current features... Sort in descending order, with the first... The features corresponding to the dynamic sparse coefficients are determined as the key feature set, or the features corresponding to the second dynamic sparse coefficients that are greater than the sparse coefficient threshold are determined as the key feature set.

[0052] In this embodiment of the invention, the key feature set is selected from all feature sets by using the second dynamic sparse coefficients corresponding to the features of each dimension in the current iteration, which reduces redundant information transmission and improves communication efficiency.

[0053] Here, the initial weight is used to represent the proportion of the loss value corresponding to a sample to the sum of the loss values ​​of all samples. The larger the proportion, the larger the initial weight, and vice versa.

[0054] Here, the initial model can be the global model obtained from the previous iteration, or it can be a global model trained in the database.

[0055] Here, the method for determining the key sample set can be either to substitute the initial weights and Gaussian noise into the calculation formula to obtain the key sample set, or to input the initial weights and Gaussian noise of each sample into the sample model to directly output the key sample set.

[0056] Optionally, determining the key sample set from the plurality of samples based on the initial weights and Gaussian noise of each sample using the initial model includes: determining the privacy protection weights corresponding to each sample based on the initial weights and Gaussian noise of each sample; and determining the key sample set from the plurality of samples based on the privacy protection weights.

[0057] Here, the privacy protection weight can be the sum of the initial weight and Gaussian noise, or it can be a weighted sum of the initial weight and Gaussian noise.

[0058] Here, Gaussian noise is actually a Gaussian distribution used to generate noise and ensure privacy protection.

[0059] For example, privacy protection weight The calculation is as follows: (3)

[0060] (3)

[0061] in, Indicates the first The initial weights of each sample, Indicates Gaussian noise. σ represents variance, which is dynamically adjusted based on the privacy budget ϵ.

[0062] Here, the key sample set is determined. The method may include: after obtaining the privacy protection weights corresponding to each sample, sorting the privacy protection weights of all current samples in descending order, and then... The samples corresponding to the privacy protection weight are determined as the key sample set, or the samples that are greater than the privacy protection weight threshold are determined as the key sample set.

[0063] In this embodiment of the invention, key sample sets are screened by adding Gaussian noise perturbation to the weight of each sample. Perturbing the sample weights achieves differential privacy protection, reduces redundant information transmission, and improves communication efficiency.

[0064] The target model parameters are obtained by the client compressing the first model, which is trained by the client based on low-dimensional feature data generated by the client based on the key feature set and the key sample set.

[0065] Specifically, after receiving the key feature set and key sample set, the client uses a federated contrastive learning loss function to transform them into a low-dimensional data representation. This low-dimensional representation is then used to train the original model to obtain a first model. Lightweight knowledge distillation is then used to compress the parameters of the first model to obtain the target model parameters. Here, the compression ratio of the first model can be any suitable ratio, such as 50%, 60%, etc.

[0066] For example, federated learning loss function The following formula (4):

[0067] (4)

[0068] in, data Low-dimensional feature data is a low-dimensional representation. The batch size represents the number of samples used in each training iteration. Representing data The augmented view is achieved through the encoder. The generated low-dimensional feature representations, such as augmented views, include the original data. Perform data augmentation (such as cropping, rotating, adding noise, etc.) to generate augmented samples. ;Will Input encoder The corresponding feature representation is obtained. , : Represents the temperature parameter, used to adjust the smoothness of the Softmax function.

[0069] Here, lightweight knowledge distillation actually transforms the teacher model, i.e. the first model, into the student model, i.e. the target model. The first model has fewer parameters than the target model.

[0070] Here, the loss function of knowledge distillation The following formula (5):

[0071] (5)

[0072] in, The divergence, denoted as Kullback-Leibler divergence, measures the difference between two probability distributions. This represents the distribution relationship in the KL divergence (Kullback-Leibler, KL). Indicates the sample Low-dimensional feature data, Indicates the enhanced sample The low-dimensional feature data, wherein the standard form of the KL divergence is as follows (6):

[0073] (6)

[0074] in, It is the reference distribution (usually the output distribution of the teacher model, i.e., the first model); It is the target distribution (usually the output distribution of the student model, i.e., the target model). This indicates that the divergence of P relative to Q is calculated with Q as the reference.

[0075] Here, the client's status information can include computing power, bandwidth, etc.

[0076] Here, the selection strategy can be determined based on the state information and the corresponding target model parameters, and the first global model can be determined based on the selection strategy; alternatively, the client's state information and the corresponding target model parameters can be input into the calculation model to directly output the first global model.

[0077] Optionally, determining the first global model based on the state information of each client and the corresponding target model parameters includes: determining the policy distribution of the client selection probability based on the state information of each client; identifying key clients from among the clients based on the policy distribution; and aggregating the target model parameters corresponding to the key clients to obtain the first global model.

[0078] Here, the policy distribution of the client's choice probability is given by the following formula (7):

[0079] (7)

[0080] in, This represents the probability of the client's choice. This indicates the client's status information in round t. This represents the mean. Let represent the covariance matrix of round t. This indicates a normal distribution.

[0081] Specifically, clients with high probabilities are selected as the key client set. Then, the target model parameters corresponding to each client in the key client set are aggregated to obtain the first global model.

[0082] It should be noted that after each round of training, the client selection strategy is updated based on the client's latest state information to optimize client selection in future rounds.

[0083] In this embodiment of the invention, the client selection strategy is dynamically adjusted by using the client's state information, which optimizes the priority of the client's participation in training, further reduces the amount of communication, and accelerates model convergence.

[0084] Here, the change in loss is the rate of change between the model loss value of the previous iteration and the model loss value of the current iteration, i.e.

[0085] For example, the loss of the t-round model The calculation is as follows (8):

[0086] (8)

[0087] in, This represents the total number of samples. Indicates a label, Indicates model parameters, This represents the loss function.

[0088] Here, the communication change is the rate of change between the initial communication volume and the communication volume in the current iteration, i.e.

[0089] Here, the communication volume of the current iteration is calculated using the following formula (9):

[0090] (9)

[0091] in, This indicates that the sparsity parameters are dynamically adjusted. Indicates the quantization parameter. This represents the total number of samples.

[0092] In this embodiment of the invention, key feature sets for each sample are determined based on initial dynamic sparse coefficients corresponding to features of each dimension in multiple samples; key sample sets are determined from multiple samples based on initial weights and Gaussian noise for each sample; the key feature sets and key sample sets are sent to at least two clients, and target model parameters are received from each client; the target model parameters are obtained by the client compressing a first model, which is trained by the client using low-dimensional feature data generated by the client based on the key feature sets and key sample sets; a first global model is determined based on the state information of each client and the corresponding target model parameters; the first global model is optimized based on the loss change and communication change of the first global model to obtain the target global model. Thus, through a phased multi-objective optimization strategy, including selecting feature sets and sample sets, reducing model parameters, and optimizing the global model using communication and loss changes, a systematic and synergistic improvement in data privacy protection, communication overhead reduction, and model convergence acceleration in a vertical federated learning scenario is achieved.

[0093] Optionally, optimizing the first global model based on the change in loss and the change in communication of the first global model to obtain a target global model includes: substituting the change in loss and the change in communication into a multi-objective function to obtain a first parameter to be optimized; the multi-objective function is constructed based on the change in loss and the change in communication of the first global model; continuously adjusting the model parameters of the first global model based on the first parameter to be optimized until the first parameter to be optimized corresponding to the model parameters reaches the convergence condition to obtain a first candidate global model; adjusting the weight coefficients corresponding to the change in loss and the weight coefficients corresponding to the change in communication in the multi-objective function respectively to obtain an updated multi-objective function; performing the next round of iterative training on the first candidate global model based on the second dynamic sparsity coefficient and the second weight of each sample to obtain a second global model; optimizing the second global model based on the updated multi-objective function to obtain a second candidate global model; and determining the target global model based on the first candidate global model and the second candidate global model.

[0094] Here, multi-objective function The expression is as follows: (10)

[0095] (10)

[0096] in, The weighting coefficients represent the changes in loss. Weighting coefficients representing changes in communication.

[0097] Here, the first parameter to be optimized is the result obtained by substituting the change in loss and the change in communication into formula (10).

[0098] It should be noted that when the first parameter to be optimized is relatively small, the first parameter to be optimized is improved by continuously adjusting the model parameters of the first global model. The first parameter to be optimized corresponding to the model parameters after each adjustment is calculated until the first parameter to be optimized reaches its maximum value and converges. Then, the model corresponding to the largest first parameter to be optimized is determined as the first candidate global model.

[0099] Here, the rules for updating the weight coefficients corresponding to the loss change and the communication change in the multi-objective function are as follows: formulas (11) and (12):

[0100] (11)

[0101] (12)

[0102] in, This indicates that the parameters are being adjusted. Indicates the number of iterations. This represents the weighting coefficient corresponding to the change in loss. The weight coefficients corresponding to the change in loss are represented here. When calculating the weight coefficients for the next iteration (t+1), t in formulas (11) and (12) is changed to t+1, that is, the weight coefficients corresponding to the change in loss and the weight coefficients corresponding to the change in communication in the multi-objective function are adjusted.

[0103] Here, based on the second dynamic sparsity coefficient and the second weight of each sample, the process of training the first candidate global model in the next round to obtain the second global model can refer to the process of determining the first global model described above, and will not be elaborated here.

[0104] Here, the target global model can be determined directly from the first candidate global model and the second candidate global model; or the second candidate global model can be further trained to obtain multiple candidate global models, and the target global model can be determined from the multiple candidate global models.

[0105] In this embodiment of the invention, the model parameters of the first global model are continuously adjusted by the change in loss and the change in communication, thereby balancing the model loss and the reduction in communication volume and achieving a balance between communication efficiency and model performance.

[0106] For example, determining the target global model based on the first candidate global model and the second candidate global model includes: performing multiple iterations of training on the second candidate global model based on the dynamic sparsity coefficients and sample loss weights corresponding to the second candidate global model to obtain multiple third candidate global models; constructing a Pareto optimization objective function based on the loss change and communication change corresponding to each of the candidate global models, wherein all candidate global models include the first candidate global model, the second candidate global model, and the third candidate global model; and determining the target global model from all candidate global models based on the Pareto optimization objective function.

[0107] Here, the process of iteratively training the second candidate global model to obtain multiple third candidate global models based on the dynamic sparsity coefficients and sample loss weights corresponding to the second candidate global model can refer to the process of determining the first candidate global model described above, and will not be elaborated here.

[0108] Here, the Pareto optimization objective function is as follows (13):

[0109] (13)

[0110] The Fast Elitist Non-Dominated Sorting Genetic Algorithm II (NSGA-II) is used to generate the Pareto front solution set corresponding to the Pareto optimization objective function. The Pareto front solution set is a set of all candidate global models sorted based on communication volume and accuracy. Appropriate communication volume and accuracy are selected from the Pareto front solution set, and the candidate global models corresponding to the communication volume and accuracy are determined to determine the target global model.

[0111] In another embodiment, after determining the target global model, evaluation metrics for the target global model can be determined. These metrics include the loss change rate and the communication change rate. The loss change rate is the rate of change between the loss value of the target global model and the loss value of the global model obtained in the previous iteration. The communication change rate is the rate of change between the communication volume of the target global model and the communication volume of the global model obtained in the previous iteration. Thus, the performance and communication efficiency of the target global model are evaluated using these evaluation metrics.

[0112] In this embodiment of the invention, the Pareto optimization objective function is constructed by the loss change and communication change corresponding to each candidate global model. The optimal global model is determined among multiple candidate global models, balancing model loss and communication reduction, thus achieving a balance between communication efficiency and model performance.

[0113] Figure 2 This is a flowchart illustrating the training method for the target global model provided by the present invention, as shown below. Figure 2 As shown, this includes a central server 310 and a client 320. The specific steps are as follows:

[0114] Step 301: The central server receives all sample sets and all feature sets uploaded by the client.

[0115] Step 302: The central server determines the key feature set of each sample based on the initial dynamic sparsity coefficients corresponding to the features of each dimension in multiple samples.

[0116] Step 303: The central server determines the key sample set from multiple samples based on the initial weights of each sample and Gaussian noise.

[0117] Step 304: The central server sends the key feature set and key sample set to at least two clients.

[0118] Step 305: The client uses federated contrastive learning and lightweight knowledge distillation to train the original model based on the key feature set and key sample set to obtain the target model parameters.

[0119] Step 306: The client sends the target model parameters to the central server.

[0120] Step 307: The central server determines the first global model based on the status information of each client and the corresponding target model parameters.

[0121] Step 308: The central server optimizes the first global model based on the change in loss and the change in communication of the first global model to obtain the first candidate global model.

[0122] Step 309: The central server adjusts the weight coefficients corresponding to the change in loss and the change in communication in the multi-objective function respectively to obtain the updated multi-objective function;

[0123] Step 310: The central server performs the next round of iterative training on the first candidate global model based on the second dynamic sparsity coefficient and the second weight of each sample to obtain the second global model;

[0124] Step 311: The central server determines the target global model based on the first candidate global model and the second candidate global model, and determines the evaluation index of the target global model.

[0125] The evaluation metrics include changes in loss and changes in communication.

[0126] In this embodiment of the invention, a phased multi-objective optimization strategy is adopted to achieve end-to-end collaborative optimization of sample and feature selection, client-side local pre-training, and server-side aggregation of target model parameters, significantly reducing communication overhead and accelerating model convergence. Dynamic gradient sparsity feature selection and differential privacy-preserving sample selection methods ensure data privacy while reducing redundant information transmission and improving communication efficiency. A combination of federated contrastive learning and lightweight knowledge distillation generates low-dimensional feature representations during local pre-training, reducing communication volume during subsequent formal training. Dynamically adjusting the client selection strategy optimizes the priority of client participation in training, further reducing communication volume and accelerating model convergence. A multi-dimensional evaluation index system achieves a dynamic balance between communication efficiency and model performance, ensuring the efficient operation of the federated learning system in various scenarios.

[0127] Figure 3 This is a schematic diagram of the structure of the efficient task inference device based on the federated learning model provided by the present invention, as shown below. Figure 3 As shown, the efficient task inference device 300 based on the federated learning model includes:

[0128] The acquisition module 310 is used to acquire the data to be reasoned corresponding to the task to be reasoned.

[0129] The inference module 320 is used to input the data to be inferred into the target global model and obtain the inference result corresponding to the data to be inferred output by the target global model. The target global model is an efficient global model trained on a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to the features of each dimension in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample.

[0130] In another embodiment, the efficient task inference device 300 based on a federated learning model further includes a training module, specifically configured to: determine a key feature set for each sample based on initial dynamic sparse coefficients corresponding to features of each dimension in multiple samples; determine a key sample set from the multiple samples based on the initial loss weights and Gaussian noise of the initial model for each sample; send the key feature set and the key sample set to at least two clients, and receive target model parameters sent by each client; wherein the clients are configured to perform low-dimensional processing on the key feature set and the key sample set based on a federated contrastive learning loss function to obtain low-dimensional feature data; train the initial model based on the low-dimensional feature data to obtain a first model; compress the model parameters of the first model based on lightweight knowledge distillation to obtain the target model parameters; determine a first global model based on the state information of each client and the corresponding target model parameters; and optimize the first global model based on the loss change and communication change of the first global model to obtain a target global model.

[0131] In another embodiment, the training module is further specifically configured to: determine the gradient corresponding to each dimension feature of the sample; determine the gradient importance score corresponding to each dimension feature based on the initial dynamic sparsity coefficients corresponding to each dimension feature and the gradients corresponding to each dimension feature; determine the second dynamic sparsity coefficient corresponding to each dimension feature based on the gradient importance score corresponding to each dimension feature and the initial dynamic sparsity coefficients corresponding to each dimension feature; and determine the key feature set of each sample based on the second dynamic sparsity coefficients corresponding to each dimension feature.

[0132] In another embodiment, the training module is further specifically configured to: determine the privacy-preserving weights corresponding to each of the samples based on the initial weights and Gaussian noise of each sample; and determine a key sample set from the plurality of samples based on the privacy-preserving weights.

[0133] In another embodiment, the training module is further configured to: determine the policy distribution of the client selection probability based on the state information of each client; determine key clients from among the clients based on the policy distribution; and aggregate the target model parameters corresponding to the key clients to obtain the first global model.

[0134] In another embodiment, the training module is further specifically configured to: substitute the loss change and the communication change into a multi-objective function to obtain a first parameter to be optimized; the multi-objective function is constructed based on the loss change and the communication change of the first global model; based on the first parameter to be optimized, continuously adjust the model parameters of the first global model until the first parameter to be optimized corresponding to the model parameters reaches the convergence condition to obtain a first candidate global model; adjust the weight coefficients corresponding to the loss change and the weight coefficients corresponding to the communication change in the multi-objective function respectively to obtain an updated multi-objective function; perform the next round of iterative training on the first candidate global model based on the second dynamic sparsity coefficient and the second weight of each sample to obtain a second global model; optimize the second global model based on the updated multi-objective function to obtain a second candidate global model; and determine the target global model based on the first candidate global model and the second candidate global model.

[0135] In another embodiment, the training module is further specifically configured to: perform multiple iterations of training on the second candidate global model based on the dynamic sparsity coefficients and sample loss weights corresponding to the second candidate global model, to obtain multiple third candidate global models; construct a Pareto optimization objective function based on the loss change and communication change corresponding to each of the candidate global models, wherein all candidate global models include the first candidate global model, the second candidate global model, and the third candidate global model; and determine the target global model from all candidate global models based on the Pareto optimization objective function.

[0136] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an efficient task inference method based on a federated learning model. This method includes: acquiring the data to be inferred corresponding to the task to be inferred; inputting the data to be inferred into a target global model to obtain the inference result corresponding to the data to be inferred output by the target global model; the target global model is obtained by training a federated learning model based on low-dimensional feature data, the low-dimensional feature data being an efficient global model generated based on a key feature set and a key sample set, the key feature set being determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples, and the key sample set being determined based on the initial loss weights and Gaussian noise of each sample in the initial model.

[0137] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the efficient task inference method based on the federated learning model provided by the above methods. The method includes: acquiring the data to be inferred corresponding to the task to be inferred; inputting the data to be inferred into a target global model to obtain the inference result corresponding to the data to be inferred output by the target global model; the target global model is obtained by training the federated learning model based on low-dimensional feature data, the low-dimensional feature data is an efficient global model generated based on a key feature set and a key sample set, the key feature set is determined based on the initial dynamic sparse coefficients corresponding to the features of each dimension in the samples, and the key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample.

[0139] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an efficient task inference method based on a federated learning model provided by the above methods. This method includes: acquiring data to be inferred corresponding to a task to be inferred; inputting the data to be inferred into a target global model to obtain an inference result corresponding to the data to be inferred output by the target global model; wherein the target global model is obtained by training a federated learning model based on low-dimensional feature data, the low-dimensional feature data being an efficient global model generated based on a key feature set and a key sample set, the key feature set being determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples, and the key sample set being determined based on the initial loss weights and Gaussian noise applied to each sample by the initial model.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient task reasoning method based on a federated learning model, characterized in that, Applied to central processing units, including: Obtain the reasoning data corresponding to the reasoning task; wherein, the reasoning task includes at least one of the following: classification, recognition, and object detection; the reasoning data includes text or images; The data to be inferred is input into the target global model to obtain the inference result corresponding to the data to be inferred output by the target global model. The target global model is an efficient global model trained on a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise of the initial model for each sample. The inference result includes at least one of the following: classification result, recognition result, and object detection result. The target global model is trained based on the following steps: Determine the gradients corresponding to each dimension of the features of the sample; Based on the initial dynamic sparsity coefficients and gradients corresponding to each dimension feature, the gradient importance score corresponding to each dimension feature is determined. Based on the gradient importance scores corresponding to each dimension feature and the initial dynamic sparsity coefficients corresponding to each dimension feature, the second dynamic sparsity coefficients corresponding to each dimension feature are determined. Based on the second dynamic sparsity coefficients corresponding to the features of each dimension, the key feature set of each sample is determined. Based on the initial loss weights and Gaussian noise of each sample in the initial model, a key sample set is determined from the multiple samples; The key feature set and the key sample set are sent to at least two clients, and the target model parameters sent by each client are received. The clients are configured to perform low-dimensional processing on the key feature set and the key sample set based on a federated contrastive learning loss function to obtain low-dimensional feature data; train an initial model based on the low-dimensional feature data to obtain a first model; and compress the model parameters of the first model based on lightweight knowledge distillation to obtain the target model parameters. Based on the state information of each client and the corresponding target model parameters, a first global model is determined. Based on the change in loss and the change in communication of the first global model, the first global model is optimized to obtain the target global model.

2. The efficient task inference method based on a federated learning model according to claim 1, characterized in that, The determination of the key sample set from the plurality of samples based on the initial weights and Gaussian noise of the initial model includes: Based on the initial weights and Gaussian noise of each sample, the privacy protection weights corresponding to each sample are determined. Based on the privacy protection weights, a key sample set is determined from the plurality of samples.

3. The efficient task inference method based on a federated learning model according to claim 1, characterized in that, The step of determining the first global model based on the state information of each client and the corresponding target model parameters includes: Based on the state information of each client, determine the strategy distribution of the client's selection probability; Based on the strategy distribution, key clients are identified from among the clients. The first global model is obtained by aggregating the target model parameters corresponding to the key clients.

4. The efficient task inference method based on a federated learning model according to claim 1, characterized in that, The optimization of the first global model based on the loss change and communication change of the first global model to obtain the target global model includes: Substituting the change in loss and the change in communication into the multi-objective function yields the first parameter to be optimized; the multi-objective function is constructed based on the change in loss and the change in communication of the first global model. Based on the first parameter to be optimized, the model parameters of the first global model are continuously adjusted until the first parameter to be optimized corresponding to the model parameters reaches the convergence condition, thereby obtaining the first candidate global model; The weight coefficients corresponding to the change in loss and the change in communication in the multi-objective function are adjusted respectively to obtain the updated multi-objective function; Based on the second dynamic sparsity coefficient and the second weight of each sample, the first candidate global model is trained in the next round of iteration to obtain the second global model; Based on the updated multi-objective function, the second global model is optimized to obtain a second candidate global model; The target global model is determined based on the first candidate global model and the second candidate global model.

5. The efficient task inference method based on a federated learning model according to claim 4, characterized in that, The step of determining the target global model based on the first candidate global model and the second candidate global model includes: Based on the dynamic sparsity coefficients and sample loss weights corresponding to the second candidate global model, the second candidate global model is trained iteratively multiple times to obtain multiple third candidate global models. Based on the loss and communication changes corresponding to each of the candidate global models, a Pareto optimization objective function is constructed. All candidate global models include the first candidate global model, the second candidate global model, and the third candidate global model. Based on the Pareto optimization objective function, the target global model is determined from all candidate global models.

6. A high-efficiency task inference apparatus based on a federated learning model, which applies the efficient task inference method based on a federated learning model as described in any one of claims 1 to 5, characterized in that, include: The acquisition module is used to acquire the data to be reasoned for the task to be reasoned. The reasoning module is used to input the data to be reasoned into the target global model and obtain the reasoning result corresponding to the data to be reasoned output by the target global model. The target global model is an efficient global model trained on a federated learning model based on low-dimensional feature data. The low-dimensional feature data is generated based on a key feature set and a key sample set. The key feature set is determined based on the initial dynamic sparse coefficients corresponding to each dimension of features in the samples. The key sample set is determined based on the initial loss weights and Gaussian noise applied to each sample by the initial model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the efficient task reasoning method based on the federated learning model as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the efficient task reasoning method based on the federated learning model as described in any one of claims 1 to 5.

Images