Federal learning acceleration method and device and nonvolatile storage medium

By executing the encryption algorithm in parallel in federated learning, and using multiple engines to process modular exponents and modular multiplication operations, the problems of high computational complexity and slow training speed of federated learning on edge devices are solved, and the encryption process is realized Improve efficiency and applicability.

CN118917387APending Publication Date: 2024-11-08STATE GRID BEIJING ELECTRIC POWER CO +2

Patent Information

Application Number
CN202411384744.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing federated learning acceleration methods are not universal, resulting in high computational complexity and slow training speed on edge devices.

Method used

By determining multiple steps that can be parallelized in the encryption algorithm and executing them in parallel, these steps are performed separately using multiple engines to generate encryption results of the target data, specifically including splitting the encryption step as basic operation steps such as modulo exponentiation operation and Modular multiplication and calculations are performed in Montgomery space format to improve efficiency.

Benefits of technology

It shortens the execution time of the encryption algorithm, improves the applicability and overall efficiency of the acceleration method of federated learning, and solves the problem of low universality of federated learning acceleration method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917387A_ABST
    Figure CN118917387A_ABST
Patent Text Reader

Abstract

The invention discloses a federal learning acceleration method and device and a nonvolatile storage medium. The method comprises the following steps: receiving target data generated in a training stage of a federated learning-based model training method, and an encryption algorithm identifier adopted for encrypting the target data; determining a processing step of encrypting the target data according to the identifier of the encryption algorithm; in the processing step, determining a plurality of steps which can be executed in parallel; respectively distributing the plurality of steps which can be executed in parallel to the engines for parallel execution, so that the plurality of steps which can be executed in parallel are respectively executed by the corresponding engines to obtain respective execution results of the plurality of engines; and generating an encryption result of the target data according to respective execution results of the plurality of engines. According to the invention, the technical problem of low universality of an acceleration method for federal learning in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of distributed machine learning, and in particular to a method and device for accelerating federated learning, and a non-volatile storage medium. Background Art

[0002] With the continuous expansion and technological progress of power systems, especially driven by smart grids and power Internet of Things, the operation and management of power systems are increasingly dependent on efficient and secure data processing capabilities. Power systems are increasingly dependent on cross-domain data flow. With the convenience of data sharing and utilization, it also brings severe challenges to data privacy protection. Especially when sensitive data is shared between multiple power system entities, how to efficiently use this data without leaking data source information has become an urgent problem to be solved.

[0003] Federated learning, as an emerging distributed machine learning method, can achieve cross-device collaborative learning while protecting data privacy. However, federated learning requires a large number of encryption operations during the training process, which often leads to high computational complexity and slow training speed on edge devices. Existing federated learning acceleration methods for encryption processes mainly include hardware acceleration, algorithm optimization, and communication optimization, but they generally have the disadvantage of low universality.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, device and non-volatile storage medium for accelerating federated learning, so as to at least solve the technical problem that the acceleration method for federated learning in the related art is not universal.

[0006] According to one aspect of an embodiment of the present invention, there is provided a method for accelerating federated learning, comprising: receiving target data generated in a training phase of a model training method based on federated learning, and an identifier of an encryption algorithm used to encrypt the target data; determining, according to the identifier of the encryption algorithm, a processing step for encrypting the target data; among the processing steps, determining a plurality of steps that can be executed in parallel; respectively allocating the plurality of steps that can be executed in parallel to engines for parallel execution, so that the plurality of steps that can be executed in parallel are respectively executed by corresponding engines, and respective execution results of the plurality of engines are obtained; and generating an encryption result of the target data according to the respective execution results of the plurality of engines.

[0007] Optionally, in the processing steps, multiple steps that can be executed in parallel are determined, including: determining an encryption step of an encryption algorithm; splitting the encryption step into basic operation steps, wherein the basic operation steps include modular exponentiation operation steps and modular multiplication operation steps; based on the basic operation steps, determining multiple steps that can be executed in parallel.

[0008] Optionally, multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines to obtain execution results of each of the multiple engines, including: assigning basic operation steps of each of the multiple steps to corresponding engines for parallel execution to obtain execution results of each of the multiple engines.

[0009] Optionally, the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines, including: determining the amount of calculation required for each of the multiple steps; with the goal of minimizing the difference in the amount of calculation between the multiple engines, the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines.

[0010] Optionally, the modular exponentiation step is performed in the following manner: converting the input data of the modular exponentiation from an initial format to a Montgomery space format; calculating the input data in the Montgomery space format to obtain a calculation result; converting the calculation result from the Montgomery space format to the initial format to obtain a modular exponentiation result of the input data.

[0011] Optionally, the multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, and also include: determining the engines corresponding to the multiple steps that can be executed in parallel; sending the multiple steps that can be executed in parallel to a data distributor, and the data distributor distributes the multiple steps that can be executed in parallel to the corresponding engines, wherein the data distributor includes a multi-layer distributor, and the multi-layer distributor distributes data in a tree structure.

[0012] According to another aspect of an embodiment of the present invention, there is also provided an acceleration device for federated learning, including: a receiving module, used to receive target data generated in a training phase of a model training method based on federated learning, and an encryption algorithm identifier used to encrypt the target data; a first determination module, used to determine the processing steps for encrypting the target data according to the identifier of the encryption algorithm; a second determination module, used to determine, in the processing steps, a plurality of steps that can be executed in parallel; an allocation module, used to allocate the plurality of steps that can be executed in parallel to engines for parallel execution, so that the plurality of steps that can be executed in parallel are respectively executed by corresponding engines, and execution results of the plurality of engines are obtained; and a generation module, used to generate an encryption result of the target data according to the execution results of the plurality of engines.

[0013] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium including a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned federated learning acceleration methods.

[0014] According to another aspect of an embodiment of the present invention, a computer device is provided, including a processor, and the processor is used to run a program, wherein any one of the above-mentioned federated learning acceleration methods is executed when the program is running.

[0015] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which implements any one of the above-mentioned federated learning acceleration methods when executed by a processor.

[0016] In an embodiment of the present invention, a method of determining multiple steps that can be executed in parallel in an encryption algorithm and executing the multiple steps that can be executed in parallel is adopted. The target data generated in the training phase of the model training method based on federated learning and the encryption algorithm identifier used to encrypt the target data are received; according to the identifier of the encryption algorithm, the processing steps for encrypting the target data are determined; in the processing steps, multiple steps that can be executed in parallel are determined; the multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines to obtain execution results of multiple engines; according to the execution results of multiple engines, the encryption results of the target data are generated, thereby achieving the purpose of shortening the execution time of the encryption algorithm. As long as the encryption algorithm is used for encryption in federated learning, the method provided by the present invention can be used for acceleration, thereby achieving the technical effect of improving the applicability of the acceleration method of federated learning, and further solving the technical problem of low universality of the acceleration method for federated learning in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing an acceleration method for federated learning is shown;

[0019] Figure 2 is a flowchart of a method for accelerating federated learning provided according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of an acceleration architecture of federated learning provided according to an optional embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of data flow scheduling provided according to an optional embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of hierarchical data distribution provided according to an optional embodiment of the present invention;

[0023] Figure 6 It is a structural block diagram of a federated learning acceleration device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to an embodiment of the present invention, a method embodiment for accelerating federated learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing an acceleration method for federated learning. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the acceleration method of federated learning in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the acceleration method of federated learning of the above-mentioned application program is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0031] Figure 2 is a flow chart of a method for accelerating federated learning according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0032] Step S201, receiving target data generated in the training phase of the model training method based on federated learning, and an encryption algorithm identifier used to encrypt the target data.

[0033] In this step, in federated learning, each participating device (such as IoT devices, etc.) generates local model training data. In order to protect data privacy, federated learning needs to use encryption technology during data exchange. The encryption algorithm identifier is the name or code used to indicate the encryption algorithm.

[0034] Step S202: Determine the processing steps for encrypting the target data according to the identifier of the encryption algorithm.

[0035] In this step, since different encryption algorithms may have different processing flows, it is necessary to determine the specific steps for encrypting the target data according to the encryption algorithm used.

[0036] Step S203, in the processing step, determining multiple steps that can be executed in parallel.

[0037] In this step, in order to improve the computing efficiency, the encryption processing steps may be analyzed to determine multiple steps that can be performed simultaneously (multiple steps that can be executed in parallel) to achieve parallel processing.

[0038] Step S204, assigning the multiple steps that can be executed in parallel to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines, and obtaining execution results of the multiple engines.

[0039] In this step, an engine is a hardware or software unit that performs computing tasks. Steps that can be executed in parallel are assigned to different engines to achieve simultaneous execution. Each engine executes the steps assigned to it and obtains the corresponding results, ultimately generating encrypted data.

[0040] Step S205: Generate encryption results of target data according to the execution results of each of the multiple engines.

[0041] Through the above steps, the purpose of shortening the execution time of the encryption algorithm can be achieved. As long as the encryption algorithm is used for encryption in federated learning, the method provided by the present invention can be used for acceleration, thereby achieving the technical effect of improving the applicability of the acceleration method of federated learning, and thus solving the technical problem that the acceleration method for federated learning in the related art is not universal. The present invention improves the efficiency of encryption operations in federated learning by analyzing the processing steps of the encryption algorithm and executing them in parallel, thereby improving the acceleration effect of the entire federated learning process and having good universality.

[0042] As an optional embodiment, in the processing steps, multiple steps that can be executed in parallel are determined, including: determining an encryption step of an encryption algorithm; splitting the encryption step into basic operation steps, wherein the basic operation steps include modular exponentiation operation steps and modular multiplication operation steps; based on the basic operation steps, determining multiple steps that can be executed in parallel.

[0043] Optionally, in the process of encrypting data, it is necessary to find out the steps that can be performed simultaneously to achieve parallel processing, thereby improving the efficiency of the encryption process. First, the encryption algorithm used can be analyzed, and the steps that need to be executed when the algorithm performs encryption processing can be determined, and the specific steps include initialization, key generation, data encryption, etc. In order to facilitate the processing of computer hardware, the above encryption steps can be refined into basic operation steps. The present invention splits the steps of the encryption algorithm into modular exponentiation steps and modular multiplication steps, wherein modular exponentiation is a common mathematical operation, which is usually used in the encryption and decryption process in the public key cryptography system; modular multiplication is also a basic mathematical operation, which is used to calculate the product of two numbers and take the result modulo a certain modulus.

[0044] It should be noted that after analyzing the common application scenarios of federated learning, nine cryptographic operations that appear frequently in federated learning can be sorted out, mainly including the Paillier semi-homomorphic encryption algorithm, the RSA encryption algorithm, etc. These operations are the focus of federated learning acceleration. However, these operations have different operation processes, which brings great challenges to the design of a unified hardware acceleration circuit. For hardware-based acceleration devices such as DPU, once the circuit logic is burned or the chip is solidified, the function of the circuit is difficult to change further. Therefore, in order to support nine different operations, a simple design idea is to divide the system into nine independent parts and build a computing circuit for one of the operations in each part. However, due to the limited resources on the chip, dividing the resources will result in serious insufficient performance of each part. Therefore, the present invention further analyzes these cryptographic operations and finds that these nine cryptographic operations have common operator parts: modular exponentiation and modular multiplication, that is, these nine cryptographic operations can be combined with modular exponentiation and / or modular multiplication, combined with some simple operations. In addition, more than 95% of the computing time of these nine cryptographic operations is consumed in modular exponentiation and / or modular multiplication operations.

[0045] After identifying the basic operation steps, multiple steps that can be executed simultaneously can be found. These steps may not interfere with each other during execution, or their execution results will not affect each other, so parallel processing can be achieved. In this way, the original encryption process can be decomposed into multiple basic operation steps that can be executed in parallel, thereby improving the efficiency of the encryption process.

[0046] Figure 3 is a schematic diagram of an acceleration architecture of federated learning provided according to an optional embodiment of the present invention, such as Figure 3As shown, the DPU can be connected to the edge IoT agent through a communication interface (PCIe third-generation interface). The edge IoT agent is equipped with DPU driver software, which uses the driver to offload federated learning encryption operations to the federated learning acceleration component in the DPU. The component is equipped with the federated learning acceleration method provided by the present invention, performs data flow scheduling, splits the encryption steps into basic operation steps, and sends them to the engine for calculation, thereby improving the efficiency of federated learning. The present invention uses limited programmable resources to accelerate the basic operations that have the greatest impact on performance, namely modular exponentiation and modular multiplication.

[0047] As an optional embodiment, multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines, and the execution results of the multiple engines are obtained, including: the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution, and the execution results of the multiple engines are obtained.

[0048] Optionally, after splitting the multiple steps that can be executed in parallel into basic operation steps, the basic operation steps can be assigned to engines for execution, that is, each basic operation step will be assigned to one or more engines, so that the basic operation steps can be executed in parallel. In this way, the encryption process can be parallelized, thereby improving the efficiency of encryption operations in federated learning.

[0049] As an optional embodiment, the basic operation steps of each of the multiple steps are respectively assigned to the corresponding engines for parallel execution to obtain the execution results of each of the multiple engines, including: determining the amount of calculation required for each of the multiple steps; with the goal of minimizing the difference in the amount of calculation between the multiple engines, the basic operation steps of each of the multiple steps are respectively assigned to the corresponding engines for parallel execution to obtain the execution results of each of the multiple engines.

[0050] Optionally, before allocating the basic operation steps to be performed to each engine, the computational complexity of each basic operation step, i.e., the amount of operations required to complete the step, can be evaluated. Modular exponentiation and modular multiplication operations require different execution times and resource requirements. When allocating steps, the goal is to make the amount of operations that each engine needs to process as balanced as possible, so as to avoid some engines completing their tasks too early and being idle while other engines are still processing heavier tasks, thereby improving the overall computational efficiency.

[0051] Taking the matrix multiplication operation as an example, it first performs the multiplication of the ciphertext and the plaintext (requiring modular exponentiation), and then the addition of the ciphertext (requiring modular multiplication). Considering the modular exponentiation operation, the exponent e is a plaintext, which usually has a bit width of 64 bits. The number of modular multiplications required may vary depending on the specific value of the plaintext, from 64 to 127 times. On average, about 96 modular multiplications need to be performed. Therefore, the throughput of modular exponentiation should be about 1 / 96 of modular multiplication. Based on this, the ratio of engines running in different modes can be adjusted to balance the throughput of modular exponentiation and modular multiplication. In this way, hardware resources can be efficiently utilized and no engine will be idle. In this way, computing resources can be used more efficiently, waiting time can be reduced, and the overall efficiency of encryption operations in federated learning can be improved.

[0052] Figure 4 is a schematic diagram of data flow scheduling provided according to an optional embodiment of the present invention, such as Figure 4 As shown, the DPU on-chip controller is used to dynamically determine the active data path (logical rather than physical path) and the content configuration of the engine slot. Each engine slot is equipped with a data segmentation and merging module for data distribution and result aggregation. The logical allocation of the engine is achieved by activating specific physical lines. Specifically, a Paillier encryption operator can be constructed. Paillier encryption follows the formula: By Assigned to the modular multiplication engine (E1) calculation , and then Assigned to the modular exponentiation engine (E2) for calculation .final, Calculated by the modular multiplication engine (E3). It is worth noting that The input data preprocessing stage is completed by dedicated hardware modules. According to the corresponding strategy, the ratio of E1, E2 and E3 is determined, and the engines are allocated accordingly. In Paillier decryption, in addition to the modular exponentiation and modular multiplication operation engines, other modules are also used to implement decryption. The CRT module can be used to optimize the decryption algorithm to accelerate federated learning.

[0053] As an optional embodiment, the modular exponentiation step is performed in the following manner: converting the input data of the modular exponentiation from an initial format to a Montgomery space format; calculating the input data in the Montgomery space format to obtain a calculation result; converting the calculation result from the Montgomery space format to the initial format to obtain a modular exponentiation result of the input data.

[0054] Optionally, before performing modular exponentiation, the input data may be converted to a Montgomery space format. Montgomery space is a special numerical representation that allows modular exponentiation to be performed faster. After the input data is converted to the Montgomery space format, modular exponentiation can be performed directly in this format. Since Montgomery space optimizes the computational process of modular exponentiation, using shift operations without modular operations, it is usually faster than performing the same operation in a traditional format. After completing the modular exponentiation in the Montgomery space, the result is in the Montgomery space format. In order to obtain the final modular exponentiation result, the result needs to be converted from the Montgomery space format back to the original format. The Montgomery space format simplifies some steps in modular exponentiation, especially in encryption algorithms that require frequent modular exponentiation. By using the Montgomery space, the operation speed can be significantly improved, the computational complexity can be reduced, and the overall encryption performance can be improved.

[0055] The mathematical formulas for the two basic operators, modular exponentiation and modular multiplication, are as follows:

[0056]

[0057]

[0058] When used in cryptographic operations, a, b, m, and n are all large numbers, which leads to high computational complexity. Therefore, before designing a high-performance computing engine, the present invention first applies a fast exponentiation optimization strategy to reduce the number of multiplications used in modular exponentiation operations from O( ) to O(N) (where N is the bit width of e), and uses Montgomery modular multiplication to replace modular multiplication operations with hardware-friendly bit shift operations. After applying these optimization methods, the cryptographic calculation process using the DPU-based high-performance modular exponentiation and modular multiplication calculation engine has the following four stages:

[0059] S1. Public data preparation: Based on the input data n, first prepare the public data required by the Montgomery space. In this process, considering the universality of public keys in the Paillier and RSA encryption systems, the same public key is used to prepare data for all subsequent calculations.

[0060] S2. Data preprocessing: Perform preprocessing steps on the input data to convert it into Montgomery space format for subsequent accelerated processing.

[0061] S3. Montgomery format calculation: Calculation in Montgomery format mainly includes large number multiplication, addition and shift operations without modular operations.

[0062] S4. Data restoration: Finally, all output data are converted from Montgomery space back to the original format to ensure the correctness of the results and the usability of the application.

[0063] In order to efficiently execute the four main computing stages on the DPU by pipeline processing between different engines and within the engine, the present invention adopts an engine pipeline stage manager in the inter-engine pipeline to control the execution strategy of different computing stages. Among them, the S1 stage is set as an independent stage, which can be executed in advance once the public key is obtained; and for all computing tasks using the same public key, once the data is ready, they can be executed in parallel with the S1 stage; for tasks using different keys, they can be executed independently.

[0064] In terms of the in-engine pipeline, the present invention further implements the in-engine pipeline in the most computationally intensive S3 stage to speed up the execution speed within the stage. The key steps include: (1) supporting variable bit width so that applications can choose key lengths according to their security requirements; (2) making full use of hardware resources. To achieve the first goal, the present invention constructs an efficient processing 2 w Data pipeline for arithmetic operations (w=64 is used in the implementation of the present invention). For any given input data, it can be divided into d w-bit integers. For example, when the bit width of the input data is 1024, d = 16; when the bit width is 2048, d = 32. In theory, this pipeline can be applied to input data of any bit width as long as the bit width is an integer multiple of d, or can be extended to an integer multiple of d by padding zeros. The optimized Montgomery modular multiplication algorithm used in the present invention in the S3 stage after data segmentation is made more hardware-friendly through the following optimization designs: (1) The calculation of S is divided into the calculation of the low w bits and the high w bits, thereby halving the bit width required in the operation (such as addition); (2) The first iteration of the inner loop is expanded to remove the conditional statements in the original algorithm and maintain the consistency of the calculation logic. In the optimized Montgomery modular multiplication algorithm, the most computationally intensive operation is X j ×Y i and q×N j multiplication (both operations require d 2 w-bit multiplications). Considering that the data required for the two multiplications are completely independent, the two multipliers Mul 1 and Mul 2 can be used to perform the two multiplication operations respectively, and they can also be reused to perform other multiplication operations. Since the multiplier can process data continuously, in order to make full use of the multiplier, the present invention constructs a fully pipelined inner loop circuit inside the DPU to cooperate with the pipeline processing inner loop; secondly, when the multiplication of a certain iteration is completed and other operations are still being executed, the present invention allows the multiplication of the next iteration to be started directly to minimize the delay between different iterations.

[0065] As an optional embodiment, the multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, and also include: determining the engines corresponding to the multiple steps that can be executed in parallel; sending the multiple steps that can be executed in parallel to a data distributor, and the data distributor distributes the multiple steps that can be executed in parallel to the corresponding engines, wherein the data distributor includes a multi-layer distributor, and the multi-layer distributor distributes data in a tree structure.

[0066] Optionally, in order to achieve efficient task allocation, an intermediate link, a data distributor, can be set between the execution subject scheduler and the engine of the present invention, which is responsible for receiving and forwarding tasks to the corresponding engine. The data distributor forwards the received parallel steps to the corresponding engine according to the task allocation strategy. This ensures that each engine can receive the tasks it needs to execute. The present invention can set up multiple levels of data distributors, which can be responsible for task distribution of different ranges or types to adapt to parallel computing requirements of different scales or structures. The tree structure refers to a hierarchical data distribution method, in which the top-level distributor distributes tasks to the distributor of the next layer, and so on, until the bottom-level distributor directly assigns tasks to specific engines. This structure helps to achieve high efficiency and flexibility in task allocation. This method is suitable for large-scale parallel computing scenarios, such as distributed machine learning, big data processing, etc., and can significantly improve the efficiency of task allocation and execution.

[0067] Figure 5 is a schematic diagram of data hierarchical distribution according to an optional embodiment of the present invention, such as Figure 5 As shown in FIG. 1 , since the on-chip storage units are placed near each engine to achieve low latency, simply distributing data from the AXI interconnect to these units will lead to high fan-out and long data paths near the interconnect. The present invention proposes a hierarchical data distribution mechanism, in which a data distributor receives data from the upper layer at each layer and further distributes the data to the data distributor or engine at the lower layer. Assuming there are m engines and n layers, the fan-out of each data distributor is approximately , which is much smaller than the traditional fan-out number m. This design effectively reduces the fan-out of the data distributor and the length of the logic data path, reduces the design complexity, and allows a higher operating frequency.

[0068] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0069] Through the description of the above implementation methods, those skilled in the art can clearly understand that the acceleration method of federated learning according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0070] According to an embodiment of the present invention, a federated learning acceleration device for implementing the above-mentioned federated learning acceleration method is also provided. Figure 6 is a structural block diagram of an acceleration device for federated learning provided according to an embodiment of the present invention, such as Figure 6 As shown, the acceleration device for federated learning includes: a receiving module 61, a first determining module 62, a second determining module 63, an allocating module 64 and a generating module 65. The acceleration device for federated learning is described below.

[0071] The receiving module 61 is used to receive the target data generated in the training phase of the model training method based on federated learning, and the encryption algorithm identifier used to encrypt the target data.

[0072] The first determining module 62 is connected to the receiving module 61 and is used to determine the processing steps for encrypting the target data according to the identifier of the encryption algorithm.

[0073] The second determining module 63 is connected to the first determining module 62 and is used to determine a plurality of steps that can be executed in parallel in the processing steps.

[0074] The allocation module 64 is connected to the second determination module 63 and is used to allocate the multiple steps that can be executed in parallel to the engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by the corresponding engines to obtain the execution results of the multiple engines.

[0075] The generation module 65 is connected to the allocation module 64 and is used to generate encryption results of the target data according to the execution results of each of the multiple engines.

[0076] It should be noted that the receiving module 61, the first determining module 62, the second determining module 63, the allocating module 64 and the generating module 65 correspond to steps S201 to S205 in the embodiment, and the examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0077] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0078] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the acceleration method and device of federated learning in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the acceleration method of federated learning described above is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receive the target data generated in the training phase of the model training method based on federated learning, and the encryption algorithm identifier used to encrypt the target data; determine the processing steps for encrypting the target data according to the identifier of the encryption algorithm; among the processing steps, determine multiple steps that can be executed in parallel; assign the multiple steps that can be executed in parallel to the engine for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by the corresponding engines to obtain the execution results of the multiple engines; generate the encryption result of the target data according to the execution results of the multiple engines.

[0080] Optionally, in the processing steps, multiple steps that can be executed in parallel are determined, including: determining an encryption step of an encryption algorithm; splitting the encryption step into basic operation steps, wherein the basic operation steps include modular exponentiation operation steps and modular multiplication operation steps; based on the basic operation steps, determining multiple steps that can be executed in parallel.

[0081] Optionally, multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines to obtain execution results of each of the multiple engines, including: assigning basic operation steps of each of the multiple steps to corresponding engines for parallel execution to obtain execution results of each of the multiple engines.

[0082] Optionally, the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines, including: determining the amount of calculation required for each of the multiple steps; with the goal of minimizing the difference in the amount of calculation between the multiple engines, the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines.

[0083] Optionally, the modular exponentiation step is performed in the following manner: converting the input data of the modular exponentiation from an initial format to a Montgomery space format; calculating the input data in the Montgomery space format to obtain a calculation result; converting the calculation result from the Montgomery space format to the initial format to obtain a modular exponentiation result of the input data.

[0084] Optionally, the multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, and also include: determining the engines corresponding to the multiple steps that can be executed in parallel; sending the multiple steps that can be executed in parallel to a data distributor, and the data distributor distributes the multiple steps that can be executed in parallel to the corresponding engines, wherein the data distributor includes a multi-layer distributor, and the multi-layer distributor distributes data in a tree structure.

[0085] By adopting the embodiment of the present invention, a scheme for accelerating federated learning is provided. By adopting a method of determining multiple steps that can be executed in parallel in an encryption algorithm and executing multiple steps that can be executed in parallel in parallel, the target data generated in the training phase of the model training method based on federated learning and the encryption algorithm identifier used to encrypt the target data are received; according to the identifier of the encryption algorithm, the processing steps for encrypting the target data are determined; in the processing steps, multiple steps that can be executed in parallel are determined; the multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by the corresponding engines, and the execution results of the multiple engines are obtained; according to the execution results of the multiple engines, the encryption results of the target data are generated, and the purpose of shortening the execution time of the encryption algorithm is achieved. As long as the encryption algorithm is used for encryption in federated learning, the method provided by the present invention can be used for acceleration, thereby achieving the technical effect of improving the applicability of the acceleration method of federated learning, and then solving the technical problem that the acceleration method for federated learning in the related technology is not universal.

[0086] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0087] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the acceleration method of federated learning provided by the above embodiment.

[0088] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0089] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: receiving target data generated in the training phase of the model training method based on federated learning, and an encryption algorithm identifier used to encrypt the target data; determining a processing step for encrypting the target data based on the identifier of the encryption algorithm; determining a plurality of steps that can be executed in parallel among the processing steps; respectively allocating the plurality of steps that can be executed in parallel to engines for parallel execution, so that the plurality of steps that can be executed in parallel are respectively executed by corresponding engines to obtain execution results of the plurality of engines; and generating an encryption result of the target data based on the execution results of the plurality of engines.

[0090] Optionally, in the processing steps, multiple steps that can be executed in parallel are determined, including: determining an encryption step of an encryption algorithm; splitting the encryption step into basic operation steps, wherein the basic operation steps include modular exponentiation operation steps and modular multiplication operation steps; based on the basic operation steps, determining multiple steps that can be executed in parallel.

[0091] Optionally, multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines to obtain execution results of each of the multiple engines, including: assigning basic operation steps of each of the multiple steps to corresponding engines for parallel execution to obtain execution results of each of the multiple engines.

[0092] Optionally, the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines, including: determining the amount of calculation required for each of the multiple steps; with the goal of minimizing the difference in the amount of calculation between the multiple engines, the basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines.

[0093] Optionally, the modular exponentiation step is performed in the following manner: converting the input data of the modular exponentiation from an initial format to a Montgomery space format; calculating the input data in the Montgomery space format to obtain a calculation result; converting the calculation result from the Montgomery space format to the initial format to obtain a modular exponentiation result of the input data.

[0094] Optionally, the multiple steps that can be executed in parallel are respectively assigned to engines for parallel execution, and also include: determining the engines corresponding to the multiple steps that can be executed in parallel; sending the multiple steps that can be executed in parallel to a data distributor, and the data distributor distributes the multiple steps that can be executed in parallel to the corresponding engines, wherein the data distributor includes a multi-layer distributor, and the multi-layer distributor distributes data in a tree structure.

[0095] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the acceleration method of federated learning in each embodiment of the present application.

[0096] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0097] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0100] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for accelerating federated learning, characterized in that: include: Receiving target data generated during a training phase of a model training method based on federated learning, and an identifier of an encryption algorithm used to encrypt the target data; Determining, according to the identifier of the encryption algorithm, a processing step for encrypting the target data; In the processing step, a plurality of steps that can be performed in parallel are determined; Assigning the multiple steps that can be executed in parallel to engines for parallel execution respectively, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines, and obtaining execution results of the multiple engines respectively; An encryption result of the target data is generated according to the execution results of each of the multiple engines.

2. The method according to claim 1, characterized in that In the processing step, determining a plurality of steps that can be executed in parallel includes: Determining an encryption step of the encryption algorithm; Splitting the encryption step into basic operation steps, wherein the basic operation steps include modular exponentiation operation steps and modular multiplication operation steps; The multiple steps that can be executed in parallel are determined according to the basic operation steps.

3. The method according to claim 2, characterized in that The step of assigning the plurality of steps that can be executed in parallel to engines for parallel execution respectively, so that the plurality of steps that can be executed in parallel are respectively executed by corresponding engines, and obtaining respective execution results of the plurality of engines, includes: The basic operation steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution, so as to obtain execution results of each of the multiple engines.

4. The method according to claim 3, characterized in that The step of assigning the basic operation steps of each of the plurality of steps to corresponding engines for parallel execution to obtain the execution results of each of the plurality of engines includes: Determining the amount of computation required for each of the plurality of steps; With the goal of minimizing the difference in computational load between the multiple engines, the basic computational steps of each of the multiple steps are respectively assigned to corresponding engines for parallel execution to obtain the execution results of each of the multiple engines.

5. The method according to claim 3, characterized in that: The modular exponentiation step is performed in the following manner: Converting the input data of the modular exponentiation operation from an initial format to a Montgomery space format; Calculating the input data in the Montgomery space format to obtain a calculation result; The calculation result is converted from the Montgomery space format to the initial format to obtain a modular exponentiation result of the input data.

6. The method according to any one of claims 1 to 5, characterized in that The step of allocating the plurality of steps that can be executed in parallel to engines for parallel execution also includes: Determining engines corresponding to the multiple steps that can be executed in parallel; The multiple steps that can be executed in parallel are sent to a data distributor respectively, and the data distributor distributes the multiple steps that can be executed in parallel to corresponding engines, wherein the data distributor includes a multi-layer distributor, and the multi-layer distributor distributes data in a tree structure.

7. A federated learning acceleration device, characterized in that: include: A receiving module, used to receive target data generated in a training phase of a model training method based on federated learning, and an encryption algorithm identifier used to encrypt the target data; A first determination module, used to determine a processing step for encrypting the target data according to an identifier of the encryption algorithm; A second determining module, used to determine a plurality of steps that can be executed in parallel in the processing steps; An allocation module, used to allocate the multiple steps that can be executed in parallel to engines for parallel execution, so that the multiple steps that can be executed in parallel are respectively executed by corresponding engines to obtain execution results of the multiple engines; A generating module is used to generate an encryption result of the target data according to the execution results of each of the multiple engines.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the acceleration method of federated learning as described in any one of claims 1 to 6.

9. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program is run, the processor executes the acceleration method of federated learning as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: The computer instructions are executed by a processor to implement the method for accelerating federated learning as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Data encryption and decryption processing method and device based on Montgomery modular multiplication operation

    CN102207847A

  • Processing architecture, accelerator and method for federated learning

    CN112070222A

  • Ciphertext computing device and method for federated learning and privacy computing

    CN113946846A

  • Paillier homomorphic encryption method and system based on low-complexity modular multiplication algorithm

    CN117692126A

  • Cryptographic system, method and multiplier

    US20110264720A1

Cited By

  • Cooperative assembly line task scheduling method and system suitable for hierarchical federated network

    CN120492101A