System and method for supporting joint training of universal equipment and neuromorphic equipment

Abstract

The invention provides a system and a method for supporting joint training of universal equipment and neuromorphic equipment. The system comprises an ANN client, an SNN client, a federated learning center server and an ANN-SNN conversion module, and the ANN client, the SNN client and the ANN-SNN conversion module are all in wireless communication connection with the federal learning center server. According to the federated learning framework supporting ANN-SNN heterogeneous training provided by the invention, the heterogeneity between two devices is overcome for the first time through an ANN-SNN conversion method, a general computing device and neuromorphic hardware can be utilized at the same time, and two different models of SNN and ANN are supported to be added into the federated learning process for the first time. In this way, the framework can fully utilize the edge data obtained by the two types of clients to support training, and the problem that the data utilization rate is low due to equipment heterogeneity is solved.

Description

Technical Field

[0001] The present invention relates to a system and method for supporting joint training of general-purpose devices and neuromorphic devices, belonging to the technical field of federated learning. Background Art

[0002] As mobile Internet technology and Internet of Things technology mature, the explosive growth of edge data has provided strong support for the development of edge artificial intelligence. At present, edge devices generally have the defect of low computing power and have great limitations in edge computing tasks. Different from traditional artificial neural networks (ANN) and general computing devices, software and hardware solutions based on spiking neural networks (SNN) and neuromorphic hardware have made significant progress in edge scenarios, and are more likely to complete corresponding edge computing tasks with low power consumption and low latency in the future. However, the existing mainstream edge devices are still ahead of spiking neural networks in terms of accuracy and network architecture because they are equipped with mature traditional artificial neural networks. This may lead to the fact that general computing devices and neuromorphic hardware will be used in actual production and deployment for a long time in the future, and there is no method to support cross-device data sharing and model training in this scenario.

[0003] As neuromorphic hardware becomes more popular in the future, organizing large-scale edge devices for joint training will face a series of difficulties, such as: (1) Only solutions that conduct federated learning on homogeneous edge devices, such as training artificial neural networks only between general-purpose computing devices, or training spiking neural networks only between neuromorphic hardware, cannot make full use of all edge data for training; (2) Joint edge devices and general-purpose computing devices for federated learning, using artificial neural networks or spiking neural networks, will inevitably fail to give full play to the high-precision advantages of artificial neural networks or the low-power characteristics of neuromorphic hardware, resulting in cost waste or loss of model accuracy.

[0004] This paper addresses the difficulties brought about by the mixing of neuromorphic hardware and traditional general-purpose computing devices in edge computing scenarios, and for the first time provides a framework that can support collaborative training of two devices equipped with SNN and ANN respectively. Through the ANN-SNN parameter conversion method, a breakthrough has been made in overcoming the heterogeneity between the two devices, thereby supporting a wider range of joint training and edge data utilization at a lower cost. Summary of the invention

[0005] The present invention aims to solve the technical problem in the prior art that it is impossible to realize cross-device data sharing and model training in actual production and deployment scenarios when general-purpose computing devices and neuromorphic hardware are simultaneously applied, and further proposes a system and method that supports joint training of general-purpose devices and neuromorphic devices.

[0006] The technical solution adopted by the present invention to solve the above problems is: the present invention proposes a system supporting joint training of general devices and neuromorphic devices, comprising:

[0007] General computing devices equipped with ANN models, neuromorphic hardware devices and central servers equipped with SNN models;

[0008] The general computing device equipped with the ANN model, the neuromorphic hardware device equipped with the SNN model, and the ANN-SNN conversion module are all wirelessly connected to the central server.

[0009] Optionally, the central server includes a parameter conversion submodule and a global model allocation submodule;

[0010] The global model allocation submodule is used to allocate corresponding global models to general computing devices equipped with ANN models and neuromorphic hardware devices equipped with SNN models;

[0011] The parameter conversion submodule is used to convert ANN or SNN models on the federated learning central server.

[0012] A method for supporting joint training of a general purpose device and a neuromorphic device, comprising:

[0013] Step 1: The central server obtains the initial model structure of ANN (artificial neural network) and initializes the parameters;

[0014] Step 2: The central server distributes the current global ANN model to the general computing device equipped with the ANN model, and then converts the current global ANN model into an SNN (spiking neural network) model through the parameter conversion module, and distributes it to the neuromorphic hardware device equipped with the SNN model;

[0015] Step 3: The general computing device equipped with the ANN model and the neuromorphic hardware device equipped with the SNN model are trained using their own local data sets based on the global model distributed by the central server to obtain the corresponding local ANN or SNN model parameters. After the training is completed, the local model parameters are uploaded to the central server;

[0016] Step 4: The central server aggregates the received ANN and SNN networks respectively to obtain temporary global ANN model parameters and temporary global SNN model parameters, and then converts the temporary global SNN model into an ANN model through a parameter conversion module, and aggregates it with the temporary global ANN model to obtain the global ANN model parameters of this round;

[0017] Step 5: Repeat steps 2-4 until all global communication rounds are completed to obtain the ANN global model.

[0018] Furthermore, in step 2, the step in which the parameter conversion module converts the current global ANN model into the SNN model includes: converting the activation layer in the ANN network into a pulse neuron, such as an Integrate-and-Fire or Leaky Integrate-and-Fire model, and mapping the weights and biases of the ANN to the SNN through a scaling factor and setting the number of time steps according to the device conditions.

[0019] Furthermore, the steps of updating the local ANN model and updating the local SNN model in step 3 include: the general computing device equipped with the ANN model trains the ANN local model through a gradient-based optimization method in combination with the ANN global model parameters obtained from the central server to obtain the ANN local model parameters; the neuromorphic hardware device equipped with the SNN model trains the SNN local model through a proxy gradient method in combination with the SNN global model parameters obtained from the central server to obtain the SNN local model parameters;

[0020] Furthermore, the step of obtaining the temporary global ANN model parameters and the temporary global SNN model parameters in step 4 includes: the central server obtains the temporary global ANN model and the temporary global SNN model by a weighted average method for the received local ANN model and SNN model parameters;

[0021] Furthermore, the parameter conversion module in step 4 converts the temporary global SNN model to include: converting the pulse neurons in the SNN model into the activation functions in the ANN model, mapping the weights and biases of the temporary global SNN to the ANN model according to the time step and scaling factor of the temporary global SNN model, and obtaining the converted ANN model.

[0022] The beneficial effects of the present invention are:

[0023] 1. The federated learning method supporting heterogeneous ANN-SNN training provided by the present invention overcomes the heterogeneity between the two devices for the first time through the ANN-SNN conversion method, can simultaneously utilize general computing devices and neuromorphic hardware, and for the first time supports adding two different types of models, SNN and ANN, to the federated learning process. This enables the framework to fully utilize the edge data obtained by the two types of clients to support training, overcoming the problem of low data utilization caused by device heterogeneity.

[0024] 2. While overcoming the problem of device heterogeneity and expanding the amount of training data, the present invention also enables the final deployed SNN model to have a higher level of accuracy on neuromorphic hardware than traditional SNN federated learning, thus overcoming the shortcomings of SNN in terms of accuracy level.

[0025] 3. Because the present invention adopts neuromorphic hardware, when using the same number of clients for training, it has lower power consumption and wider applicability than the traditional ANN federated learning framework, and is more suitable for edge artificial intelligence scenarios than pure ANN solutions.

[0026] 4. The present invention also enables the client to select equipment according to its own environmental conditions before training begins, and selectively carry ANN or SNN to participate in training, so as to balance the high precision and low power consumption of the final deployed model, with higher flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A structural diagram of a system supporting joint training of general-purpose devices and neuromorphic devices provided by the present invention;

[0028] Figure 2 A flowchart of a method for supporting joint training of a general device and a neuromorphic device provided by the present invention. Example

[0029] Example 1

[0030] like Figure 1 As shown, the structure of a system supporting joint training of a general device and a neuromorphic device described in this embodiment includes:

[0031] General computing devices equipped with ANN models, neuromorphic hardware devices equipped with SNN models, federated learning central servers, and ANN-SNN conversion modules;

[0032] The general computing device equipped with the ANN model is used to provide the ANN local model parameters;

[0033] The neuromorphic hardware device equipped with the SNN model is used to provide the SNN local model parameters;

[0034] The ANN-SNN conversion module is used to provide the SNN initial model parameters;

[0035] The federated learning central server is used to provide ANN initial model parameters and SNN initial model parameters, and to convert the format of model parameters uploaded by general computing devices equipped with ANN models and neuromorphic hardware devices equipped with SNN models to obtain ANN global models and SNN global models.

[0036] General-purpose computing devices equipped with ANN models, general-purpose computing devices equipped with SNN models, and ANN-SNN conversion modules all establish wireless communication connections with the federated learning central server, upload the corresponding trained model parameters to the federated learning central server, and the federated learning central server transmits the global model parameters obtained by global aggregation to the corresponding client.

[0037] The federated learning central server includes a parameter conversion submodule and a global model allocation submodule;

[0038] The global model allocation submodule is used to allocate corresponding global models to the general computing devices equipped with ANN models and the general computing devices equipped with SNN models;

[0039] The parameter conversion submodule is used to convert ANN or SNN models on the federated learning central server.

[0040] Example 2

[0041] like Figure 2 As shown, the steps of a method for supporting joint training of a general device and a neuromorphic device described in this embodiment include:

[0042] 1. Initialization phase:

[0043] The central server obtains the ANN network architecture and initial parameters specified by the task Replace the activation function in the specified architecture with the step ReLU function:

[0044]

[0045] In formula (1), z (l) is the input of the current activation layer, θ (l) For input z (l) The actual maximum value after being mapped to the output, δ1 is the shift deviation, and c is the coefficient that controls the step function and quantized ReLU. After the conversion is completed, the parameters of other convolutional layers, fully connected layers, etc. are mapped through the scaling coefficient.

[0046] 2. Broadcasting stage:

[0047] In the tth round, the central server now has both the ANN global network and the corresponding SNN global network after conversion. For general hardware devices participating in training, the central server distributes the ANN global network, and for neuromorphic hardware devices participating in training, the central server distributes the SNN global network.

[0048] 3. Local training phase:

[0049] When the general hardware device receives the ANN global network distributed by the central server, it uses a gradient-based optimization method to update the local model on the local data:

[0050]

[0051] In formula (2), is the local dataset used in the current round, ηi The learning rate calculated locally, optim(), represents the gradient-based optimization method.

[0052] When the neuromorphic hardware device receives the SNN global network distributed by the central server, it uses the proxy gradient method to update its local model on the local data. Here, the spatiotemporal back propagation model is used:

[0053]

[0054] In formulas (3)-(6), represents pulse output, t is the timestamp, n is the layer number, It is not differentiable, and can be solved by the following formula:

[0055]

[0056] After completing local calculations, each device uploads its trained local model

[0057] 4. Aggregation stage

[0058] After receiving all local networks uploaded by general hardware devices and neuromorphic hardware devices, the central server performs weighted aggregation on all received SNN networks:

[0059]

[0060] In formula (8), is the set of all neuromorphic hardware devices participating in the training in the current round, α i is the aggregation coefficient.

[0061] After the SNN network is aggregated, it is aggregated with the remaining ANN networks to obtain the global model

[0062]

[0063] In formula (9), is the set of all general hardware devices participating in the training in the current round, and β is the aggregation coefficient controlling the ANN network and the SNN network. After the aggregation of the ANN global model is completed, the SNN global network is obtained through the ANN global network.

[0064] Repeat steps 2-4 until all global communication rounds are completed, and obtain the ANN global model and the SNN global model;

[0065] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A system supporting joint training of general purpose devices and neuromorphic devices, characterized in that: The system structure supporting joint training of general-purpose devices and neuromorphic devices includes: General computing devices equipped with ANN models, neuromorphic hardware devices and central servers equipped with SNN models; The general computing device equipped with the ANN model and the neuromorphic hardware device equipped with the SNN model are both wirelessly connected to the central server.

2. A system for supporting joint training of general purpose devices and neuromorphic devices according to claim 1, characterized in that: The central server includes a parameter conversion submodule and a global model allocation submodule; The global model allocation submodule is used to allocate corresponding global models to the general computing device equipped with the ANN model and the neuromorphic hardware device equipped with the SNN model; The parameter conversion submodule is used to convert the ANN or SNN models on the federated learning central server.

3. A method for supporting joint training of general-purpose devices and neuromorphic devices, applied to a system for supporting joint training of general-purpose devices and neuromorphic devices as claimed in any one of claims 1-2, characterized in that: The method for supporting joint training of a general device and a neuromorphic device comprises the following steps: Step 1: The central server obtains the ANN initial model structure and initializes the parameters; Step 2: The central server distributes the current global ANN model to the general computing device equipped with the ANN model, and then converts the current global ANN model into an SNN model through the parameter conversion module, and distributes it to the neuromorphic hardware device equipped with the SNN model; Step 3: The general computing device equipped with the ANN model and the neuromorphic hardware device equipped with the SNN model are trained using their own local data sets based on the global model distributed by the central server to obtain the corresponding local ANN or SNN model parameters. After the training is completed, the local model parameters are uploaded to the central server; Step 4: The central server aggregates the received ANN and SNN networks respectively to obtain temporary global ANN model parameters and temporary global SNN model parameters, and then converts the temporary global SNN model into an ANN model through a parameter conversion module, and aggregates it with the temporary global ANN model to obtain the global ANN model parameters of this round; Step 5: Repeat steps 2-4 until all global communication rounds are completed to obtain the ANN global model.

4. A method for supporting joint training of general purpose devices and neuromorphic devices according to claim 3, characterized in that: Step 2 converts the current global ANN model into an SNN model, specifically including: Convert the activation layer in the ANN network into spiking neurons, and map the weights and biases of the ANN to the SNN through the scaling factor and set the number of time steps according to the device conditions.

5. A method for supporting joint training of general purpose devices and neuromorphic devices according to claim 3, characterized in that: The updating of the local ANN model and the updating of the local SNN model in step 3 specifically include: The general computing device equipped with the ANN model trains the ANN local model through a gradient-based optimization method combined with the ANN global model parameters obtained from the central server to obtain the ANN local model parameters; the neuromorphic hardware device equipped with the SNN model trains the SNN local model through a proxy gradient method combined with the SNN global model parameters obtained from the central server to obtain the SNN local model parameters.

6. A method for supporting joint training of general purpose devices and neuromorphic devices according to claim 3, characterized in that: The temporary global ANN model parameters and the temporary global SNN model parameters obtained in step 4 specifically include: The central server obtains a temporary global ANN model and a temporary global SNN model by weighted averaging the received local ANN model and SNN model parameters.

7. A method for supporting joint training of general purpose devices and neuromorphic devices according to claim 3, characterized in that: The parameter conversion module in step 4 converts the temporary global SNN model into an ANN model, specifically including: The pulse neurons in the SNN model are converted into activation functions in the ANN model, and the weights and biases of the temporary global SNN are mapped to the ANN model according to the time step and scaling factor of the temporary global SNN model to obtain the converted ANN model.

Images