Hybrid neural network accelerator, model training method, and system

Abstract

The application provides a hybrid neural network accelerator, a model training method and a system. The accelerator comprises: an instruction prefetch unit configured to obtain an operation code; a control unit configured to control a first read-write unit and a second read-write unit to perform different mode data read-write actions according to the operation code, and determine a target calculation mode; a calculation array configured to perform tensor operation on data output by the first read-write unit and the second read-write unit according to the target calculation mode, to obtain an operation result of a current network layer at a current operation stage; and a nonlinear acceleration component configured to, in a forward propagation stage, perform index calculation and backup according to the operation result; and in a backward propagation stage, calculate an output gradient of the current network layer according to the backup index and an output gradient returned by a subsequent network layer, to perform backward propagation update. The accelerator realizes low-delay, low-power-consumption and on-end continuous learning supporting hybrid neural network training acceleration with small hardware resource consumption.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a hybrid neural network accelerator, model training method and system. Background Technology

[0002] As artificial intelligence technology shifts to the edge, edge devices not only need to have low-power inference capabilities, but also face the overall technical requirement of continuous online learning based on new data in order to adapt to complex, ever-changing, and personalized application environments.

[0003] To meet the aforementioned edge processing requirements, some existing technologies employ a hybrid architecture that physically splices independent Artificial Neural Network (ANN) computing hardware and Spiking Neural Network (SNN) computing hardware through asynchronous caching. During data flow, the corresponding computing path is selected based on the data sparsity characteristics to complete the deployment and operation of the model.

[0004] However, for edge training, the data flow of forward propagation, backward propagation, and weight gradient calculation in gradient descent training methods are significantly different, and the state cache requires a large amount of memory access and computational resources, resulting in high hardware resource consumption. Summary of the Invention

[0005] This invention provides a hybrid neural network accelerator, model training method, and system to address the shortcomings of high hardware resource consumption in existing technologies. It achieves low latency, low power consumption, and supports continuous on-device learning acceleration of hybrid neural network training with relatively low hardware resource consumption.

[0006] This invention provides a hybrid neural network accelerator, comprising: a convolution acceleration component and a nonlinear acceleration component, wherein the convolution acceleration component includes an instruction prefetching unit, a control unit, a first read / write unit, a second read / write unit, and a computing array; The instruction prefetching unit is used to obtain an opcode when training the current network layer of the hybrid neural network model to be trained in the current round of training; the opcode is used to indicate the current operation stage of the convolution acceleration component, the current operation stage includes a forward propagation stage, a back propagation stage, or a weight gradient update stage; the hybrid neural network model to be trained includes an artificial neural network model and / or a spiking neural network model. The control unit is configured to control the first read / write unit to cache the activation data corresponding to the state-aware data of the sample object input to the current network layer during the forward propagation phase and the weight gradient update phase, and to cache the output gradient returned by the next network layer of the current network layer during the back propagation phase, according to the operation code. The control unit is further configured to control the second read / write unit to cache the weight data of the current network layer during the forward propagation phase, cache the weight data after data rearrangement during the back propagation phase, and cache the output gradient during the weight gradient update phase, according to the opcode. The control unit is also configured to determine the target calculation mode based on the operation code; The computing array is used to perform tensor operations on the data output by the first read / write unit and the second read / write unit according to the target computing mode, so as to obtain the computing result of the current network layer in the current computing stage; The nonlinear acceleration component is used to perform index calculation and backup based on the calculation result during the forward propagation phase; and to calculate the output gradient of the current network layer based on the backed-up index and the output gradient returned by the next network layer during the back propagation phase, and to perform back propagation update based on the output gradient of the current network layer.

[0007] According to a hybrid neural network accelerator provided by the present invention, the control unit is further configured to: If the current computation stage indicated by the opcode is the forward propagation stage or the backward propagation stage, the target computation mode is determined to be the inner product mode. If the current computation phase indicated by the opcode is the weight gradient update phase, the target computation mode is determined to be the outer product mode.

[0008] According to a hybrid neural network accelerator provided by the present invention, the hybrid neural network accelerator further includes an output buffer unit; The computing array includes at least one computing unit; Each of the aforementioned computing units is configured to perform tensor operations on each data block in the data output by the first read / write unit and the second read / write unit according to the target computing mode, obtain local computation results, and send the local computation results to the output buffer unit; The output buffer unit is used to read the historical accumulated value recorded at the storage address corresponding to the local calculation result, perform an addition operation between the local calculation result and the historical accumulated value to obtain an updated accumulated value, and write the updated accumulated value back to the storage address until all calculation loops of the current network layer in the current calculation stage are completed, and the last accumulated value written back is used as the calculation result.

[0009] According to a hybrid neural network accelerator provided by the present invention, the convolution acceleration component further includes a gating unit; Each of the aforementioned computing units includes a multiplier and an adder; The gating unit is configured to control each of the computation units to perform the tensor operations in the following manner: For each data block output by the first read / write unit and the second read / write unit, if the data value of the data block is zero, the target computing unit allocated the data block is controlled to skip the tensor operation for the data block. When the data value of the data block is non-zero, the hybrid neural network model to be trained is the spiking neural network model, and the current operation stage is the forward propagation stage or the weight gradient update stage, the multiplier in the target computing unit is turned off, and the tensor operation for the data block is performed by the adder in the target computing unit. When the data value of the data block is non-zero and the hybrid neural network model to be trained is the artificial neural network model, or when the data value of the data block is non-zero, the hybrid neural network model to be trained is the spiking neural network model, and the current operation stage is the backpropagation stage, the multiplier in the target computing unit is activated, and tensor operations on the data block are performed through the adder and multiplier in the target computing unit.

[0010] According to a hybrid neural network accelerator provided by the present invention, the second read / write unit is used to cache the weight data after data rearrangement during the backpropagation stage in the following manner: Obtain the number of channels and the number of convolutional kernels of the input feature map corresponding to the weight data; Based on the number of channels and the number of convolutional kernels, the weight data is subjected to a dimension swap operation between the input channel dimension and the convolutional kernel number dimension. The weighted data after the dimension swap operation is rotated in the spatial dimension to obtain the rearranged weighted data; The rearranged weight data is cached.

[0011] According to a hybrid neural network accelerator provided by the present invention, the hybrid neural network accelerator further includes a normalization component and a pooling component; The control unit is further configured to configure the data processing path between the computing array, the normalization component, the pooling component, and the nonlinear acceleration component according to the configuration parameters of the current network layer. The normalization component is used to perform batch normalization processing or layer normalization processing on the received data when the data processing path includes a normalization processing node. The pooling component is used to perform pre-downsampling or reverse upsampling processing on the received data when the data processing path includes a pooling processing node.

[0012] According to a hybrid neural network accelerator provided by the present invention, the nonlinear acceleration component is specifically used for: In the forward propagation phase, if the data processing path does not contain normalization processing nodes and pooling processing nodes, the calculation result is used as the target feature data. If the data processing path includes a normalization processing node and / or a pooling processing node, the data output by the normalization component and / or the pooling component after processing the calculation result shall be used as the target feature data. Based on the target feature data, obtain the internal state data corresponding to the current network layer; Based on the segment number in the piecewise linear fitting function corresponding to the hybrid neural network model to be trained, the internal state data is indexed and backed up.

[0013] According to a hybrid neural network accelerator provided by the present invention, the nonlinear acceleration component is further used for: When the hybrid neural network model to be trained is the artificial neural network model, the target feature data is used as the internal state data; When the hybrid neural network model to be trained is the spiking neural network model, the membrane voltage of each neuron in the nonlinear layer of the current network layer at each time step is calculated based on the target feature data. The membrane voltage is used as the internal state data.

[0014] The present invention also provides a model training method, which is implemented based on any of the hybrid neural network accelerators described above, the method comprising: The artificial neural network model to be trained is trained based on the state perception data of the first sample object and the state recognition label of the first sample object. Based on the model parameters of the trained artificial neural network model, the spiking neural network model to be trained is initialized. The spiking neural network model to be trained is trained based on the state perception data and state recognition labels of the second sample object to obtain the state recognition model.

[0015] The present invention also provides a hybrid neural network acceleration system, including a main controller, a memory, an input device, a bus, and an accelerator cluster; the accelerator cluster includes at least one hybrid neural network accelerator as described in any of the above embodiments. The bus is used to establish communication links between each of the hybrid neural network accelerators, the main controller, the memory, and the input device, as well as communication links between each unit within the hybrid neural network accelerator. The main controller is used to compile and generate the configuration parameters and control information of each network layer of the hybrid neural network model to be trained, and to configure and schedule each hybrid neural network accelerator. The memory is used to store the model parameters and local dataset updated during the training process of the hybrid neural network model to be trained. The input device is used to establish an interaction channel between each of the hybrid neural network accelerators and external devices.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model training method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model training method as described above.

[0019] The hybrid neural network accelerator, model training method, and system provided by this invention, through a control unit based on opcodes, control the first and second read / write units to cache data of corresponding types during the forward propagation, backpropagation, and weight gradient update stages. The computing array completes tensor operations in all three stages on the same set of physical computing power, achieving unified reuse of the asymmetric data streams across the three training stages. This overcomes the limitation of existing spliced ​​hardware architectures, which struggle to coordinate three data streams under the same physical resources, resulting in low resource utilization. Simultaneously, the nonlinear acceleration component uses an index with a bit width smaller than the result bit width to replace the original data for backup, significantly reducing the amount of memory accessed and the storage resource consumption during the backpropagation stage, effectively alleviating the memory bandwidth bottleneck. Furthermore, this accelerator is compatible with the hybrid computing requirements of artificial neural network models and spiking neural network models, thus achieving low-latency, low-power hybrid neural network training acceleration with minimal hardware resource consumption and support for continuous on-device learning. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of the hybrid neural network accelerator provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the model training method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of the hybrid neural network acceleration system provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] As artificial intelligence technology shifts to the edge, edge devices not only need to have low-power inference capabilities, but also face the overall technical requirement of continuous online learning based on new data in order to adapt to complex, ever-changing, and personalized application environments.

[0026] To meet the aforementioned edge processing requirements, some technologies employ ANN or SNN accelerators. ANN accelerators, such as Neural Processing Units (NPUs) and Tensor Processing Units (TPUs), excel at handling high-precision matrix operations, achieving high accuracy in tasks like image recognition and object detection. However, their clock-based intensive computation mode results in high power consumption at the edge and lacks biological interpretability. SNN accelerators, such as neuromorphic chips, are based on event-driven mechanisms, possessing inherent sparse computation characteristics and extremely low power consumption. However, due to the non-differentiable nature of spiking neurons, their training algorithms are complex, and their accuracy when directly deployed on high-precision tasks is often lower than that of ANNs. Furthermore, existing SNN hardware struggles to support complex error backpropagation algorithms.

[0027] To address this, existing technologies employ a hybrid architecture that physically combines independent artificial neural network computing hardware with spiking neural network computing hardware through asynchronous caching. During data flow, the corresponding computing path is selected based on the data sparsity characteristics to complete the deployment and operation of the model.

[0028] However, for edge training, the data flow of forward propagation, backward propagation, and weight gradient calculation in gradient descent training methods are significantly different, and the state cache requires a large amount of memory access and computational resources, resulting in high hardware resource consumption.

[0029] Therefore, how to simultaneously consider the forward and backward computation of ANN and the spatiotemporal information processing of SNN in a unified architecture, and achieve efficient collaborative training of the two, is an urgent problem to be solved.

[0030] To address this, this application provides a hybrid neural network accelerator that accelerates the training of artificial neural network models and spiking neural network models on the same hardware architecture through the coordinated operation of convolutional acceleration components and nonlinear acceleration components. Specifically, during hybrid neural network training, the instruction prefetching unit in the convolutional acceleration component obtains the opcode indicating the current computation stage, and the control unit controls the first and second read / write units to cache the corresponding types of operands in the three stages of forward propagation, backward propagation, and weight gradient update, respectively, based on the opcode. At the same time, the target computation mode of the computing array is determined, thereby completing tensor operations of the differentiated data streams in the three stages of training on the same computing array. In addition, during the forward propagation stage, the nonlinear acceleration component uses an index with a bit width smaller than the bit width of the computation result to replace the high-bit-width original data for backup, and in the backward propagation stage, it completes the calculation of the current network layer output gradient and the backward propagation update based on the backed-up index.

[0031] Therefore, this application uses opcodes to uniformly schedule the first read / write unit, the second read / write unit, and the computing array, achieving compatibility and reuse of asymmetric data flows in the three stages of forward propagation, backward propagation, and weight gradient update on the same set of physical computing resources. This overcomes the limitation of low resource utilization caused by the difficulty of coordinating the three data flows under the same set of physical resources in existing spliced ​​hardware architectures. Furthermore, by using index backups with smaller bit widths to replace the full caching of high-bit-width original data in the forward propagation process, the storage resource occupation and external memory access bandwidth overhead in the backward propagation stage are significantly reduced, effectively alleviating the memory access bandwidth bottleneck. In addition, by utilizing the event-driven characteristics of the spiking neural network model to reduce redundant computations in the edge training process, and by using the mature gradient calculation method of the artificial neural network model to ensure convergence accuracy, the two are deeply integrated. This reduces hardware resource consumption in the hybrid neural network acceleration and edge training process while ensuring the training convergence accuracy and acceleration performance of the hybrid neural network, thereby achieving low-power, low-latency edge-side intelligent computing acceleration that supports continuous learning on the edge.

[0032] Figure 1 This is a schematic diagram of the hybrid neural network accelerator provided by the present invention. This hybrid neural network accelerator is applicable to accelerated training and inference of hybrid neural networks in various edge devices, especially for low-power, low-latency edge devices requiring continuous learning capabilities.

[0033] like Figure 1 As shown, the hybrid neural network accelerator 10 includes at least a convolution acceleration unit 11 and a nonlinear acceleration unit 12. The convolution acceleration unit 11 includes an instruction prefetching unit 110, a control unit 111, a first read / write unit 112, a second read / write unit 113, and a computing array 115. The units in the convolution acceleration unit and the convolution acceleration unit and the nonlinear acceleration unit can be connected via a bus to realize data interaction and instruction reading between the units in the convolution acceleration unit and between the convolution acceleration unit and the nonlinear acceleration unit.

[0034] The instruction prefetching unit is used to obtain an opcode when training the current network layer of the hybrid neural network model to be trained in the current round of training; the opcode is used to indicate the current operation stage of the convolution acceleration component, the current operation stage includes a forward propagation stage, a back propagation stage, or a weight gradient update stage; the hybrid neural network model to be trained includes an artificial neural network model and / or a spiking neural network model; The control unit is configured to control the first read / write unit to cache the activation data corresponding to the state-aware data of the sample object input to the current network layer during the forward propagation phase and the weight gradient update phase, and to cache the output gradient returned by the next network layer of the current network layer during the back propagation phase, according to the operation code. The control unit is further configured to control the second read / write unit to cache the weight data of the current network layer during the forward propagation phase, cache the weight data after data rearrangement during the back propagation phase, and cache the output gradient during the weight gradient update phase, according to the opcode. The control unit is also configured to determine the target calculation mode based on the operation code; The computing array is used to perform tensor operations on the data output by the first read / write unit and the second read / write unit according to the target computing mode, so as to obtain the computing result of the current network layer in the current computing stage; The nonlinear acceleration component is used to perform index calculation and backup based on the calculation result during the forward propagation phase; and to calculate the output gradient of the current network layer based on the backed-up index and the output gradient returned by the next network layer during the back propagation phase, and to perform back propagation update based on the output gradient of the current network layer.

[0035] Among them, the hybrid neural network accelerator, also known as the hybrid neuromorphic accelerator core, is a dedicated hardware computing structure designed for training and inference tasks of ANN and SNN models. It can be instantiated as hardware carriers such as application-specific integrated circuits, field-programmable gate arrays, or systems-on-a-chip.

[0036] The convolution acceleration component, also known as the training-inverse isomorphic convolution acceleration kernel, employs a single acceleration kernel design to support inference and training of convolutional layers in both artificial neural network (ANN) and spiking neural network (SNN) models. Furthermore, it supports the use of the same acceleration kernel for all three data processing stages during the training of convolutional layers in both ANN and SNN models: forward propagation (FP), backward propagation (BP), and weight gradient computation (WG). This achieves hardware resource reuse for both ANN and SNN models, significantly reducing chip area and making it suitable for resource-constrained edge devices. In addition, this convolution acceleration component can employ a hybrid strategy of data rearrangement and data path reconfiguration to achieve a homomorphic acceleration kernel design that is compatible with the forward propagation, backward propagation, and weight gradient computation data flows, enabling tensor computation in all three stages of the convolutional layer training process. Furthermore, the convolution acceleration component supports configurable convolution calculation parameters, such as feature map size, weight size, stride, padding size and rules, dilation size, time step, etc. This embodiment does not specifically limit these parameters.

[0037] The convolution acceleration component includes at least an instruction prefetch unit, a control unit, a first read / write unit, a second read / write unit, and a computing array.

[0038] The instruction prefetching unit is responsible for prefetching opcodes.

[0039] The control unit is responsible for parsing the opcode and executing control operations on the first read / write unit, the second read / write unit, and the computing array based on the opcode.

[0040] The first read-write unit, also known as the operand 1 prefetch and temporary storage unit, is used for prefetching and temporarily storing the first type of operand required for tensor computation. Specifically, in the forward propagation and weight gradient update phases, the first read-write unit is used to cache the activation data corresponding to the state-aware data of the sample object input to the current network layer; in the backpropagation phase, the first read-write unit is used to cache the output gradient propagated back from the next network layer of the current network layer.

[0041] The second read-write unit, also known as the operand 2 prefetch and temporary storage unit, is used for prefetching and temporarily storing the second type of operand required for tensor computation. During the forward propagation phase, the second read-write unit caches the weight data of the current network layer; during the backpropagation phase, it caches the weight data after data rearrangement; and during the weight gradient update phase, it caches the output gradient.

[0042] The computation array, also known as the computation unit array, is the core computational component responsible for performing convolution operations. It can be configured in inner product mode or outer product mode.

[0043] The nonlinear acceleration component, also known as the configurable nonlinear acceleration kernel, efficiently supports the processing of nonlinear layers in ANNs and SNNs. It can be used to calculate activation functions during the training process of artificial neural network models, and also to calculate surrogate gradient functions during the neuron calculation and training process of spiking neural network models. That is, during model training, forward propagation uses real activation functions or neurons to participate in the calculation, while back propagation uses surrogate gradient functions or the derivatives of activation functions to pass the output gradient.

[0044] It should be noted that this nonlinear acceleration component can efficiently support the calculation of different activation functions during the training process of artificial neural network models, such as the Rectified Linear Unit (ReLU), Leaky Rectified Linear Unit (Leaky ReLU), Parametric Rectified Linear Unit (PReLU), Exponential Linear Unit (ELU), Sigmoid Function (Sigmoid), Sigmoid Weighted Linear Unit (SiLU), Hyperbolic Tangent Function (Tanh), SoftSign Function (SoftSign), Softplus Function (Softplus), and Gaussian Error Linear Unit (GeLU). It can also efficiently support the calculation of different surrogate gradient functions when spiking neural network models are trained directly using surrogate gradients, such as the Sigmoid Function (Sigmoid) and Fast Sigmoid Function (Fast). This nonlinear acceleration component offers various functions, including the sigmoid function (fast sigmoid), arctangent function (arctan), soft sign function (SoftSign), and Leaky Rectified Linear Unit (LeakyReLU), demonstrating good versatility. It can also efficiently support the forward and backward propagation computation requirements of novel activation functions or surrogate gradient functions.

[0045] The term "hybrid neural network model to be trained" refers to a neural network model that is currently in the process of parameter iterative update and has not yet converged. It can be an artificial neural network model, a spiking neural network model, or a hybrid neural network model that combines an artificial neural network model and a spiking neural network model. An artificial neural network model refers to a neural network model that relies on backpropagation algorithm and differentiable activation function for training and performs intensive computation based on a synchronous clock. A spiking neural network model refers to a neural network model that is based on an event-driven mechanism and simulates the information processing mechanism of biological neurons through discrete pulse sequences.

[0046] The term "current network layer" refers to the specific network layer currently performing forward computation, backpropagation, or weight gradient update operations during a given training iteration. The term "current training iteration" refers to a complete forward propagation, backpropagation, and weight gradient update process triggered by the current batch of training samples.

[0047] It should be noted that the forward propagation, back propagation, and weight gradient update stages of artificial neural network model training all require multiply-accumulate (MAC) operations; however, due to the event-driven nature of its spikes, spiking neural network models only require addition operations during the forward propagation and weight gradient update stages, while multiply-accumulate operations are required during the back propagation stage. Furthermore, the data processed in the forward propagation, back propagation, and weight gradient update stages of gradient descent training methods differ significantly in terms of data type, dimensionality distribution, and arrangement.

[0048] Therefore, the hybrid neural network accelerator provided in this embodiment adopts a hybrid strategy combining data rearrangement and data path reconfiguration. It uses a convolutional acceleration unit to uniformly schedule the same set of physical computing resources based on opcodes, thereby achieving compatibility with the differentiated data streams of the three stages mentioned above. Furthermore, it combines this with the index backup mechanism of the nonlinear acceleration unit to reduce memory access overhead. The specific implementation process is as follows: First, the instruction prefetching unit is responsible for prefetching the opcode. The opcode can be a machine-level instruction generated by the main controller and issued, representing the detailed data scheduling of the designed convolutional acceleration unit for neural network model training. This opcode determines whether the convolutional acceleration unit is currently operating in the forward propagation, backward propagation, or weight gradient update phase, and enables unified driving of the read / write behavior of the first and second read / write units, as well as the computation mode of the computing array.

[0049] The forward propagation phase refers to the computation phase in which activation data is calculated and output layer by layer along the network from shallow to deep; the back propagation phase refers to the computation phase in which error gradients are propagated back layer by layer along the network from deep to shallow; and the weight gradient update phase refers to the computation phase in which the update amount of weight data for each network layer is calculated based on the activation data and the error gradient.

[0050] Then, the control unit is responsible for parsing and executing the opcode. That is, after parsing the opcode, the control unit can control the read and write behavior of the first read and write unit and the second read and write unit according to the current operation stage indicated by the opcode. In other words, it controls the specific type of data to be cached by the first read and write unit and the data arrangement in the storage process.

[0051] Specifically, when the current computation stage indicated by the opcode is the forward propagation stage or the weight gradient update stage, the control unit controls the first read / write unit to cache the activation data corresponding to the state-aware data of the sample object input to the current network layer; when the current computation stage indicated by the opcode is the backpropagation stage, the control unit controls the first read / write unit to cache the output gradient returned by the next network layer of the current network layer.

[0052] The state-aware data of the sample object refers to the data collected to characterize its state attributes, such as image data collected by a Dynamic Vision Sensor (DVS), sound signals collected by an audio sensor, and brainwave data collected by a brain-computer interface. The activation data refers to the feature data generated after the state-aware data of the sample object has undergone pre-processing and activation by the current network layer, serving as the input for the convolutional layer operations of the current network layer. The next network layer refers to the adjacent network layer located after the current network layer in the forward propagation direction. The output gradient backpropagated from the next network layer refers to the error gradient calculated by the next network layer using the chain rule and propagated back to the current network layer during backpropagation, i.e., the backpropagation error.

[0053] Accordingly, when the current computation stage indicated by the opcode is the forward propagation stage, the control unit controls the second read / write unit to cache the weight data of the current network layer; when the current computation stage indicated by the opcode is the back propagation stage, the control unit controls the second read / write unit to perform data rearrangement on the weight data and then cache it; when the current computation stage indicated by the opcode is the weight gradient update stage, the control unit controls the second read / write unit to cache the output gradient returned by the next network layer.

[0054] The weight data refers to the synaptic parameters connecting neurons in the current network layer, i.e., the convolution kernel parameters. Data rearrangement refers to the channel dimension transformation and spatial rotation of the weight data to adapt to the gradient convolution operation in the backpropagation stage. Since the weights used in the forward and backward propagation stages differ in channel arrangement and spatial orientation, data rearrangement ensures that the data format fed into the computation array during backpropagation is consistent with that during forward propagation, thus resolving the inconsistency in the data path from the forward and backward propagation stages to the computation array.

[0055] It should be noted that the first read / write unit and the second read / write unit can be implemented in hardware structure by random access memory (RAM) and its peripheral read / write control logic, and they work together to achieve data reuse and reduce the external storage bandwidth usage. Their read / write behavior is controlled by opcodes.

[0056] Specifically, for the first read / write unit, the opcode can instruct the following actions: control the bus to generate read behavior, determine the data placement format according to the current computation stage and specific parameters; after the corresponding data is loaded into the random access memory, control the read operation module to read the data from the random access memory and send it to the computing array. The bus can be an Advanced eXtensible Interface (AXI) bus.

[0057] For the second read / write unit, the opcode can indicate the following actions: control the bus to generate read behavior, determine the data placement format according to the current operation stage and specific parameters, especially for the backpropagation stage, before the data enters the random access memory, the data blocks in the channel and convolution kernel (CK) dimensions need to be rotated; after the corresponding data is loaded into the random access memory, control the read operation module to read the data from the random access memory and send it to the computing array.

[0058] The RAM can be configured in two sets to support ping-pong operation. That is, while one set of RAM outputs data to the computing array for computation, the other set of RAM loads the next batch of data simultaneously. By overlapping the data loading and data computation times, the waiting idle time of the computing array is reduced, and the computing efficiency is improved.

[0059] In addition, when loading data from external storage into the first and second read / write units, the same data is loaded only once to maximize data reuse and reduce external storage bandwidth usage.

[0060] The control unit then determines the target computation mode based on the opcode. The target computation mode refers to the way the computation array organizes and operates on operands, which can be achieved through data path reconfiguration, such as inner product mode or outer product mode.

[0061] Then, the computing array performs tensor operations on the data output by the first and second read / write units according to the target computing pattern, obtaining the computation result of the current network layer at the current computing stage. The tensor operations include, but are not limited to, multiplication and summation of multidimensional matrices, vector dot products, and other operations.

[0062] In practical implementation, the computing array can employ a low-precision combination of variable data precision multipliers and adders. This low-precision combination method involves splitting large-bit-width data into multiple small-bit-width segments, and obtaining the multiplicative sum of the large-bit-width data by combining the products of these segments. This allows for configuration with different data precisions, such as half-precision floating-point (Float Point 16, FP16), brain floating-point (BF16), octet floating-point (Float Point 8, FP8), 16-bit integer (Integer 16, Int16), octet integer (Integer 8, Int8), and 4-bit integer (Integer 4, Int4). When the bit-width precision is below a certain value, i.e., in low-bit-width precision mode, the computing array can be configured with multiple parallel computation paths. This supports time-step parallel computing, multi-sample parallel computing, or multi-layer parallel computing of the model without increasing operand read bandwidth, significantly improving processing efficiency.

[0063] Then, during the forward propagation phase, the nonlinear acceleration unit performs index calculation and backup based on the computation results. Specifically, during the forward propagation phase, the nonlinear acceleration unit can obtain the corresponding index value based on the computation results, and then read the corresponding slope and intercept. It uses the multiply-accumulate unit to calculate the output result. There is no need to back up the high-bit-width original input data of the forward propagation phase; only the corresponding index needs to be backed up for use in the backward propagation phase, which can significantly reduce storage requirements.

[0064] The index refers to the number of a segment in the piecewise linear fitting function of the surrogate gradient function or activation function corresponding to the hybrid neural network model to be trained, obtained by segmenting and quantizing the result of the operation or the result after processing the operation. Its data bit width is smaller than the data bit width of the operation result. It should be noted that, at the hardware implementation level, the above-mentioned surrogate gradient function or activation function can be characterized by fitting a piecewise linear function. The piecewise linear function can be configured as a uniform piecewise linear function or a non-uniform piecewise linear function, and it is specifically characterized by slope, intercept and endpoint data.

[0065] Furthermore, during the backpropagation phase, the nonlinear acceleration component calculates the output gradient of the current network layer based on the backed-up index and the output gradient returned from the next network layer.

[0066] Specifically, the target slope can be read from the backed-up index, and multiplied by the target slope with the output gradient propagated back from the next network layer to obtain the output gradient of the current network layer. The output gradient of the current network layer refers to the error gradient propagated back from the current network layer to its predecessor along the backpropagation direction.

[0067] Then, the nonlinear acceleration component performs backpropagation updates based on the output gradient of the current network layer. The backpropagation update refers to the process of correcting the parameters of the current network layer based on the output gradient of the current network layer. It can be implemented using any available optimization algorithm such as stochastic gradient descent (SGD), adaptive moment estimation (Adam), or momentum gradient descent.

[0068] It should be noted that in the surrogate gradient direct training method, during backpropagation, if the hybrid neural network model to be trained is an artificial neural network model, the derivative value of the activation function needs to be calculated based on the data generated during forward propagation; if the hybrid neural network model to be trained is a spiking neural network model, the derivative value of the surrogate gradient function needs to be calculated based on the data generated during forward propagation. Traditional solutions require caching all the complete high-bit-width original data generated during forward propagation for backpropagation, which places a huge storage burden on the hardware and incurs significant external memory access bandwidth overhead. In this embodiment, since the surrogate gradient function or activation function has been pre-discretized into a piecewise linear function, the complex derivative calculation during backpropagation is simplified to table lookup and multiplication operations. Only the index that uniquely points to the slope value needs to be backed up; the backup of the original data is replaced by index backup. Because the bit width of the index is smaller than the bit width of the original data, the memory access pressure is effectively alleviated, and the on-chip storage resource consumption is significantly reduced.

[0069] Furthermore, the hybrid neural network accelerator provided in this application not only supports training mode but also inference mode. That is, the hybrid neural network accelerator also supports the inference mode of ANN, the inference mode of SNN and the inference mode of hybrid neural network model formed by combining ANN and SNN. The specific inference logic can be referred to the processing logic of the forward propagation stage.

[0070] For example, in the inference mode of SNN, the convolution acceleration unit and the nonlinear acceleration unit can sequentially complete the calculation of the forward propagation nodes of each network layer in the trained network model, and pass the output pulse sequence as the input data stream of the next network layer until the output pulse sequence of the last network layer is obtained. The specific calculation process can be referred to the calculation process of the forward propagation stage in the model training process, which will not be elaborated here.

[0071] After obtaining the output pulse sequence of the last layer in the trained network model, the output pulse sequence of the last layer can be decoded to obtain the recognition result that reflects the current state of the target object, that is, the state recognition result of the target object.

[0072] Therefore, the training mode and inference mode can be homogeneously integrated on the same hardware base, which enables the training phase to fully perceive and compensate for the real hardware characteristics such as quantization errors and disturbances of the underlying hardware. This fundamentally eliminates the deployment accuracy loss caused by the separate configuration mode of training and inference, and improves the actual performance of the model after deployment.

[0073] The hybrid neural network accelerator provided in this embodiment uses a control unit to control the first and second read / write units to cache data of corresponding types during the forward propagation, backpropagation, and weight gradient update stages, based on opcodes. The computing array performs tensor operations in all three stages on the same set of physical computing power, achieving unified reuse of asymmetric data streams in the three training stages. This overcomes the limitation of existing spliced ​​hardware architectures, which struggle to coordinate three data streams under the same set of physical resources, resulting in low resource utilization. Simultaneously, the nonlinear acceleration component uses an index with a bit width smaller than the bit width of the operation result to replace the original data for backup, significantly reducing the amount of memory access data and storage resource consumption during the backpropagation stage, effectively alleviating the memory access bandwidth bottleneck. Furthermore, this accelerator is compatible with the hybrid computing requirements of artificial neural network models and spiking neural network models, thus achieving low-latency, low-power hybrid neural network training acceleration with minimal hardware resource consumption and support for continuous on-device learning.

[0074] In some embodiments, the control unit is further configured to: If the current computation stage indicated by the opcode is the forward propagation stage or the backward propagation stage, the target computation mode is determined to be the inner product mode. If the current computation phase indicated by the opcode is the weight gradient update phase, the target computation mode is determined to be the outer product mode.

[0075] The inner product mode refers to the computational array multiplying the two vectors output by the first and second read / write units by their corresponding elements and summing them to obtain a scalar result.

[0076] The outer product mode refers to the computational array multiplying the two vectors output by the first and second read / write units pairwise in a Cartesian product manner to obtain a matrix result.

[0077] Optionally, after parsing the opcode, the control unit can determine the target computing mode of the computing array based on the current computing stage indicated by the opcode, and then switch the computing array to the corresponding computing organization form through data path reconfiguration.

[0078] Specifically, in the forward propagation stage, the computation array needs to perform convolution operations on the activation data and the weight data; in the backpropagation stage, the computation array needs to perform convolution operations on the output gradient back from the next network layer and the rearranged weight data; both of these stages can be mathematically reduced to vector dot product operations within a sliding window, so they can be uniformly implemented using the inner product mode.

[0079] During the weight gradient update phase, the computation array needs to calculate the update amount of the weight data based on the activation data and the output gradient returned by the next network layer. Its mathematical essence is the outer product operation of the activation vector and the gradient vector, so the outer product mode is adopted.

[0080] In this embodiment, by dynamically switching between inner product mode and outer product mode according to the current operation stage, the same computing array does not need to configure independent operation circuits for the three stages of forward propagation, backward propagation and weight gradient update. The same physical computing power can be reused to complete the differentiated tensor operations of the three stages by simply reconfiguring the data path, which further reduces the hardware logic scale and improves the utilization rate of computing resources.

[0081] like Figure 1 As shown, in some embodiments, the hybrid neural network accelerator further includes an output cache unit 114; the computing array includes at least one computing unit. Each of the aforementioned computing units is configured to perform tensor operations on each data block in the data output by the first read / write unit and the second read / write unit according to the target computing mode, obtain local computation results, and send the local computation results to the output buffer unit; The output buffer unit is used to read the historical accumulated value recorded at the storage address corresponding to the local calculation result, perform an addition operation between the local calculation result and the historical accumulated value to obtain an updated accumulated value, and write the updated accumulated value back to the storage address until all calculation loops of the current network layer in the current calculation stage are completed, and the last accumulated value written back is used as the calculation result.

[0082] The output buffer unit, also known as the output result temporary storage unit, is used to temporarily store and accumulate the intermediate results output by the computing array.

[0083] The term "computation unit" refers to the smallest operational unit in a computation array that performs basic multiplication or addition operations. A computation array can be composed of one or more computation units arranged side by side.

[0084] The data block refers to the data fragments that are divided and allocated to each computing unit from the data output by the first and second read / write units.

[0085] The so-called local operation result refers to the partial multiplication and accumulation sum obtained by a certain computing unit after performing tensor operations on the data block it is allocated.

[0086] Optionally, in convolution operations, since the current network layer often involves multiple computation loops in the channel or spatial dimensions at the current stage of operation, the result obtained in a single operation is only a part of the complete operation result. Therefore, it is necessary to gradually aggregate the results through accumulation. To this end, after completing the tensor operation on its data block, each computing unit can send the obtained local operation result along with the corresponding storage address to the output buffer unit.

[0087] The output buffer unit can read the historical accumulated value recorded at the storage address, add the result of the current local calculation to the historical accumulated value to obtain the updated accumulated value, and write the updated accumulated value back to the storage address, and repeat this process.

[0088] After all computation loops of the current network layer in the current computation stage have been completed, the output buffer unit outputs the last accumulated value written back at the storage address as the computation result of the current network layer in the current computation stage.

[0089] It should be noted that the output buffer unit can be K parallel, independent and synchronous buffer blocks in hardware structure. Each buffer block has the ability to perform read, accumulate and write-back operations on random access memory, and also has the ability to exchange data with the bus. The operations of loading, accumulating and outputting each buffer block are also uniformly organized by opcodes.

[0090] In its implementation, this computational array employs low-precision combined variable-precision multiply-accumulate units, which are essentially multiple computational units with variable data precision. This low-precision combination refers to splitting large-bit-width data into multiple small-bit-width segments.

[0091] For example, taking a computation array where both operands are 1024 bits wide, its tensor operations can be flexibly configured to any of the following data precision modes: The input data is a vector dot product with a precision of 16-bit integers (Integer 16, Int16) and a vector dimension of 16. The input data is an 8-bit integer (Integer 8, Int8) vector dot product with a vector dimension of 64, or a dot product of four input data with an Int8 precision and a vector dimension of 16. The input data is a vector dot product with a precision of four-digit integers (Integer 4, Int4) and a vector dimension of 256, or a vector dot product of 16 input data with a precision of Int4 and a vector dimension of 16. The input data is a vector dot product with a precision of half-precision floating point (FP16) and a vector dimension of 16. The input data is a vector dot product with a precision of Brain Float Point 16 (BF16) and a vector dimension of 64, or a vector dot product of four input data with a precision of BF16 and a vector dimension of 16. The input data is a vector dot product with an 8-bit floating-point precision (FP8) and a vector dimension of 256, or a vector dot product of 16 input data with an FP8 precision and a vector dimension of 16.

[0092] Multiply and accumulate 16 input data points with a precision of Int16; Multiply and accumulate 64 input data points with a precision of Int8; Multiply and accumulate 256 input data points with a precision of Int4; Multiply-accumulate 16 input data points with FP16 precision; Multiply-accumulate 64 input data points with a precision of BF16; Multiply-accumulate 256 input data points with a precision of FP8.

[0093] Furthermore, when in low bit-width precision mode, the computing array can be configured as multiple parallel computing paths, specifically as one of the following parallel modes: intra-convolutional parallel computing mode; a mode combining intra-convolutional parallel computing and multi-sample parallel computing; a mode combining intra-convolutional parallel computing and time-step parallel computing; a mode combining intra-convolutional parallel computing, time-step parallel computing, and multi-sample parallel computing; and a mode combining intra-convolutional parallel computing and multi-layer parallel computing, so as to efficiently meet the training requirements of convolutional layers of ANN, convolutional layers of SNN, and convolutional layers of hybrid ANN and SNN models.

[0094] In this embodiment, by setting up an output cache unit outside the computing array to perform read-accumulate-write-back operations on local computation results, the local computation results generated by multiple computation loops can be accumulated on-chip, avoiding frequent writing of intermediate results to external storage and then reading them back. This reduces data read and write operations between on-chip and off-chip, lowering the power consumption and memory access bandwidth usage for data read and write. At the same time, thanks to the variable data precision design of low-precision combinations and rich parallel computing modes, the computing array can efficiently meet the high-speed training requirements of convolutional layers of artificial neural network models, spiking neural network models, and their hybrid models at different precisions and dimensions without increasing external read bandwidth.

[0095] In some embodiments, the convolution acceleration component further includes a gating unit; each of the computing units includes a multiplier and an adder; the multiplier and adder can switch states through the gating unit to effectively reduce power consumption.

[0096] The gating unit is configured to control each of the computation units to perform the tensor operations in the following manner: For each data block output by the first read / write unit and the second read / write unit, if the data value of the data block is zero, the target computing unit allocated the data block is controlled to skip the tensor operation for the data block. When the data value of the data block is non-zero, the hybrid neural network model to be trained is the spiking neural network model, and the current operation stage is the forward propagation stage or the weight gradient update stage, the multiplier in the target computing unit is turned off, and the tensor operation for the data block is performed by the adder in the target computing unit. When the data value of the data block is non-zero and the hybrid neural network model to be trained is the artificial neural network model, or when the data value of the data block is non-zero, the hybrid neural network model to be trained is the spiking neural network model, and the current operation stage is the backpropagation stage, the multiplier in the target computing unit is activated, and tensor operations on the data block are performed through the adder and multiplier in the target computing unit.

[0097] Among them, the gating unit refers to the circuit module used to enable or bypass the operation of each computing unit based on the data value and model type.

[0098] The target computing unit refers to the computing unit that has been allocated corresponding data blocks.

[0099] The term "multiplier off" refers to preventing the multiplier from participating in operations through a gating circuit.

[0100] Alternatively, the control logic of the gating unit can be divided into the following three cases: Firstly, zero-value skipping is addressed. Due to the event-driven nature of spiking neural network models, their pulse data exhibits significant sparsity, and the gradient data during backpropagation also exhibits considerable sparsity. Consequently, the output data from the first and second read / write units often contains a large number of data blocks with zero values. For such data blocks, regardless of the operation they participate in, the result does not contribute to the final sum. Therefore, when the gating unit detects that a data block has a zero value, it controls the target computation unit allocated to that data block to skip the tensor operation for that data block, i.e., skip the multiplication-accumulation or addition operation at the corresponding position; if the entire data block is zero, then the entire calculation for that single operation is skipped, thereby eliminating invalid computational overhead.

[0101] Secondly, since the spiking neural network model only requires addition operations during the forward propagation and weight gradient update phases of training, the gating unit can cut off the clock or power supply network of the multiplier in the target computing unit for non-zero data, that is, turn off the multiplier and only perform tensor operations on the data block through the adder in the target computing unit, thereby saving the power consumption of the multiplier while ensuring the correctness of the operation.

[0102] Third, for artificial neural network models, the forward propagation, backward propagation, and weight gradient update all require multiplication and accumulation operations. Furthermore, spiking neural network models also require multiplication and accumulation operations during the backward propagation stage. Therefore, in this case, for data blocks with non-zero values, the gating unit can control the multiplier in the target computing unit to be turned on, and the multiplier and adder together perform multiplication and accumulation operations on the data block.

[0103] In this embodiment, the gating unit dynamically opens and closes the multiplier based on the data value and model type, thereby eliminating invalid operations with the help of the zero-value skipping mechanism. The multiplier is turned off for the stage where the spiking neural network model only needs addition. Thus, the computational needs of both artificial neural network models and spiking neural network models are taken into account on the same computing array, which effectively improves the processing speed and significantly reduces the dynamic power consumption of the edge device, thereby improving resource utilization.

[0104] In some embodiments, the second read / write unit is configured to cache the weight data after data rearrangement during the backpropagation phase as follows: Obtain the number of channels and the number of convolutional kernels of the input feature map corresponding to the weight data; Based on the number of channels and the number of convolutional kernels, the weight data is subjected to a dimension swap operation between the input channel dimension and the convolutional kernel number dimension. The weighted data after the dimension swap operation is rotated in the spatial dimension to obtain the rearranged weighted data; The rearranged weight data is cached.

[0105] Optionally, in order to solve the problem of inconsistent data paths to the computing array during the forward propagation and backward propagation stages, this embodiment provides a detailed description of the underlying addressing logic for the second read / write unit to achieve data rearrangement without additional storage overhead.

[0106] Specifically, the data rearrangement process of the second read / write unit during the backpropagation phase is as follows: First, the second read / write unit obtains the number of channels and the number of convolutional kernels of the input feature map corresponding to the weight data. Then, based on the number of channels and the number of convolutional kernels, the second read / write unit performs a dimension swap operation on the weight data between the input channel dimension and the convolutional kernel number dimension.

[0107] For example, suppose the input feature map is denoted as I, the weights as W, the height, width, and number of channels of the input feature map as H, W, and C respectively, the height, width, and number of convolutional kernels of the weights as R, S, and K respectively, and the time step as T. In standard convolution calculation, it can be calculated by iterating through seven layers from the inside out, T, C, S, R, W, H, and K.

[0108] Building upon this, if we decompose the number of channels C into the product of constants M1 and C′, and the number of convolutional kernels K into the product of constants M2 and K′, i.e., C = M1 × C′, K = M2 × K′, then the aforementioned seven-layer loop can be further expanded into a nine-layer loop traversing from the innermost to the outermost layers: T, C′, S, R, M1, W, H, K′, and M2. Furthermore, the M1 and M2 loops are moved to the innermost layer. Thus, the transpose problem between the input channel dimension C and the convolutional kernel number dimension K required in the backpropagation phase can be transformed into a transpose problem between the two dimensions M1 and M2.

[0109] Accordingly, the second read / write unit can perform dimension swapping operations on the weight data between the input channel dimension and the convolution kernel number dimension based on this conversion relationship. That is, during the process of writing data into the random access memory, the channel and convolution kernel dimensions are swapped by changing the reading order of dimensions M1 and M2.

[0110] Then, the second read / write unit performs a rotation operation on the weight data after the dimension swap operation in the spatial dimension, specifically a 180° rotation in the spatial dimension, to obtain the rearranged weight data. The rearranged weight data is then cached for the computing array to read, thereby achieving consistency of the data path from the FP stage and BP stage to the computing array and maximizing the utilization of the computing unit.

[0111] In this embodiment, by decomposing and rearranging the loop dimension, the transpose of the weights in the backpropagation stage between the channel dimension and the kernel number dimension is converted into the transpose of the innermost M1 and M2 dimensions. Thus, the data format sent to the computing array in the backpropagation stage can be kept consistent with that in the forward propagation stage by directly using the read and write address logic transformation of the second read and write unit. This achieves consistency of the data path from the forward propagation stage to the computing array with the backpropagation stage, without additional storage space overhead. It solves the data flow inconsistency problem between the FP stage and the BP stage with low overhead and ensures the utilization rate of the computing unit.

[0112] like Figure 1 As shown, in some embodiments, the hybrid neural network accelerator further includes a normalization component 13 and a pooling component 14; the normalization component and the pooling component are connected to the convolution acceleration component, the nonlinear acceleration component, and external devices such as RAM, the main controller, and other devices via a bus, so that different neural network models can interact with each other according to actual needs, and at the same time facilitate the sequential data flow between the normalization component and the pooling component and the convolution acceleration component and the nonlinear acceleration component to reduce external storage and bandwidth pressure.

[0113] The control unit is further configured to configure the data processing path between the computing array, the normalization component, the pooling component, and the nonlinear acceleration component according to the configuration parameters of the current network layer. The normalization component is used to perform batch normalization processing or layer normalization processing on the received data when the data processing path includes a normalization processing node. The pooling component is used to perform pre-downsampling or reverse upsampling processing on the received data when the data processing path includes a pooling processing node.

[0114] The normalization component, also known as the normalization kernel, is used for training and inference of the normalization layer in a neural network model. It supports batch normalization and layer normalization, and supports configurable parameters such as feature map size, batch size, and time step.

[0115] The pooling component, also known as the pooling kernel, is used for processing the pooling layers in a neural network model. It supports max pooling and average pooling, and supports configurable parameters such as feature map size, window size, stride, padding size and rules, and time step.

[0116] The configuration parameters refer to parameters used to describe the operational attributes of the current network layer, which can be compiled and generated by the main controller and then distributed. The data processing path refers to the order of data flow between the computing array, normalization unit, pooling unit, and nonlinear acceleration unit. The normalization processing node and pooling processing node refer to the links in the data processing path that need to be processed by the normalization unit and pooling unit, respectively.

[0117] Optionally, since the processing steps and their order often differ between different network layers, the control unit dynamically configures the data processing paths between the computation array, normalization unit, pooling unit, and nonlinear acceleration unit based on the configuration parameters of the current network layer. If the configured data processing path includes a normalization node, the normalization unit performs batch normalization or layer normalization on the received data to eliminate data distribution offsets. If the configured data processing path includes a pooling node, the pooling unit performs downsampling during forward propagation or upsampling during backward propagation on the received data to achieve feature dimensionality reduction or gradient dimension restoration.

[0118] In this embodiment, by introducing normalization and pooling components, and by having the control unit flexibly configure the data processing paths between the components according to the configuration parameters, a single hardware architecture can adapt to diverse network models that include different hierarchical structures such as normalization layers and pooling layers. At the same time, the intermediate data generated by the calculation of the previous component can be transmitted to the next component in a sequential data stream via the bus, avoiding frequent write-back of intermediate data to external storage and further reducing external storage and bandwidth pressure.

[0119] In some embodiments, the nonlinear acceleration component is specifically used for: In the forward propagation phase, if the data processing path does not contain normalization processing nodes and pooling processing nodes, the calculation result is used as the target feature data. If the data processing path includes a normalization processing node and / or a pooling processing node, the data output by the normalization component and / or the pooling component after processing the calculation result shall be used as the target feature data. Based on the target feature data, obtain the internal state data corresponding to the current network layer; Based on the segment number in the piecewise linear fitting function corresponding to the hybrid neural network model to be trained, the internal state data is indexed and backed up.

[0120] The target feature data refers to the input data actually received by the nonlinear acceleration component during the forward propagation phase and used for subsequent calculations. Since the data processing path can be flexibly configured according to the current network layer configuration, the sources of target feature data can be as follows: In the absence of normalization and pooling nodes in the data processing path, the nonlinear acceleration unit directly uses the computation results output by the computing array as the target feature data. When the data processing path includes a normalization processing node, the nonlinear acceleration unit uses the data output by the normalization unit after processing the calculation result as the target feature data. When the data processing path includes a pooling processing node, the nonlinear acceleration component uses the data output by the pooling component after processing the calculation result as the target feature data. When the data processing path includes normalization and pooling nodes, the nonlinear acceleration unit uses the data output by the pooling and normalization units after jointly processing the calculation results as the target feature data.

[0121] Then, the nonlinear acceleration component obtains the internal state data corresponding to the current network layer based on the target feature data. The so-called internal state data refers to the data used to characterize the internal state of the nonlinear operation of the current network layer and to locate the segment of the piecewise linear fitting function.

[0122] Then, the nonlinear acceleration component performs index calculation and backup based on the segment number in the piecewise linear fitting function corresponding to the internal state data of the hybrid neural network model to be trained. The piecewise linear fitting function is composed of several linear segments, each with a corresponding slope and intercept, and each segment is bounded by its endpoint data. The segment number is the identifier of the segment in which the internal state data falls. By using this number as an index for calculation and backup, the slope of the corresponding segment can be quickly located during the backpropagation stage.

[0123] In this embodiment, by flexibly determining the source of the target feature data based on the configuration of the data processing path, and mapping the internal state data to the number of the segment it is located in as an index for backup, not only can the nonlinear acceleration component be compatible with multiple data processing paths, but also the segment number with a smaller bit width can replace the full backup of the internal state data, thus balancing gradient calculation accuracy and on-chip storage overhead.

[0124] In some embodiments, the nonlinear acceleration component is further configured to: When the hybrid neural network model to be trained is the artificial neural network model, the target feature data is used as the internal state data; When the hybrid neural network model to be trained is the spiking neural network model, the membrane voltage of each neuron in the nonlinear layer of the current network layer at each time step is calculated based on the target feature data. The membrane voltage is used as the internal state data.

[0125] Optionally, when the hybrid neural network model to be trained is an artificial neural network model, since the artificial neural network model performs operations based on continuous activation values ​​and its nonlinear layers are activation function layers, the nonlinear acceleration component can directly use the target feature data as internal state data. In this case, index calculation involves obtaining the corresponding index value based on the target feature data, and then reading the corresponding slope and intercept during the forward propagation stage and calculating the output result of the activation function through the multiply-accumulate unit. At the same time, only this index is backed up for use in the backpropagation stage to read the slope.

[0126] When the hybrid neural network model to be trained is a spiking neural network model, since the nonlinear layers of the spiking neural network model are neuron layers simulating the electrophysiological characteristics of biological neurons, the nonlinear acceleration component needs to use the target feature data as equivalent excitation for synaptic input to calculate the membrane voltage of each neuron in the nonlinear layer of the current network layer at each time step, and use the membrane voltage as internal state data. In this case, the forward propagation stage uses real neurons for calculation, but the original membrane voltage data used for backup is replaced by its corresponding index backup. This index backup logic is consistent with the forward propagation stage of the artificial neural network model; correspondingly, the backward propagation stage also reuses the table lookup and multiplication processing logic of the backward propagation stage of the artificial neural network model. The membrane voltage refers to the potential value reflecting the activation state of the neuron.

[0127] It should be noted that the nonlinear acceleration component can instantiate the neurons of the spiking neural network model into different neuron models, including but not limited to the Integrate-and-Fire (IF) neuron model, the Leaky Integrate-and-Fire (LIF) neuron model, the Izhikevich neuron model, and the Hodgkin-Huxley (HH) neuron model.

[0128] When calculating membrane voltage, the nonlinear acceleration component can adopt the corresponding membrane voltage calculation method according to the configuration parameters of the current nonlinear layer of the network, such as time step and threshold voltage, indicating the neuron model type. Using target feature data as equivalent synaptic input current, the membrane voltage of each neuron is iteratively updated at multiple discrete time steps using a forward Euler discretization method, thereby obtaining the membrane voltage of each neuron in the nonlinear layer at each time step. The forward Euler discretization method refers to a numerical integration method that discretizes the continuous-time differential equation of the neuron according to the time step, transforming the continuous-time differential equation into a discrete-time difference equation, which facilitates hardware implementation.

[0129] In this embodiment, by using target feature data and neuron membrane voltage as internal state data for artificial neural network models and spiking neural network models respectively, the same set of nonlinear acceleration components can handle both the static activation function calculation of artificial neural network models and the spatiotemporal neuron dynamics calculation of spiking neural network models. Furthermore, both models share the same index lookup table and multiplication processing logic during the backpropagation stage, thus realizing hardware reuse for nonlinear layer processing of the two types of models.

[0130] In some embodiments, this application also provides a model training method, which is implemented based on the hybrid neural network accelerator described in any of the above embodiments.

[0131] Figure 2 This is a flowchart illustrating the model training method provided by the present invention; as shown below. Figure 2 As shown, the method includes the following steps: Step 210: Train the artificial neural network model to be trained based on the state perception data of the first sample object and the state recognition label of the first sample object.

[0132] Step 220: Initialize the spiking neural network model to be trained based on the model parameters of the trained artificial neural network model.

[0133] Step 230: Train the spiking neural network model to be trained based on the state perception data of the second sample object and the state recognition label of the second sample object to obtain the state recognition model.

[0134] Optionally, in order to learn high-precision feature extraction capabilities by utilizing the mature gradient calculation methods of artificial neural network models, gradient descent algorithms, such as adaptive moment estimation algorithm and stochastic gradient descent algorithm, can be used first to train the artificial neural network model to be trained, using the state perception data of the first sample object and the state recognition label of the first sample object as training data, until the artificial neural network model converges.

[0135] The so-called state-aware data of the first sample object refers to the data collected for the first sample object to characterize its state attributes, which is used for pre-training of the artificial neural network model. Such data can be image data collected by a dynamic visual sensor, sound signals collected by an audio sensor, or brainwave data collected by a brain-computer interface, which are event stream signals used to describe the state of the sample object. The so-called state recognition label refers to the real state label corresponding to the state-aware data. For example, when the state-aware data is facial image data, the corresponding state recognition result can reflect whether the object is an authorized user, so as to directly trigger the physical unlocking action of the underlying smart access control device or terminal device. When the state-aware data is voice data, the corresponding state recognition result can reflect the object's operational intention state, so as to convert it into physical execution instructions to control the underlying smart home device or vehicle device to perform corresponding mechanical or electrical actions.

[0136] Furthermore, after the artificial neural network model training converges, its model parameters, such as convolution kernel parameters and normalized layer parameters, are converted into weights and thresholds usable by the neurons of the spiking neural network model to be trained, thereby completing the initialization of the spiking neural network model to be trained. Parameter conversion can be achieved using methods such as firing rate-based data normalization or threshold balancing.

[0137] Furthermore, based on the needs of actual deployment scenarios, the spiking neural network model is lightly fine-tuned on the edge using real-world scenario data. Specifically, the training mode of the spiking neural network model can be switched, and the spiking neural network model to be trained can be obtained through initialization. Real-time data generated by edge sensors, namely the state perception data and state recognition labels of the second sample object, is received. The surrogate gradient algorithm is used to fine-tune and train the spiking neural network model, ultimately obtaining a state recognition model that can identify the state of an object based on its state perception data.

[0138] The model training method provided in this embodiment starts by learning dense static features through high-precision pre-training of an artificial neural network model. Then, it utilizes the low-power pulse computation of a spiking neural network model for on-device deployment and fine-tuning to learn sparse spatiotemporal dynamic features. This overcomes both the difficulty in training pure spiking neural network models and the high on-device power consumption of pure artificial neural network models. Furthermore, the joint training of ANN and SNN enables the model to learn both dense static features and sparse spatiotemporal dynamic features, enhancing the model's generalization performance. This allows the model trained in this way to have better robustness and adaptability for tasks such as video streaming and sensor fusion. At the same time, since the fine-tuning process is completed on-device, user data does not need to be uploaded to the cloud to update the model, thus balancing data privacy protection and reduced communication latency.

[0139] The following example, using a DVS-based image recognition scenario, illustrates the complete process of training and inference of the state recognition model performed by the aforementioned hybrid neural network accelerator. It should be noted that the following scenario is for illustrative purposes only and does not constitute a limitation on the scope of this application.

[0140] In this scenario, the edge device is an intelligent access control terminal deployed with a hybrid neural network acceleration system. Its goal is to identify whether the object to be identified is a legitimate user with an authorized status based on facial event stream data collected by dynamic visual sensors, and accordingly trigger the physical unlocking or locking action of the access control device. The hybrid neural network model to be trained adopts a hybrid structure combining artificial neural network models and spiking neural network models.

[0141] The first stage involves the pre-training of the artificial neural network model. In this stage, the main controller first parses the network structure of the artificial neural network model to be trained, compiles and generates the configuration parameters, control information, and operation codes for each network layer of the model, and then sends them to the hybrid neural network accelerators in the accelerator cluster configured for artificial neural network training mode. At this point, the first sample object is a collection of faces from a general face sample database; the state-aware data of the first sample object is the face image data; and the state recognition label of the first sample object is the identity authorization status annotation corresponding to that face image.

[0142] When performing the forward propagation phase on the current network layer of the artificial neural network model, the instruction prefetching unit obtains the opcode indicating the forward propagation phase. The control unit controls the first read / write unit to cache the activation data corresponding to the face image data and controls the second read / write unit to cache the weight data of the current network layer, and determines the target calculation mode as the inner product mode. The computing array performs tensor operations on the activation data and weight data in the inner product mode to obtain the calculation result. If the data processing path of the current network layer also includes normalization processing nodes and pooling processing nodes, the calculation result is processed by the normalization unit and the pooling unit in sequence and used as the target feature data. The nonlinear acceleration unit uses the target feature data as internal state data, performs index calculation and backup according to the number of the segment in the piecewise linear fitting function of the activation function, and obtains the activation data output of the network layer through multiplication and addition calculation.

[0143] During the backpropagation phase, the instruction prefetching unit acquires the opcode indicating the backpropagation phase. The control unit controls the first read / write unit to cache the output gradient returned from the next network layer and controls the second read / write unit to rearrange and cache the weight data, maintaining the inner product mode as the target computation mode. The nonlinear acceleration unit reads the corresponding target slope based on the backed-up index and multiplies it by the output gradient returned from the next network layer to obtain the output gradient of the current network layer. During the weight gradient update phase, the control unit determines the target computation mode to be the outer product mode. The computation array calculates the update amount of the weight data based on the activation data and the output gradient, thereby completing the parameter update of the artificial neural network model. The above process is executed iteratively until the artificial neural network model converges.

[0144] The second stage involves the initialization of the spiking neural network model. The main controller converts the model parameters of the trained artificial neural network model, including convolutional kernel parameters and normalized layer parameters, into weights and threshold voltages usable by the neurons of the spiking neural network model using a firing rate-based data normalization method. This initializes the spiking neural network model to be trained, and its model parameters are stored in memory.

[0145] In the third stage, the spiking neural network model is fine-tuned at the edge. The main controller switches the hybrid neural network accelerator executing the training task to spiking neural network training mode. At this time, the second sample object is the registered user of the location where the smart access control terminal is located, the state perception data of the second sample object is the real-time facial event stream data of the registered user collected by the dynamic visual sensor on the smart access control terminal, and the state recognition label of the second sample object is the identity authorization status corresponding to the registered user.

[0146] During the forward propagation phase, after acquiring the target feature data, the nonlinear acceleration component uses this target feature data as an equivalent excitation for synaptic input. It iteratively calculates the membrane voltage of each neuron in the nonlinear layer of the current network layer at each time step using a forward Euler discretization method. This membrane voltage is used as internal state data and mapped to the index of the corresponding segment for backup. Simultaneously, the membrane voltage is compared with a threshold voltage to determine whether each neuron fires a pulse, thus obtaining the output pulse sequence of the current network layer. Due to the significant sparsity of this output pulse sequence, the gating unit skips tensor operations on data blocks with zero values ​​in the data blocks output by the first and second read / write units. Furthermore, during the forward propagation phase of the spiking neural network model, the multiplier is disabled, and only the adder is used for computation, thereby reducing computational power consumption.

[0147] During the backpropagation phase, the nonlinear acceleration component reuses the lookup and multiplication logic consistent with the artificial neural network model. Based on the backed-up index, it reads the target slope and multiplies it with the output gradient returned from the next network layer to obtain the output gradient of the current network layer, thus completing the parameter update of the spiking neural network model. After several rounds of iterative fine-tuning, a state recognition model adapted to the actual lighting and user characteristics of the access control location is obtained.

[0148] The fourth stage involves the inference application of the state recognition model. After fine-tuning, the main controller fixes the network weights of the state recognition model and switches the hybrid neural network accelerator to spiking neural network inference mode. At this time, the proxy gradient correlation processing logic in the nonlinear acceleration unit is bypassed. Only the convolutional acceleration unit and the nonlinear acceleration unit sequentially complete the calculation of the forward propagation nodes of each network layer in the state recognition model, and use the output pulse sequence of the current network layer as the input data of the next network layer to stream through the bus until the output pulse sequence of the last network layer is obtained. Then, the output pulse sequence of the last network layer is decoded to obtain the identity authorization status reflecting the current state of the object to be identified, that is, the state recognition result of the target object. When the state recognition result indicates that the object to be identified is a legitimate user, the smart access control terminal is triggered to perform a physical unlocking action; otherwise, it remains locked.

[0149] Therefore, this scenario combines the facial event stream data collected by the hybrid neural network accelerator and the physical execution actions of the access control system with the dynamic visual sensor. It not only ensures the basic accuracy of facial feature extraction by pre-training the artificial neural network model, but also uses the spiking neural network model to perform low-power online fine-tuning of the locally registered user data on the edge. This allows the resulting state recognition model to adapt to the real environment of specific access control locations, achieving low-power, low-latency identity state recognition and physical execution control while protecting the privacy of user facial data.

[0150] Figure 3This is a schematic diagram of the structure of the hybrid neural network acceleration system provided by the present invention; as shown. Figure 3 As shown, in some embodiments, this application also provides a hybrid neural network acceleration system, which includes a main controller 3, a memory 4, an input device 5, a bus 2, and an accelerator cluster 1; the accelerator cluster includes at least one hybrid neural network accelerator 10 as described in any of the above embodiments.

[0151] The bus is used to establish communication links between each of the hybrid neural network accelerators, the main controller, the memory, and the input device, as well as communication links between each unit within the hybrid neural network accelerator. The main controller is used to compile and generate the configuration parameters and control information of each network layer of the hybrid neural network model to be trained, and to configure and schedule each hybrid neural network accelerator. The memory is used to store the model parameters and local dataset updated during the training process of the hybrid neural network model to be trained. The input device is used to establish an interaction channel between each of the hybrid neural network accelerators and external devices.

[0152] The hybrid neural network acceleration system can be a System on Chip (SoC) chip or a System on Chip Field Programmable Gate Array (SoC FPGA) hardware platform. This embodiment does not specifically limit it. It can support ANN training mode, ANN inference mode, SNN training mode, SNN inference mode, training mode of hybrid neural network combining ANN and SNN, and inference mode of hybrid neural network combining ANN and SNN.

[0153] The accelerator cluster consists of at least one hybrid neural network accelerator, which is used to perform the core computational work for training and inference tasks of hybrid neural network models.

[0154] The bus is used to establish communication links between the accelerator cluster and the memory, input devices, and main controller; communication links between the hybrid neural network accelerators within the accelerator cluster; and communication links between the units within each hybrid neural network accelerator. The communication links established by this bus support data interaction between different hybrid neural network models according to actual needs. Simultaneously, it supports sequential data flow between components within each hybrid neural network accelerator, allowing intermediate features generated by the current component's computation to be pipelined through the bus to the next component, thereby reducing external storage and bandwidth pressure.

[0155] The main controller, also known as the main control core, can be a general-purpose processor core such as an Advanced RISC Machine (ARM), x86, or Reduced Instruction Set Computer-V (RISC-V) architecture CPU. This main controller is responsible for parsing, compiling, and mapping the hybrid neural network model to be processed. It compiles and generates configuration parameters and control information for each network layer of the model, and configures and schedules the hybrid neural network accelerators in the accelerator cluster. The hybrid neural network model to be processed can refer to either a model to be trained (for training operations) or a trained model to perform inference operations.

[0156] It should be noted that when the accelerator cluster includes multiple hybrid neural network accelerators, the main controller can configure all of the hybrid neural network accelerators to inference mode or training mode, or configure some of them to inference mode and others to training mode, thereby enabling parallel processing or serial streaming processing of multiple accelerators according to application requirements.

[0157] Memory, also known as local memory, can be implemented by a combination of various storage media such as on-chip random access memory, off-chip dynamic random access memory (DRAM) or flash memory. It is used to store the model parameters updated during the training process of the hybrid neural network model to be trained, as well as the local dataset for local retraining.

[0158] The input device may include interface protocols such as serial port, gigabit network port, universal serial bus (USB), serial peripheral interface (SPI), and mobile industry processor interface (MIPI) to establish an interaction channel for input and output data and control commands between various hybrid neural network accelerators and various external devices such as sensors, display devices, and host computers.

[0159] The hybrid neural network acceleration system provided in this embodiment efficiently couples the reconfigurable accelerator cluster with the main controller, memory, and input device via a bus. It can not only support high-throughput inference of hybrid neural network models, but also autonomously complete closed-loop online training scheduling on the edge. This builds a complete, low-power edge-side intelligent computing platform that supports continuous learning on the edge and can be flexibly expanded according to the training and application needs of different scales of networks.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid neural network accelerator, characterized in that, include: The convolution acceleration unit includes a convolution acceleration unit and a nonlinear acceleration unit, wherein the convolution acceleration unit includes an instruction prefetching unit, a control unit, a first read / write unit, a second read / write unit, and a computing array; The instruction prefetching unit is used to obtain an opcode when training the current network layer of the hybrid neural network model to be trained in the current round of training; the opcode is used to indicate the current operation stage of the convolution acceleration component, the current operation stage includes a forward propagation stage, a back propagation stage, or a weight gradient update stage; the hybrid neural network model to be trained includes an artificial neural network model and / or a spiking neural network model; The control unit is configured to control the first read / write unit to cache the activation data corresponding to the state-aware data of the sample object input to the current network layer during the forward propagation phase and the weight gradient update phase, and to cache the output gradient returned by the next network layer of the current network layer during the back propagation phase, according to the operation code. The control unit is further configured to control the second read / write unit to cache the weight data of the current network layer during the forward propagation phase, cache the weight data after data rearrangement during the back propagation phase, and cache the output gradient during the weight gradient update phase, according to the opcode. The control unit is also configured to determine the target calculation mode based on the operation code; The computing array is used to perform tensor operations on the data output by the first read / write unit and the second read / write unit according to the target computing mode, so as to obtain the computing result of the current network layer in the current computing stage; The nonlinear acceleration component is used to perform index calculation and backup based on the calculation results during the forward propagation phase. During the backpropagation phase, the output gradient of the current network layer is calculated based on the backed-up index and the output gradient returned by the next network layer, and a backpropagation update is performed based on the output gradient of the current network layer.

2. The hybrid neural network accelerator according to claim 1, characterized in that, The control unit is also used for: If the current computation stage indicated by the opcode is the forward propagation stage or the backward propagation stage, the target computation mode is determined to be the inner product mode. If the current computation phase indicated by the opcode is the weight gradient update phase, the target computation mode is determined to be the outer product mode.

3. The hybrid neural network accelerator according to claim 1, characterized in that, The hybrid neural network accelerator also includes an output buffer unit; The computing array includes at least one computing unit; Each of the aforementioned computing units is configured to perform tensor operations on each data block in the data output by the first read / write unit and the second read / write unit according to the target computing mode, obtain local computation results, and send the local computation results to the output buffer unit; The output buffer unit is used to read the historical accumulated value recorded at the storage address corresponding to the local calculation result, perform an addition operation between the local calculation result and the historical accumulated value to obtain an updated accumulated value, and write the updated accumulated value back to the storage address until all calculation loops of the current network layer in the current calculation stage are completed, and the last accumulated value written back is used as the calculation result.

4. The hybrid neural network accelerator according to claim 3, characterized in that, The convolution acceleration component also includes a gating unit; Each of the aforementioned computing units includes a multiplier and an adder; The gating unit is configured to control each of the computation units to perform the tensor operations in the following manner: For each data block output by the first read / write unit and the second read / write unit, if the data value of the data block is zero, the target computing unit allocated the data block is controlled to skip the tensor operation for the data block. When the data value of the data block is non-zero, the hybrid neural network model to be trained is the spiking neural network model, and the current operation stage is the forward propagation stage or the weight gradient update stage, the multiplier in the target computing unit is turned off, and the tensor operation for the data block is performed by the adder in the target computing unit. When the data value of the data block is non-zero and the hybrid neural network model to be trained is the artificial neural network model, or when the data value of the data block is non-zero, the hybrid neural network model to be trained is the spiking neural network model, and the current operation stage is the backpropagation stage, the multiplier in the target computing unit is activated, and tensor operations on the data block are performed through the adder and multiplier in the target computing unit.

5. The hybrid neural network accelerator according to any one of claims 1-4, characterized in that, The second read / write unit is used to cache the weight data after data rearrangement during the backpropagation phase in the following manner: Obtain the number of channels and the number of convolutional kernels of the input feature map corresponding to the weight data; Based on the number of channels and the number of convolutional kernels, the weight data is subjected to a dimension swap operation between the input channel dimension and the convolutional kernel number dimension. The weighted data after the dimension swap operation is rotated in the spatial dimension to obtain the rearranged weighted data; The rearranged weight data is cached.

6. The hybrid neural network accelerator according to any one of claims 1-4, characterized in that, The hybrid neural network accelerator also includes a normalization component and a pooling component; The control unit is further configured to configure the data processing path between the computing array, the normalization component, the pooling component, and the nonlinear acceleration component according to the configuration parameters of the current network layer. The normalization component is used to perform batch normalization processing or layer normalization processing on the received data when the data processing path includes a normalization processing node. The pooling component is used to perform pre-downsampling or reverse upsampling processing on the received data when the data processing path includes a pooling processing node.

7. The hybrid neural network accelerator according to claim 6, characterized in that, The nonlinear acceleration component is specifically used for: In the forward propagation phase, if the data processing path does not contain normalization processing nodes and pooling processing nodes, the calculation result is used as the target feature data. If the data processing path includes a normalization processing node and / or a pooling processing node, the data output by the normalization component and / or the pooling component after processing the calculation result shall be used as the target feature data. Based on the target feature data, obtain the internal state data corresponding to the current network layer; Based on the segment number in the piecewise linear fitting function corresponding to the hybrid neural network model to be trained, the internal state data is indexed and backed up.

8. The hybrid neural network accelerator according to claim 7, characterized in that, The nonlinear acceleration component is also used for: When the hybrid neural network model to be trained is the artificial neural network model, the target feature data is used as the internal state data; When the hybrid neural network model to be trained is the spiking neural network model, the membrane voltage of each neuron in the nonlinear layer of the current network layer at each time step is calculated based on the target feature data. The membrane voltage is used as the internal state data.

9. A model training method, characterized in that, The method is implemented based on the hybrid neural network accelerator as described in any one of claims 1-8, and the method includes: The artificial neural network model to be trained is trained based on the state perception data of the first sample object and the state recognition label of the first sample object. Based on the model parameters of the trained artificial neural network model, the spiking neural network model to be trained is initialized. The spiking neural network model to be trained is trained based on the state perception data and state recognition labels of the second sample object to obtain the state recognition model.

10. A hybrid neural network acceleration system, characterized in that, It includes a main controller, memory, input device, bus, and accelerator cluster; the accelerator cluster includes at least one hybrid neural network accelerator as described in any one of claims 1-8; The bus is used to establish communication links between each of the hybrid neural network accelerators, the main controller, the memory, and the input device, as well as communication links between each unit within the hybrid neural network accelerator. The main controller is used to compile and generate the configuration parameters and control information of each network layer of the hybrid neural network model to be trained, and to configure and schedule each hybrid neural network accelerator. The memory is used to store the model parameters and local dataset updated during the training process of the hybrid neural network model to be trained. The input device is used to establish an interaction channel between each of the hybrid neural network accelerators and external devices.

Images