Data processing method and device in pulse neural network layer, and processing chip

Abstract

The embodiments of the present invention provide a method and device for processing data within a pulse neural network layer, and a processing chip, and relate to the field of data processing technology. The method for processing data within a pulse neural network layer includes: encoding the audio and video signals to be processed into an input pulse signal that lasts for multiple pulse cycles; dividing each layer of the pulse neural network into multiple network parts, each network part can be mapped in the processing chip; for each layer of the pulse neural network, mapping each network part included in the layer to the processing chip in turn, and the processing chip obtains the output pulse signal of each network part of the layer of the pulse neural network in each pulse cycle based on the input pulse signal of multiple pulse cycles. In the present invention, when the processing chip cannot complete the mapping of one layer of the pulse neural network, a single processing chip can also be used to implement streamline operations on the pulse neural network, reducing the computing power requirements for the processing chip.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for processing data within a pulse neural network layer, and a processing chip. Background Art

[0002] Research on deep neural networks (DNNs) has made rapid progress in recent years and has been initially applied. However, deep neural network algorithms usually consume a lot of computing power, which also leads to greater power consumption. For example, the classic deep convolutional network (CNN) model AlexNet requires at least 720 million multiplication operations, and generally consumes about 10 watts to 100 watts of power.

[0003] In order to improve classification accuracy, the structure of deep neural networks has become more and more complex. Currently, there are deep neural networks with more than 1,000 layers. Even at the edge, deep neural networks generally require about 50 layers. Due to the limitations of chip computing power and storage resources, when hardware acceleration of complex deep neural networks is performed, due to the limitations of chip computing resources and storage resources, it is rare to map the entire deep neural network to the chip at one time. At present, a streamlined operation method is generally used. For example, the first layer is mapped to the chip, the chip performs operations, and the weights of the second layer are prepared at the same time. After the chip finishes calculating the first layer, it performs the second layer operation. And so on, until all layers are calculated.

[0004] In recent years, the spiking neural network (SNN) has attracted the attention of academia and industry for its low power consumption and characteristics that are closer to the human brain. In a spiking neural network, the axon is the unit that receives pulses, and the neuron is the unit that sends pulses. A neuron is connected to multiple axons through dendrites, and the connection point between the dendrite and the axon is called a synapse. After the axon receives a pulse, all dendrites that have synaptic connections with this axon will receive the pulse, which in turn affects the neurons downstream of the dendrite. The neuron adds the pulses from multiple axons and accumulates them with the previous membrane voltage. If the value exceeds the threshold, it sends a pulse downstream. The pulse propagates in the spiking neural network with a 1-bit pulse. The activation frequency of the pulse is relatively low, and only addition and subtraction operations are required, without multiplication operations. The computing power consumption and power consumption are lower than those of the deep neural network. Therefore, the deep neural network can be converted into a spiking neural network by spiking, so as to fully utilize the low power consumption advantage of the spiking neural network.

[0005] However, when using spiking neural network hardware to accelerate pulsed deep neural networks, there is also a problem of insufficient resources. If the scale of the pulsed deep neural network is large, the network needs to be mapped to multiple chips, and multiple chips work in parallel, which will lead to high costs. Summary of the invention

[0006] The purpose of the present invention is to provide a method and device for processing data within a pulse neural network layer, and a processing chip. When using a pulse neural network to process an input pulse signal including multiple pulse periods, each layer of the pulse neural network is divided into multiple network parts, and then streamline operations are performed on the periods first, and then streamline operations are performed on different network parts in each layer. Therefore, when the processing chip cannot complete the mapping of one layer of the pulse neural network, a single processing chip can also be used to implement streamline operations on the pulse neural network, thereby reducing the computing power requirements for the processing chip; at the same time, it is suitable for complex pulse neural networks obtained by pulsing deep neural networks, avoiding the cost consumption caused by the need to map complex networks to multiple chips.

[0007] To achieve the above-mentioned purpose, the present invention provides a method for processing data within a pulse neural network layer, comprising: encoding the audio and video signals to be processed into an input pulse signal lasting for multiple pulse cycles; dividing each layer of the pulse neural network into multiple network parts, each network part can be mapped in a processing chip; for each layer of the pulse neural network, mapping the network parts included in the layer to the processing chip in turn, and the processing chip obtains the output pulse signal of each network part of the layer of the pulse neural network in each pulse cycle based on the input pulse signal of multiple pulse cycles.

[0008] The present invention also provides a processing chip for executing the above-mentioned data processing method within the pulse neural network layer.

[0009] The present invention also provides a data processing device within a pulse neural network layer, comprising: the above-mentioned processing chip.

[0010] In the embodiment of the present invention, the audio and video signals to be processed are first encoded into input pulse signals that last for multiple pulse cycles, and each layer of the pulse neural network is divided into multiple network parts, so that each network part can be mapped in the processing chip; when each layer of the pulse neural network is mapped to the processing chip, each network part in the layer is sequentially mapped to the processing chip, and the processing chip obtains the output pulse signals of each network part of the pulse neural network in each pulse cycle based on the input pulse signals of multiple pulse cycles. That is, when the pulse neural network is used to process the input pulse signal including multiple pulse cycles, each layer of the pulse neural network is divided into multiple network parts, and then the cycle is streamlined, and then the different network parts in each layer are streamlined. Therefore, when the processing chip cannot complete the mapping of one layer of the pulse neural network, a single processing chip can also be used to implement the streamline operation of the pulse neural network, which reduces the computing power requirements of the processing chip; at the same time, it is suitable for the complex pulse neural network obtained by pulsing the deep neural network, avoiding the cost consumption caused by the need to map the complex network to multiple chips.

[0011] In one embodiment, for each layer of the pulse neural network, each network part included in the layer is mapped to a processing chip in sequence, and the processing chip obtains the output pulse signal of each network part of the pulse neural network in the layer in each pulse period based on the input pulse signal of multiple pulse periods, including: when each network part of the first layer of the pulse neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each network part of the first layer of the pulse neural network in each pulse period according to the input pulse signal of multiple pulse periods; when each network part of the Mth layer of the pulse neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each network part of the Mth layer of the pulse neural network in each pulse period according to the output pulse signal of the M-1th layer of the pulse neural network in each pulse period, where M is an integer greater than 1.

[0012] In one embodiment, when each of the network parts of the first layer of the pulse neural network is mapped to a processing chip, the processing chip obtains the membrane voltage and output pulse signal of each of the network parts of the first layer of the pulse neural network in each of the pulse cycles according to the input pulse signals of a plurality of the pulse cycles, including: when each of the network parts of the first layer of the pulse neural network is mapped to the processing chip, in the first pulse cycle of the input pulse signal, the processing chip multiplies the input pulse signal of the first pulse cycle corresponding to the target network part by the weight of the first layer of the pulse neural network to obtain the membrane voltage of the target network part of the first layer of the pulse neural network in the first pulse cycle, the target network part being the network part currently mapped to the processing chip; based on the target network part of the first layer of the pulse neural network in the first pulse cycle The processing chip calculates the product of the input pulse signal of the Nth pulse period corresponding to the target network part and the weight of the first layer of the pulse neural network, wherein N is an integer greater than 1; calculates the sum of the product and the membrane voltage of the target network part of the first layer of the pulse neural network in the N-1th pulse period as the membrane voltage of the target network part of the first layer of the pulse neural network in the Nth pulse period; and obtains the output pulse signal of the target network part of the first layer of the pulse neural network in the Nth pulse period based on the membrane voltage of the target network part of the first layer of the pulse neural network in the Nth pulse period and the preset membrane voltage threshold.

[0013] In one embodiment, when each of the network parts of the Mth layer of the pulse neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each of the network parts of the Mth layer of the pulse neural network in each of the pulse cycles according to the output pulse signal of the M-1th layer of the pulse neural network in each of the pulse cycles, including: when each of the network parts of the Mth layer of the pulse neural network is mapped to the processing chip, in the first pulse cycle of the input pulse signal, the processing chip multiplies the output pulse signal of the M-1th layer of the pulse neural network in the first pulse cycle with the weight of the Mth layer of the pulse neural network to obtain the membrane voltage of the target network part of the Mth layer of the pulse neural network in the first pulse cycle, and the target network part is the network part currently mapped to the processing chip; based on the target network part of the Mth layer of the pulse neural network in the first pulse cycle The membrane voltage of the target network part and a preset membrane voltage threshold are used to obtain the output pulse signal of the target network part of the Mth layer of the pulse neural network in the first pulse cycle; in the Nth pulse cycle of the input pulse signal, the processing chip calculates the product of the output pulse signal of the M-1th layer of the pulse neural network and the weight of the Mth layer of the pulse neural network in the Nth pulse cycle, where N is an integer greater than 1; the sum of the product and the membrane voltage of the target network part of the Mth layer of the pulse neural network in the N-1th pulse cycle is calculated as the membrane voltage of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle; and based on the membrane voltage of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle and the preset membrane voltage threshold, the output pulse signal of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle is obtained.

[0014] In one embodiment, the expression for the number of network parts divided in each layer of the pulse neural network is: E=ceil(R / U); wherein E represents the number of network parts in each layer of the pulse neural network, R represents the number of neuron cores required for each layer of the pulse neural network, and U represents the number of neuron cores in the processing chip.

[0015] In one embodiment, each layer of a pulse neural network is divided into multiple network parts, including: when each layer in the pulse neural network is divided into multiple network parts, multiple neuron cores that meet a first preset condition among the neuron cores required for each layer are divided into the same network part, and the first preset condition is that the communication bandwidth is greater than or equal to a preset bandwidth threshold.

[0016] In one embodiment, the number of input channels required for a layer of the pulse neural network is greater than the number of input channels of a single neuron core in the processing chip; each layer of the pulse neural network is divided into multiple network parts, including: when dividing each layer in the pulse neural network into multiple network parts, multiple neuron cores that meet a second preset condition among the neuron cores required for each layer are divided into the same network part, and the second preset condition is a neuron core related to multi-bit width data inter-core transmission.

[0017] In one embodiment, the number of output channels required for a layer of the pulse neural network is greater than the number of output channels of a single neuron core in the processing chip; each layer of the pulse neural network is divided into multiple network parts, including: when dividing each layer in the pulse neural network into multiple network parts, multiple neuron cores that meet a third preset condition among the neuron cores required for each layer are divided into the same network part, and the third preset condition is used to map all output channels of a layer of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a specific flow chart of the data processing method within the pulse neural network layer according to the first embodiment of the present invention;

[0019] Figure 2 A specific flow chart of the method for processing data in a spiking neural network layer according to the second embodiment of the present invention;

[0020] Figure 3 yes Figure 3 A specific flow chart of sub-step 2031 of the method for processing data in the pulse neural network layer;

[0021] Figure 4 yes Figure 3 A specific flow chart of sub-step 2032 of the method for processing data in the spiking neural network layer;

[0022] Figure 5 A schematic diagram of a processing chip performing data processing using a data processing method within a pulse neural network layer according to a second embodiment of the present invention;

[0023] Figure 6 is a specific flow chart of the method for processing data in a pulse neural network layer according to the first mode in the third embodiment of the present invention;

[0024] Figure 7 is a specific flow chart of the method for processing data in a pulse neural network layer according to the second mode in the third embodiment of the present invention;

[0025] Figure 8is a schematic diagram of dividing the network part according to the data processing method within the pulse neural network layer of the second method in the third embodiment of the present invention;

[0026] Fig. 9 is a specific flow chart of a method for processing data in a pulse neural network layer according to a third embodiment of the present invention;

[0027] Fig.10 is a schematic diagram of dividing the network part according to the data processing method in the pulse neural network layer of the third form in the third embodiment of the present invention;

[0028] Fig.11 is a schematic diagram of a processing chip according to a fourth embodiment of the present invention;

[0029] Fig.12 is a schematic diagram of a neuron core in a processing chip according to a fourth embodiment of the present invention;

[0030] Fig.13 is a schematic diagram of a data processing device within a pulse neural network layer according to a fifth embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be described in detail with reference to the accompanying drawings to provide a clearer understanding of the purpose, features and advantages of the present invention. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0032] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0033] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0034] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0035] As used in this specification and the appended claims, the singular forms "a", "an", and "" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "or / and" unless the context clearly dictates otherwise.

[0036] In the following description, in order to clearly show the structure and working mode of the present invention, many directional words will be used for description, but the words "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", "down", etc. should be understood as convenient terms and should not be understood as restrictive terms.

[0037] The first embodiment of the present invention relates to a method for processing data in a pulse neural network layer, which is applied to a processing chip. The processing chip can use the data processing method to divide each layer of the pulse neural network into multiple network parts for mapping, and complete the processing of audio and video signals. The results of the audio and video signal processing can be used to classify the audio and video signals. Among them, the pulse neural network can also be obtained by pulsing a deep neural network.

[0038] The specific process of the data processing method in the pulse neural network layer of this embodiment is as follows Figure 1 shown.

[0039] Step 101: Encode the audio and video signals to be processed into an input pulse signal lasting for a plurality of pulse periods.

[0040] Specifically, after receiving the audio and video signal to be processed, the processing chip can use the pulse encoding module in the processing chip to encode the audio and video signal into an input pulse signal including X pulse periods, where X is an integer greater than 1, and the input pulse signal of X pulse periods can be respectively expressed as: per(1), per(2), ..., per(X), and per(N) represents the Nth pulse period in the X pulse periods, 1<N≤X. It should be noted that the present embodiment is described by taking the pulse encoding module included in the processing chip as an example, but it is not limited to this. The audio and video signal can also be encoded by an external encoding module connected to the processing chip, and then the encoded input pulse signal including X pulse periods is directly input into the processing chip.

[0041] Step 102, divide each layer of the pulse neural network into multiple network parts, each of which can be mapped in a processing chip.

[0042] Specifically, in case that the processing chip cannot fully map one layer of the pulse neural network, each layer of the pulse neural network is divided into multiple network parts, and each divided network part can be mapped in the processing chip. For example, the pulse neural network includes Y layers, Y is an integer greater than 1, and the Y layers of the pulse neural network can be represented as: layer(1), layer(2), ..., layer(Y), and layer(M) represents the Mth layer of the Y layers of the pulse neural network, 1<M≤Y; the Mth layer layer(M) of the pulse neural network is divided into E network parts, namely seg(1), seg(2), ..., seg(E).

[0043] In one example, the expression for the number of network parts that each layer of a spiking neural network is divided into is:

[0044] E = ceil(R / U);

[0045] Among them, E represents the number of network parts of each layer of the pulse neural network, R represents the number of neuron cores required for each layer of the pulse neural network, and U represents the number of neuron cores in the processing chip.

[0046] Specifically, the number of neuron cores on the processing chip is U. If the number of neuron cores required to map each layer of the pulse neural network to the processing chip is R, then each layer of the pulse neural network needs to be divided into at least E network parts, E=ceil(R / U), and the number of neuron cores required to map each network part to the processing chip is less than or equal to U.

[0047] Step 103, for each layer of the pulse neural network, the network parts included in the layer are mapped to the processing chip in sequence, and the processing chip obtains the output pulse signal of each network part of the layer of the pulse neural network in each pulse period based on the input pulse signal of multiple pulse periods.

[0048] Specifically, when using a pulse neural network to process an input pulse signal including X pulse periods, streamline operations are first performed on the periods, and then on the layers. According to the layer order of the pulse neural network, each layer of the pulse neural network is mapped to the processing chip in turn. When mapping any layer of the pulse neural network to the processing chip, each network part contained in the layer is mapped to the processing chip in turn. Based on the input pulse signal of X pulse periods, the processing chip obtains the output pulse signal of each network part of each layer of the pulse neural network in each pulse period in turn.

[0049] Subsequently, the output pulse signals of each network part of the last layer of the pulse neural network can be combined in the order of the pulse period to obtain the processing result of the audio and video signal after the pulse neural network is processed, and the processing result can be used for the classification of the audio and video signal.

[0050] This embodiment provides a method for processing data within a pulse neural network layer, firstly encoding the audio and video signal to be processed into an input pulse signal that lasts for multiple pulse cycles, and dividing each layer of the pulse neural network into multiple network parts, so that each network part can be mapped in the processing chip; when each layer of the pulse neural network is mapped to the processing chip, each network part in the layer is sequentially mapped to the processing chip, and the processing chip obtains the output pulse signal of each network part of the pulse neural network in each pulse cycle based on the input pulse signal of multiple pulse cycles. That is, when using the pulse neural network to process the input pulse signal including multiple pulse cycles, each layer of the pulse neural network is divided into multiple network parts, and then the cycle is streamlined, and then the different network parts in each layer are streamlined. Therefore, when the processing chip cannot complete the mapping of one layer of the pulse neural network, a single processing chip can also be used to realize the streamline operation of the pulse neural network, which reduces the computing power requirements of the processing chip; at the same time, it is suitable for the complex pulse neural network obtained by pulsing the deep neural network, avoiding the cost consumption caused by the need to map the complex network to multiple chips.

[0051] The second embodiment of the present invention relates to a method for processing data within a pulse neural network layer. Compared with the first embodiment, this embodiment provides a specific implementation method for mapping each network part included in each layer of the pulse neural network to a processing chip in sequence, and the processing chip obtains the output pulse signal of each network part of the layer of the pulse neural network in each pulse period based on the input pulse signal of multiple pulse periods.

[0052] The specific process of the data processing method in the pulse neural network layer of this embodiment is as follows Figure 2 shown.

[0053] Step 201, encoding the audio and video signals to be processed into input pulse signals lasting for a plurality of pulse periods. This is substantially the same as step 101 of the first embodiment and will not be described in detail here.

[0054] Step 202, each layer of the spiking neural network is divided into a plurality of network parts, each of which can be mapped in a processing chip. This is substantially the same as step 102 of the first embodiment and will not be described in detail here.

[0055] Step 203 includes the following sub-steps:

[0056] Sub-step 2031, when each network part of the first layer of the pulse neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each network part of the first layer of the pulse neural network in each pulse cycle based on the input pulse signal of multiple pulse cycles.

[0057] Please refer to Figure 3 , sub-step 2031 includes:

[0058] Sub-step 20311, when each network part of the first layer of the pulse neural network is mapped to the processing chip, in the first pulse cycle of the input pulse signal, the processing chip multiplies the input pulse signal of the first pulse cycle corresponding to the target network part by the weight of the first layer of the pulse neural network to obtain the membrane voltage of the target network part of the first layer of the pulse neural network in the first pulse cycle. The target network part is the network part currently mapped to the processing chip.

[0059] Sub-step 20312, based on the membrane voltage of the target network part of the first layer of the spiking neural network in the first pulse cycle and the preset membrane voltage threshold, obtain the output pulse signal of the target network part of the first layer of the spiking neural network in the first pulse cycle.

[0060] Sub-step 20313, in the Nth pulse period of the input pulse signal, the processing chip calculates the product of the input pulse signal of the Nth pulse period corresponding to the target network part and the weight of the first layer of the pulse neural network, where N is an integer greater than 1.

[0061] Sub-step 20314, calculates the sum of the product and the membrane voltage of the target network part of the first layer of the pulse neural network in the N-1th pulse cycle as the membrane voltage of the target network part of the first layer of the pulse neural network in the Nth pulse cycle; and based on the membrane voltage of the target network part of the first layer of the pulse neural network in the Nth pulse cycle and a preset membrane voltage threshold, obtains the output pulse signal of the target network part of the first layer of the pulse neural network in the Nth pulse cycle.

[0062] Specifically, when each network part of the first layer layer(1) of the spiking neural network is mapped to the processing chip, the processing chip reads and caches the weights of the first layer layer(1) of the spiking neural network from the external memory. Within the first pulse cycle per(1), the processing chip does not need to accumulate the membrane voltage when mapping each network part of each layer of the spiking neural network; and in the calculation of X pulse cycles, there is no need to update the weight of the layer. The following description uses the target network part seg(mb) to represent the network part of the first layer layer(1) of the spiking neural network currently mapped to the processing chip.

[0063] When the processing chip is configured in the first pulse period per(1) of the spiking neural network, the processing chip takes the input pulse signal enc_seg(mb)_per(1) of the first pulse period per(1) corresponding to the target network part seg(mb) as input, and the processing chip multiplies the input pulse signal enc_seg(mb)_per(1) by the weight of the first layer layer(1), and the product is the membrane voltage U_layer(1)_seg(mb)_per(1) of the target network part seg(mb) of the first layer layer(1) of the spiking neural network in the first pulse period per(1). A membrane voltage threshold is preset in the processing chip, and then the membrane voltage U_layer(1)_seg(mb)_per(1) is compared with the membrane voltage threshold. When the membrane voltage U_layer(1)_seg(mb)_per(1) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the first layer layer(1) in the first pulse cycle is 1; when the membrane voltage U_layer(1)_seg(mb)_per(1) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the first layer layer(1) in the first pulse cycle per(1) is 0.

[0064] When the processing chip is configured in the Nth pulse period per(N) of the spiking neural network, the processing chip takes the input pulse signal enc_seg(mb)_per(N) of the Nth pulse period per(N) corresponding to the target network part seg(mb) as input, calculates the product of the input pulse signal enc_seg(mb)_per(N) and the weight of the first layer layer(1), and adds the product to the membrane voltage U_layer(1)_seg(mb)_per(N-1) of the target network part seg(mb) of the first layer layer(1) in the N-1th pulse period per(N-1), and the sum obtained is the target network part seg(mb) of the first layer layer(1) of the spiking neural network in the Nth pulse period per(N). (mb) membrane voltage U_layer(1)_seg(mb)_per(N), and then compare the membrane voltage U_layer(1)_seg(mb)_per(N) with a preset membrane voltage threshold. When the membrane voltage U_layer(1)_seg(mb)_per(N) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the first layer layer(1) in the Nth pulse period per(N) is 1; when the membrane voltage U_layer(1)_seg(mb)_per(N) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the first layer layer(1) in the Nth pulse period per(N) is 0.

[0065] Taking N=2 as an example, when the processing chip is configured in the second pulse period per(2) of the spiking neural network, the processing chip takes the input pulse signal enc_seg(mb)_per(2) of the second pulse period per(2) corresponding to the target network part seg(mb) as input, calculates the product of the input pulse signal enc_seg(mb)_per(2) and the weight of the first layer layer(1), and adds the product to the membrane voltage U_layer(1)_seg(mb)_per(1) of the target network part seg(mb) of the first layer layer(1) in the first pulse period per(1). The sum obtained is the target network part se of the first layer layer(1) of the spiking neural network in the second pulse period per(2). g(mb), and then compare the membrane voltage U_layer(1)_seg(mb)_per(2) with a preset membrane voltage threshold. When the membrane voltage U_layer(1)_seg(mb)_per(2) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the first layer layer(1) in the second pulse period per(2) is 1; when the membrane voltage U_layer(1)_seg(mb)_per(2) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the first layer layer(1) in the second pulse period per(2) is 0.

[0066] By analogy, the above process is repeated, and the processing chip can obtain the membrane voltage and output pulse signal of each network part of the first layer (1) of the spiking neural network within X pulse cycles.

[0067] Sub-step 2032, when each network part of the Mth layer of the pulse neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each network part of the Mth layer of the pulse neural network in each pulse cycle according to the output pulse signal of the M-1th layer of the pulse neural network in each pulse cycle, where M is an integer greater than 1.

[0068] Please refer to Figure 4 , sub-step 2032 includes:

[0069] Sub-step 20321, when each network part of the Mth layer of the pulse neural network is mapped to the processing chip, in the first pulse cycle of the input pulse signal, the processing chip multiplies the output pulse signal of the M-1th layer of the pulse neural network in the first pulse cycle with the weight of the Mth layer of the pulse neural network to obtain the membrane voltage of the target network part of the Mth layer of the pulse neural network in the first pulse cycle, and the target network part is the network part currently mapped to the processing chip.

[0070] Sub-step 20322, based on the membrane voltage of the target network part of the Mth layer of the spiking neural network in the first pulse cycle and the preset membrane voltage threshold, obtain the output pulse signal of the target network part of the Mth layer of the spiking neural network in the first pulse cycle.

[0071] Sub-step 20323, in the Nth pulse cycle of the input pulse signal, the processing chip calculates the product of the output pulse signal of the M-1th layer of the pulse neural network in the Nth pulse cycle and the weight of the Mth layer of the pulse neural network, where N is an integer greater than 1.

[0072] Sub-step 20324, calculates the sum of the product and the membrane voltage of the target network part of the Mth layer of the pulse neural network in the N-1th pulse cycle as the membrane voltage of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle; and based on the membrane voltage of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle and a preset membrane voltage threshold, obtains the output pulse signal of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle.

[0073] Specifically, when each network part of the Mth layer layer(M) of the spiking neural network is mapped to the processing chip, the processing chip reads and caches the weights of the Mth layer layer(M) of the spiking neural network from the external memory. Within the first pulse cycle per(1), the processing chip does not need to accumulate the membrane voltage when mapping each layer of the spiking neural network; and in the calculation of X pulse cycles, there is no need to update the weights of the layers. The following explanation is given using the target network part seg(mb) to represent the network part of the Mth layer layer(M) of the spiking neural network currently mapped to the processing chip.

[0074] When the processing chip is configured in the first pulse cycle per(1) of the spiking neural network, the processing chip takes the output pulse signal layer(M-1)_per(1) of the M-1th layer of the spiking neural network in the first pulse cycle as input, and calculates the product of the output pulse signal layer(M-1)_per(1) and the weight of the Mth layer layer(M). The product is the membrane voltage U_layer(M)_seg(mb)_per(1) of the target network part seg(mb) of the Mth layer layer(M) of the spiking neural network in the first pulse cycle per(1), and then converts the membrane voltage U_layer(M)_seg(mb)_per(1) into the membrane voltage U_layer(M)_seg(mb)_per(1). er(M)_seg(mb)_per(1) is compared with a preset membrane voltage threshold. When the membrane voltage U_layer(M)_seg(mb)_per(1) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Mth layer layer(M) in the first pulse cycle is 1; when the membrane voltage U_layer(M)_seg(mb)_per(1) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Mth layer layer(M) in the first pulse cycle per(1) is 0.

[0075] When the processing chip is configured in the Nth pulse cycle per(N) of the spiking neural network, the processing chip takes the output pulse signal layer(M-1)_per(N) of the M-1th layer layer(M-1) of the spiking neural network in the Nth pulse cycle per(N) as input, calculates the product of the output pulse signal layer(M-1)_per(N) and the weight of the Mth layer layer(M), and adds the product to the membrane voltage U_layer(M)_seg(mb)_per(N-1) of the target network part seg(mb) of the Mth layer layer(M) in the N-1th pulse cycle per(N-1), and the sum obtained is the target network part seg(mb) of the Mth layer layer(M) of the spiking neural network in the Nth pulse cycle per(N). eg(mb), and then compare the membrane voltage U_layer(M)_seg(mb)_per(N) with a preset membrane voltage threshold. When the membrane voltage U_layer(M)_seg(mb)_per(N) is greater than or equal to the membrane voltage threshold, the output pulse signal of the pulse neural network in the target network part seg(mb) of the M layer layer(M) in the Nth pulse period per(N) is 1; when the membrane voltage U_layer(M)_seg(mb)_per(N) is less than the membrane voltage threshold, the output pulse signal of the pulse neural network in the target network part seg(mb) of the M layer layer(M) in the Nth pulse period per(N) is 0.

[0076] Taking M=2 as an example, the target network part seg(mb) of the second layer layer(2) of the spiking neural network is mapped to the processing chip. When the processing chip is configured in the first pulse period per(1) of the spiking neural network, the processing chip takes the output pulse signal layer(1)_per(1) of the first layer layer(1) of the spiking neural network in the first pulse period per(1) as input, and calculates the product of the output pulse signal layer(1)_per(1) and the weight of the second layer layer(2). This product is the membrane voltage U_layer( 2)_seg(mb)_per(1), and then compare the membrane voltage U_layer(2)_seg(mb)_per(1) with the membrane voltage threshold. When the membrane voltage U_layer(2)_seg(mb)_per(1) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the second layer layer(2) in the first pulse cycle is 1; when the membrane voltage U_layer(2)_seg(mb)_per(1) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the second layer layer(2) in the first pulse cycle per(1) is 0.

[0077] When the processing chip is configured in the Nth pulse cycle per(N) of the spiking neural network, the processing chip takes the output pulse signal layer(1)_per(N) of the first layer layer(1) of the spiking neural network in the Nth pulse cycle per(N) as input, calculates the product of the output pulse signal layer(1)_per(N) and the weight of the second layer layer(2), and adds the product to the membrane voltage U_layer(2)_seg(mb)_per(N-1) of the target network part seg(mb) of the second layer layer(2) in the N-1th pulse cycle per(N-1). The sum obtained is the target network part seg(mb) of the second layer layer(2) of the spiking neural network in the Nth pulse cycle per(N). b) the membrane voltage U_layer(2)_seg(mb)_per(N), and then compare the membrane voltage U_layer(2)_seg(mb)_per(N) with a preset membrane voltage threshold. When the membrane voltage U_layer(2)_seg(mb)_per(N) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the pulse neural network in the second layer layer(2) in the Nth pulse period per(N) is 1; when the membrane voltage U_layer(2)_seg(mb)_per(N) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the pulse neural network in the second layer layer(2) in the Nth pulse period per(N) is 0.

[0078] The above process is repeated until each network part of the last layer layer(Y) of the spiking neural network is mapped to the processing chip, and the processing chip reads and caches the weights of the Yth layer layer(Y) of the spiking neural network from the external memory; when the processing chip is configured in the first pulse period per(1) of the spiking neural network, the processing chip takes the output pulse signal layer(Y-1)_per(1) of the Y-1th layer layer(Y-1) of the spiking neural network in the first pulse period per(1) as input, and the processing chip calculates the product of the output pulse signal layer(Y-1)_per(1) and the weight of the Yth layer layer(Y), and the product is the target network part of the Yth layer layer(Y) of the spiking neural network in the first pulse period per(1). The membrane voltage U_layer(Y)_seg(mb)_per(1) of each seg(mb) is calculated, and then the membrane voltage U_layer(Y)_seg(mb)_per(1) is compared with a preset membrane voltage threshold. When the membrane voltage U_layer(Y)_seg(mb)_per(1) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Y layer layer(Y) in the first pulse cycle is 1; when the membrane voltage U_layer(Y)_seg(mb)_per(1) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Y layer layer(Y) in the first pulse cycle per(1) is 0.

[0079] When the processing chip is configured in the Nth pulse period per(N) of the spiking neural network, the processing chip takes the output pulse signal layer(Y-1)_per(N) of the Y-1th layer layer(Y-1) of the spiking neural network in the Nth pulse period per(N) as input, calculates the product of the output pulse signal layer(Y-1)_per(N) and the weight of the Yth layer layer(Y), and adds the product to the membrane voltage U_layer(Y)_seg(mb)_per(N-1) of the target network part seg(mb) of the Yth layer layer(Y) in the N-1th pulse period per(N-1), and the sum obtained is the target network part seg(mb) of the Yth layer layer(Y) of the spiking neural network in the Nth pulse period per(N). The membrane voltage U_layer(Y)_seg(mb)_per(N) of the spiking neural network is seg(mb)_per(N), and then the membrane voltage U_layer(Y)_seg(mb)_per(N) is compared with the preset membrane voltage threshold. When the membrane voltage U_layer(Y)_seg(mb)_per(N) is greater than or equal to the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Y layer layer(Y) in the Nth pulse cycle per(N) is 1; when the membrane voltage U_layer(Y)_seg(mb)_per(N) is less than the membrane voltage threshold, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Y layer layer(Y) in the Nth pulse cycle per(N) is 0. In this way, the output pulse signal of the target network part seg(mb) of the spiking neural network in the Y layer layer(Y) in each pulse cycle can be obtained.

[0080] Please refer to Figure 5, the audio and video signal is encoded to obtain an input pulse signal including two pulse periods, namely the first pulse period per(1) and the second pulse period per(2); each layer of the pulse neural network is divided into two network parts, namely seg(1) and seg(2), enc_seg(1)_per(1) represents the input pulse signal of the first pulse period per(1) corresponding to the first network part seg(1), enc_seg(2)_per(1) represents the input pulse signal of the first pulse period per(1) corresponding to the second network part seg(2), enc_seg(1)_per(2) represents the input pulse signal of the second pulse period per(2) corresponding to the first network part seg(1), and enc_seg(2)_per(2) represents the input pulse signal of the second pulse period per(2) corresponding to the second network part seg(2); t(0) to t(9) represent the operation cycle of the processing chip. The first layer layer(1) and the second layer layer(2) of the pulse neural network are taken as an example to illustrate the details as follows:

[0081] During the operation cycle t(0) of the processing chip 10, the first network part seg(1) of the first layer layer(1) of the spiking neural network is mapped to the processing chip, the weights of the first layer layer(1) of the spiking neural network are read from the external memory 20 and written into the processing chip, and enc_seg(1)_per(1) is written into the internal cache of the processing chip.

[0082] During the operation cycle t(1), the processing chip 10 performs the operation of the first network part seg(1) of the first layer layer(1) in the first pulse period per(1), writes the output pulse signal layer(1)_seg(1)_per(1) of the first network part seg(1) of the first layer layer(1) in the first pulse period per(1) into the external memory, and writes enc_seg(1)_per(2) into the internal cache of the processing chip.

[0083] During the operation cycle t(2), the processing chip 10 performs the operation of the first network part seg(1) of the first layer layer(1) in the second pulse cycle per(2), writes the output pulse signal layer(1)_seg(1)_per(2) of the first network part seg(1) of the first layer layer(1) in the second pulse cycle per(2) into the external memory, and writes enc_seg(2)_per(1) into the internal cache of the processing chip.

[0084] During the operation cycle t(3), the second network part seg(2) of the first layer layer(1) of the pulse neural network is mapped to the processing chip 10. The processing chip performs the operation of the second network part seg(2) of the first layer layer(1) during the first pulse cycle per(1), writes the output pulse signal layer(1)_seg(2)_per(1) of the second network part seg(2) of the first layer layer(1) during the first pulse cycle per(1) into the external memory, and writes enc_seg(2)_per(2) into the internal cache of the processing chip.

[0085] In the operation cycle t(4), the processing chip 10 performs the operation of the second network part seg(2) of the first layer layer(1) in the second pulse cycle per(2), and writes the output pulse signal layer(1)_seg(2)_per(2) of the second network part seg(2) of the first layer layer(1) in the second pulse cycle per(2) into the external memory. At this point, the mapping operation of the first layer layer(1) of the spiking neural network over two pulse cycles is completed.

[0086] During operation cycle t(5), the first network part seg(1) of the second layer layer(2) of the spiking neural network is mapped to the processing chip 10, the weights of the second layer layer(2) of the spiking neural network are read from the external memory 20 and written into the processing chip, and the pulse signal layer(1)_seg(1)_per(1) corresponding to the first network part seg(1) of the second layer layer(2) in the output pulse signal of the first layer layer(1) during the first pulse period per(1) is read from the external memory and written into the internal cache of the processing chip.

[0087] During the operation cycle t(6), the processing chip 10 performs the operation of the first network part seg(1) of the second layer layer(2) during the first pulse cycle per(1), writes the obtained output pulse signal layer(2)_seg(1)_per(1) of the first network part seg(1) of the second layer layer(2) during the first pulse cycle per(1) into the external memory, and reads the pulse signal layer(1)_seg(1)_per(2) corresponding to the first network part seg(1) of the second layer layer(2) from the output pulse signal of the first layer layer(1) during the second pulse cycle per(2) from the external memory and writes it into the internal cache of the processing chip.

[0088] During the operation cycle t(7), the processing chip 10 performs the operation of the first network part seg(1) of the second layer layer(2) in the second pulse period per(2), writes the obtained output pulse signal layer(2)_seg(1)_per(2) of the first network part seg(1) of the second layer layer(2) in the second pulse period per(2) into the external memory, and reads the pulse signal layer(1)_seg(2)_per(1) corresponding to the second network part seg(2) of the second layer layer(2) from the output pulse signal of the first layer layer(1) in the first pulse period per(1) from the external memory and writes it into the internal cache of the processing chip.

[0089] During the operation cycle t(8) of the processing chip 10, the second network part seg(2) of the second layer layer(2) of the spiking neural network is mapped to the processing chip, and the processing chip performs the operation of the second network part seg(2) of the second layer layer(2) during the first pulse cycle per(1), writes the obtained output pulse signal layer(2)_seg(2)_per(1) of the second network part seg(2) of the second layer layer(2) during the first pulse cycle per(1) into the external memory, and reads the pulse signal layer(1)_seg(2)_per(2) corresponding to the second network part seg(2) of the second layer layer(2) from the output pulse signal of the first layer layer(1) during the second pulse cycle per(2) from the external memory and writes it into the internal cache of the processing chip.

[0090] In the operation cycle t(9), the processing chip 10 performs the operation of the second network part seg(2) of the second layer layer(2) in the second pulse cycle per(2), and writes the output pulse signal layer(2)_seg(2)_per(2) of the second network part seg(2) of the second layer layer(2) in the second pulse cycle per(2) into the external memory. At this point, the mapping operation of the second layer layer(2) of the spiking neural network over two pulse cycles is completed.

[0091] The third embodiment of the present invention relates to a method for processing data within a pulse neural network layer. Compared with the first embodiment, this embodiment provides a specific implementation method for dividing each layer of the pulse neural network into multiple network parts.

[0092] This embodiment provides three ways to divide each layer of the spiking neural network into multiple network parts, as follows:

[0093] For the first method, please refer to Figure 6, which is a specific flow chart of the data processing method within the pulse neural network layer of the first mode.

[0094] Step 301, encoding the audio and video signals to be processed into input pulse signals lasting for a plurality of pulse periods. This is substantially the same as step 101 of the first embodiment and will not be described in detail here.

[0095] Step 302, when dividing each layer in the pulse neural network into multiple network parts, multiple neuron cores that meet a first preset condition among the neuron cores required for each layer are divided into the same network part, and the first preset condition is that the communication bandwidth is greater than or equal to a preset bandwidth threshold.

[0096] Specifically, the target layer represents the layer of the pulse neural network currently being divided. Multiple neuron cores whose inter-core communication bandwidth is greater than or equal to the preset bandwidth threshold are selected from the neuron cores required for mapping the target layer, and the selected multiple neuron cores are divided into the same network part of the target layer. The division of other network parts can be divided according to demand, and it is necessary to ensure that the number of neuron cores required for each network part obtained by dividing the target layer is less than the number of neuron cores in the processing chip. In this way, it can be ensured that the network part with high communication bandwidth requirements can be mapped to the processing chip at one time to complete the calculation, avoiding the processing chip from reading and caching data from the external memory, and reducing the communication bandwidth between the processing chip and the external memory.

[0097] Step 303, for each layer of the spiking neural network, the network parts included in the layer are sequentially mapped to the processing chip, and the processing chip obtains the output pulse signals of the network parts of the layer of the spiking neural network in each pulse period based on the input pulse signals of multiple pulse periods. This is substantially the same as step 103 of the first embodiment and will not be described in detail here.

[0098] For the second method, please refer to Figure 7 , which is a specific flow chart of the data processing method within the pulse neural network layer of the second mode.

[0099] Step 401, encoding the audio and video signals to be processed into input pulse signals lasting for a plurality of pulse periods. This is substantially the same as step 101 of the first embodiment and will not be described in detail here.

[0100] Step 402, when dividing each layer in the pulse neural network into multiple network parts, multiple neuron cores that meet the second preset condition among the neuron cores required for each layer are divided into the same network part, and the second preset condition is the neuron core related to the transmission between multi-bit width data cores.

[0101] Specifically, the target layer represents the layer of the pulse neural network currently being divided. When the number of input channels required for mapping a layer of the pulse neural network is greater than the number of input channels of a single neuron core in the processing chip, multiple neuron cores are required to jointly map all the input channels of the target layer when mapping the target layer. The neuron cores that map the input channels output multi-bit width data. The multi-bit width data output by these neuron cores need to be added in other neuron cores to form an output pulse signal. Therefore, multiple neuron cores related to the transmission between multi-bit width data cores are selected from the neuron cores required for mapping the target layer and divided into the same network part. The division of other network parts can be divided according to demand. It is necessary to ensure that the number of neuron cores required for each network part obtained by dividing the target layer is less than the number of neuron cores in the processing chip. Therefore, it can be ensured that the processing chip can complete the mapping operation of multiple neuron cores related to the transmission between multi-bit width data cores in the network part at one time, avoid the processing chip from reading and caching data in the external memory, and reduce the communication bandwidth between the processing chip and the external memory.

[0102] Please refer to Figure 8 The maximum number of input channels supported by the neuron core of the processing chip is 512, and the number of input channels required for mapping the target layer of the pulse neural network is 1024. When mapping the target layer, two neuron cores (including neuron core 0 and neuron core 1) are required to jointly map all the input channels of the target layer. Neuron core 0 is used to map the first 512 input channels, and neuron core 1 is used to map the last 512 input channels; the outputs of neuron core 0 and neuron core 1 are multi-bit width data (for example, 16-bit width data), and the outputs of neuron core 0 and neuron core 1 are added in neuron core 2, neuron core 3 and neuron core 4 respectively, and neuron core 2, neuron core 3 and neuron core 4 respectively output 1-bit output pulse data.

[0103] Step 403, for each layer of the spiking neural network, the network parts included in the layer are sequentially mapped to the processing chip, and the processing chip obtains the output pulse signals of the network parts of the layer of the spiking neural network in each pulse period based on the input pulse signals of multiple pulse periods. This is substantially the same as step 103 of the first embodiment and will not be described in detail here.

[0104] For the third method, please refer to Fig. 9 , which is a specific flow chart of the data processing method within the pulse neural network layer of the first mode.

[0105] Step 501, encoding the audio and video signals to be processed into input pulse signals lasting for a plurality of pulse periods. This is substantially the same as step 101 of the first embodiment and will not be described in detail here.

[0106] Step 502, when dividing each layer in the pulse neural network into multiple network parts, multiple neuron cores that meet the third preset condition among the neuron cores required for each layer are divided into the same network part, and the third preset condition is used to map all output channels of a layer of the neural network.

[0107] Specifically, the target layer represents the layer of the pulse neural network currently being divided. When the number of output channels required for mapping a layer of the pulse neural network is greater than the number of output channels of a single neuron core in the processing chip, multiple neuron cores are required to jointly map all the output channels of the target layer when mapping the target layer, and the input data of multiple neuron cores mapping the output channels are the same; thus, multiple neuron cores for mapping all the output channels of a layer of the neural network are selected from the neuron cores required for mapping the target layer and divided into the same network part, and the division of other network parts can be divided according to demand, and it is necessary to ensure that the number of neuron cores required for each network part obtained by dividing the target layer is less than the number of neuron cores in the processing chip. In this way, it can be ensured that the network part using multiple neuron cores with the same input data is mapped to the processing chip at one time, and when the input data is obtained from the external memory, the same input data can be multicast on the processing chip, and the mapping operation of the same input data can be completed in one mapping, reducing the data interaction between the processing chip and the external memory.

[0108] Please refer to Fig.10, the maximum number of output channels supported by the neuron core of the processing chip is 32, and the number of output channels required for mapping the target layer of the pulse neural network is 128. When mapping the target layer, 4 neuron cores (including neuron core 0, neuron core 1, neuron core 2 and neuron core 3) are required to jointly map all the output channels of the target layer. Neuron core 0 stores the convolution kernel weights corresponding to the 32 output channels from output channel 0 to output channel 31, which are used to output the results of output channel 0 to output channel 31; neuron core 1 stores the convolution kernel weights corresponding to the 32 output channels from output channel 32 to output channel 63, which are used to output the results of output channel 32 to output channel 63; neuron core 2 stores the convolution kernel weights corresponding to the 32 output channels from output channel 64 to output channel 95, which are used to output the results of output channel 64 to output channel 95; neuron core 3 stores the convolution kernel weights corresponding to the 32 output channels from output channel 96 to output channel 127, which are used to output the results of output channel 96 to output channel 127. Neuron core 0 to neuron core 3 use the same input data and divide neuron core 0 to neuron core 3 into the same network part. When the network part is mapped to the processing chip, the processing chip only needs to obtain the input data input from the external memory once, and the input data input is forwarded in sequence between neuron core 0 to neuron core 3, that is, the input data input is multicast on neuron core 0 to neuron core 3.

[0109] Step 503, for each layer of the spiking neural network, the network parts included in the layer are sequentially mapped to the processing chip, and the processing chip obtains the output pulse signals of the network parts of the layer of the spiking neural network in each pulse period based on the input pulse signals of multiple pulse periods. This is substantially the same as step 103 of the first embodiment and will not be described in detail here.

[0110] The fourth embodiment of the present invention relates to a processing chip, which is used in the method for processing data in the pulse neural network layer in any one of the first to third embodiments to divide each layer of the pulse neural network into multiple network parts for mapping, and complete the processing of audio and video signals, and the processing results after the audio and video signals are processed can be used for the classification of audio and video signals. Among them, the pulse neural network can also be obtained by pulsing the deep neural network.

[0111] Please refer to Fig.11 , the processing chip 10 includes a plurality of neuron cores 11; please refer to Fig.12 Each neuron core 11 includes: a computing unit 111, a first cache 112, a second cache 113 and a third cache 114.

[0112] The operation unit 111 is used to perform layer mapping operations of the spiking neural network.

[0113] The first cache 112 is used to cache input pulse signals, the second cache 113 is used to cache layer weights, and the third cache 114 is used to cache membrane voltages calculated by the operation unit 111 .

[0114] The fifth embodiment of the present invention relates to a data processing device in a pulse neural network layer, such as an electronic device such as a laptop computer, a desktop host, a tablet computer, etc. Fig.13 The data processing device in the pulse neural network layer includes the processing chip 10 in the third embodiment and an external memory 20 connected to the processing chip 10.

[0115] Preferred embodiments of the present invention have been described above in detail, but it should be understood that aspects of the embodiments can be modified, if necessary, to employ aspects, features and concepts of the various patents, applications and publications to provide further embodiments.

[0116] These and other changes can be made to the embodiments in light of the above detailed description.In general, in the claims, the terms used should not be considered limited to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which these claims are entitled.

Claims

1. A method for processing data within a spiking neural network layer, characterized in that: include: Encoding the audio and video signal to be processed into an input pulse signal lasting for a plurality of pulse periods; Divide each layer of the spiking neural network into multiple network parts, each of which can be mapped in a processing chip; For each layer of the spiking neural network, each network part included in the layer is sequentially mapped to a processing chip, and the processing chip obtains an output pulse signal of each network part of the layer of the spiking neural network in each pulse period based on the input pulse signal of a plurality of pulse periods; Among them, each layer of the pulse neural network is divided into multiple network parts, including: When dividing each layer in a pulse neural network into multiple network parts, multiple neuron cores that meet a first preset condition among the neuron cores required for each layer are divided into the same network part, and the first preset condition is that the communication bandwidth is greater than or equal to a preset bandwidth threshold.

2. The method for processing data in a pulse neural network layer according to claim 1, characterized in that: For each layer of the spiking neural network, each network part included in the layer is sequentially mapped to a processing chip, and the processing chip obtains an output pulse signal of each network part of the spiking neural network in each pulse period based on the input pulse signal of a plurality of pulse periods, including: When each of the network parts of the first layer of the spiking neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each of the network parts of the first layer of the spiking neural network in each of the pulse cycles according to the input pulse signals of the plurality of pulse cycles; When each of the network parts of the Mth layer of the pulse neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each of the network parts of the Mth layer of the pulse neural network in each of the pulse cycles according to the pulse signal output by the M-1th layer of the pulse neural network in each of the pulse cycles, where M is an integer greater than 1.

3. The method for processing data in a pulse neural network layer according to claim 2, characterized in that: When each of the network parts of the first layer of the spiking neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each of the network parts of the first layer of the spiking neural network in each of the pulse cycles according to the input pulse signals of the plurality of pulse cycles, including: When each of the network parts of the first layer of the spiking neural network is mapped to the processing chip, in the first pulse cycle of the input pulse signal, the processing chip multiplies the input pulse signal of the first pulse cycle corresponding to the target network part by the weight of the first layer of the spiking neural network to obtain the membrane voltage of the target network part of the first layer of the spiking neural network in the first pulse cycle, and the target network part is the network part currently mapped to the processing chip; Based on the membrane voltage of the target network part of the first layer of the spiking neural network in the first pulse cycle and a preset membrane voltage threshold, obtaining an output pulse signal of the target network part of the first layer of the spiking neural network in the first pulse cycle; In an Nth pulse cycle of the input pulse signal, the processing chip calculates the product of the input pulse signal of the Nth pulse cycle corresponding to the target network part and the weight of the first layer of the pulse neural network, where N is an integer greater than 1; Calculate the sum of the product and the membrane voltage of the target network part of the first layer of the spiking neural network in the N-1th pulse cycle as the membrane voltage of the target network part of the first layer of the spiking neural network in the Nth pulse cycle; and based on the membrane voltage of the target network part of the first layer of the spiking neural network in the Nth pulse cycle and a preset membrane voltage threshold, obtain the output pulse signal of the target network part of the first layer of the spiking neural network in the Nth pulse cycle.

4. The method for processing data in a pulse neural network layer according to claim 3, characterized in that: When each of the network parts of the Mth layer of the spiking neural network is mapped to the processing chip, the processing chip obtains the membrane voltage and output pulse signal of each of the network parts of the Mth layer of the spiking neural network in each of the pulse cycles according to the output pulse signal of the M-1th layer of the spiking neural network in each of the pulse cycles, including: When each of the network parts of the Mth layer of the spiking neural network is mapped to the processing chip, in the first pulse cycle of the input pulse signal, the processing chip multiplies the output pulse signal of the M-1th layer of the spiking neural network in the first pulse cycle by the weight of the Mth layer of the spiking neural network to obtain the membrane voltage of the target network part of the Mth layer of the spiking neural network in the first pulse cycle, and the target network part is the network part currently mapped to the processing chip; Based on the membrane voltage of the target network part of the Mth layer of the spiking neural network in the first pulse cycle and a preset membrane voltage threshold, obtaining an output pulse signal of the target network part of the Mth layer of the spiking neural network in the first pulse cycle; In the Nth pulse cycle of the input pulse signal, the processing chip calculates the product of the output pulse signal of the M-1th layer of the pulse neural network in the Nth pulse cycle and the weight of the Mth layer of the pulse neural network, where N is an integer greater than 1; Calculate the sum of the product and the membrane voltage of the target network part of the Mth layer of the pulse neural network in the N-1th pulse cycle as the membrane voltage of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle; and based on the membrane voltage of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle and a preset membrane voltage threshold, obtain the output pulse signal of the target network part of the Mth layer of the pulse neural network in the Nth pulse cycle.

5. The method for processing data in a pulse neural network layer according to claim 1, characterized in that: The expression for the number of network parts divided in each layer of the spiking neural network is: E = ceil(R / U); Among them, E represents the number of network parts of each layer of the pulse neural network, R represents the number of neuron cores required for each layer of the pulse neural network, and U represents the number of neuron cores in the processing chip.

6. The method for processing data in a pulse neural network layer according to claim 1, characterized in that: The number of input channels required for a layer of the spiking neural network is greater than the number of input channels of a single neuron core in the processing chip; Each layer of the spiking neural network is divided into multiple network parts, including: When dividing each layer in a pulse neural network into multiple network parts, multiple neuron cores that meet a second preset condition among the neuron cores required for each layer are divided into the same network part, and the second preset condition is a neuron core related to multi-bit width data inter-core transmission.

7. The method for processing data in a pulse neural network layer according to claim 1, characterized in that: The number of output channels required for a layer of the spiking neural network is greater than the number of output channels of a single neuron core in the processing chip; Each layer of the spiking neural network is divided into multiple network parts, including: When dividing each layer in the pulse neural network into multiple network parts, multiple neuron cores that meet the third preset condition among the neuron cores required for each layer are divided into the same network part, and the third preset condition is used to map all output channels of a layer of the neural network.

8. A processing chip, characterized in that: Used to execute the data processing method within a pulse neural network layer as described in any one of claims 1 to 7.

9. A data processing device within a pulse neural network layer, characterized in that: include: The processing chip as claimed in claim 8.

Images