Data processing method and device for intermediate neural network layer, and random binary neural network

Abstract

The invention discloses a data processing method and device of an intermediate neural network layer and a random binary neural network, and relates to the technical field of neural networks, and the method comprises the steps: obtaining the output data of a preorder neural network layer, and converting the output data of the preorder neural network layer into weighted summation data; based on preset sampling times, performing multiple independent Bernoulli sampling on the weighted summation data to obtain multiple sampling results; and based on multiple sampling results, determining target sampling data, and transmitting the target sampling data to a post-order neural network layer for reasoning. Through the above mode, a noise suppression mechanism is moved forward from an output layer to an intermediate layer, and multiple sampling and averaging operations are carried out in the intermediate layer, so that instantaneous noise introduced by single random sampling can be effectively smoothed, the statistical variance output by an activation value of the intermediate layer is reduced, and the stability of signal transmission between layers of a neural network is improved. The cascade amplification effect of noise in the deep network is effectively suppressed, and the overall reasoning precision of the model is improved.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to data processing methods and apparatus for intermediate neural network layers, and randomized binary neural networks. Background Technology

[0002] Currently, the training and inference of mainstream neural networks mainly revolve around high-precision (HP) algorithms. Since gradient calculation relies on high-precision analog-to-digital converters (ADCs) and digital-to-analog converters (DACs), the power consumption and chip area of ​​the hardware system increase significantly. Furthermore, this algorithm is highly sensitive to the inherent circuit noise of analog memristors and cannot effectively accommodate the non-ideal characteristics of hardware operation, severely limiting its applicability in low-power scenarios. It is often combined with binary stochastic (BS) to achieve a leap in hardware efficiency. However, existing BS algorithms still have shortcomings: the random noise suppression strategy only focuses on the output layer, reducing the impact of noise through a multi-cycle, multi-inference voting mechanism, while lacking effective control over the random binarization process of intermediate layer activation values. This leads to the accumulation of errors in intermediate layer activation values ​​during signal propagation layer by layer. Especially in deep network structures, the cascading amplification effect of noise significantly reduces the stability of feature representations, ultimately causing a deterioration in the overall model performance.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a data processing method and apparatus for intermediate neural network layers, and a randomized binary neural network, which aims to solve the technical problem in the prior art where the accumulation and amplification of noise in the intermediate layer due to single sampling in traditional randomized binary algorithms leads to a decrease in the stability of feature representation and affects the improvement of the model's inference accuracy.

[0005] To achieve the above objectives, this application provides a data processing method for an intermediate neural network layer, wherein the intermediate neural network layer is connected to a preceding neural network layer and a subsequent neural network layer in a stochastic binary neural network, and the method includes: Obtain the output data of the preceding neural network layer, and convert the output data of the preceding neural network layer into weighted summation data; Based on a preset number of samplings, the weighted summation data is subjected to multiple independent Bernoulli samplings to obtain multiple sampling results. Based on the multiple sampling results, target sampling data is determined and then passed to the subsequent neural network layer for inference.

[0006] In one embodiment, the step of performing multiple independent Bernoulli samplings on the weighted summation data based on a preset number of samplings to obtain multiple sampling results includes: The weighted summation data is mapped to probability values ​​within a preset interval; Bernoulli sampling is performed based on the probability values ​​obtained from the mapping to obtain a binary random state, and the binary random state is used as the corresponding sampling result. Update the number of samples taken; When the number of samples taken is less than the preset number of samples, return to the step of performing Bernoulli sampling based on the probability value obtained by mapping to obtain a binary random state, and use the binary random state as the corresponding sampling result; When the number of samples is equal to the preset number of samples, multiple sampling results are obtained, and the target sampling data is determined based on the multiple sampling results. The target sampling data is then passed to the subsequent neural network layer for inference.

[0007] In one embodiment, the step of determining target sampling data based on the multiple sampling results and passing the target sampling data to the subsequent neural network layer for inference includes: Based on the multiple sampling results and the preset number of samplings, the sampling mean is determined; The average value of the samples is used as the target sample data, and the target sample data is passed to the subsequent neural network layer for inference.

[0008] In one embodiment, the step of obtaining the output data of the preceding neural network layer and converting the output data of the preceding neural network layer into weighted sum data includes: Obtain the output data of the preceding neural network layer; Based on the output data of the preceding neural network layer, vector-matrix multiplication is performed in parallel to obtain the weighted summation data.

[0009] In addition, to achieve the above objectives, this application also proposes a data processing device for an intermediate neural network layer, wherein the intermediate neural network layer is connected to the preceding neural network layer and the following neural network layer in a randomized binary neural network, and the device includes a memristor cross array and a repetitive activation sampling control module. The memristor cross array is used to acquire the output data of the preceding neural network layer, convert the output data of the preceding neural network layer into weighted sum data, and transmit the weighted sum data to the repeated activation sampling control module. The repeated activation sampling control module is used to perform multiple independent Bernoulli samplings on the weighted summation data based on a preset number of samplings to obtain multiple sampling results; The repeated activation sampling control module is further configured to determine target sampling data based on the multiple sampling results, and transmit the target sampling data to the subsequent neural network layer for inference.

[0010] In one embodiment, the repeated activation sampling control module includes a random binarization circuit, a timing generator, and a sampling output unit; The timing generator is used to generate a corresponding number of clock cycles within the repeated sampling period based on the preset number of samplings, and to send a sampling signal to the random binarization circuit within each clock cycle. The random binarization circuit is also used to map the weighted summation data into probability values ​​within a preset interval when a sampling signal is received, perform Bernoulli sampling based on the mapped probability values ​​to obtain a binarized random state, and use the binarized random state as the sampling result. The sampling output unit is used to determine the target sampling data based on the multiple sampling results output by the random binarization circuit within the repeated sampling period, and to pass the target sampling data to the subsequent neural network layer for inference.

[0011] In one embodiment, the sampling output unit includes a counter and a digital-to-analog converter; The counter is used to accumulate the sampling results output by the random binarization circuit in real time during the repeated sampling period to obtain the target accumulated value. The digital-to-analog converter is used to convert the target accumulated value into an analog voltage signal corresponding to the mean of the target accumulated value, to obtain a sampled mean, and to use the sampled mean as the target sampled data.

[0012] In one embodiment, the counter is further used to acquire the sampling result output by the random binarization circuit, and to accumulate the sampling result in real time to obtain real-time accumulated data; Update the number of samples based on the number of clock cycles generated; When the number of samples already taken is less than the preset number of samples, wait for the random binarization circuit to output a new sampling result; When the number of samples taken equals the preset number of samples taken, the real-time accumulated data is determined as the target accumulated value.

[0013] In one embodiment, the memristor cross array is further used to perform vector matrix multiplication operations in parallel based on the output data of the preceding neural network layer applied to the rows, generate corresponding analog output currents in the columns, and use the analog output currents as weighted summation data.

[0014] In addition, to achieve the above objectives, this application also proposes a randomized binary neural network, wherein the intermediate neural network layer of the randomized binary neural network is provided with the data processing device for the intermediate neural network layer as described above.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data processing method of the intermediate neural network layer as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data processing method for the intermediate neural network layer as described above.

[0017] This application provides a data processing method for intermediate neural network layers. The method involves acquiring the output data of preceding neural network layers and converting it into weighted summation data. Based on a preset number of samplings, multiple independent Bernoulli samplings are performed on the weighted summation data to obtain multiple sampling results. Based on these multiple sampling results, target sampling data is determined and passed to subsequent neural network layers for inference. This application moves the noise suppression mechanism from the output layer to the intermediate layer, introducing a repeated sampling mechanism in the intermediate layer of the neural network. By performing multiple sampling and averaging operations in the intermediate layer, the instantaneous noise introduced by a single random sampling can be effectively smoothed, significantly reducing the statistical variance of the activation value output of the intermediate layer. This greatly improves the stability of signal transmission between neural network layers, effectively suppresses the cascading amplification effect of noise in deep networks, and improves the overall inference accuracy of the neural network model. This solves the technical problem of traditional random binarization algorithms where single sampling leads to noise accumulation and amplification in the intermediate layer, resulting in decreased stability of feature representation and affecting the inference accuracy of the model. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the data processing method for the intermediate neural network layer of this application; Figure 2 This is a flowchart illustrating the first embodiment of the data processing method for the intermediate neural network layer of this application; Figure 3 This is a schematic diagram of the structure of the data processing device for the intermediate neural network layer of this application in the first embodiment; Figure 4 A detailed structural schematic diagram of the data processing device for the intermediate neural network layer provided in this application; Figure 5 A schematic diagram of the hardware operation process of the data processing device for the intermediate neural network layer provided in this application.

[0021] Explanation of icon numbers: 10. Memristor cross array; 20. Repeated activation sampling control module; 201. Random binarization circuit; 202. Timing generator; 203. Sampling output unit; 2031. Counter; 2032. Digital-to-analog converter.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of this application embodiment is: to obtain the output data of the preceding neural network layer and convert the output data of the preceding neural network layer into weighted summation data; to perform multiple independent Bernoulli samplings on the weighted summation data based on a preset number of samplings to obtain multiple sampling results; to determine the target sampling data based on the multiple sampling results and to pass the target sampling data to the subsequent neural network layer for inference.

[0026] Currently, existing Black-Scholes algorithms still have shortcomings: random noise suppression strategies only focus on the output layer, reducing the impact of noise through multi-cycle, multi-inference voting mechanisms, but lack effective control over the random binarization process of intermediate layer activation values. This leads to the accumulation of errors in intermediate layer activation values ​​during signal propagation layer by layer. Especially in deep network structures, the cascading amplification effect of noise will significantly reduce the stability of feature representation, ultimately causing the overall performance of the model to deteriorate.

[0027] This application provides a solution that moves the noise suppression mechanism from the output layer to the intermediate layer. Specifically, it introduces a resampling mechanism in the intermediate layer of the neural network. By performing multiple sampling and averaging operations in the intermediate layer, it can effectively smooth the instantaneous noise introduced by a single random sampling, significantly reduce the statistical variance of the activation value output in the intermediate layer, greatly improve the stability of signal transmission between neural network layers, effectively suppress the cascading amplification effect of noise in deep networks, improve the overall inference accuracy of the neural network model, and solve the technical problem that the traditional random binarization algorithm causes noise accumulation and amplification in the intermediate layer due to single sampling, resulting in decreased stability of feature representation and affecting the inference accuracy of the model.

[0028] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, a data processing device for an intermediate neural network layer, etc. This embodiment does not specifically limit it. The following uses a data processing device for an intermediate neural network layer as an example to describe this embodiment and the following embodiments.

[0029] This application provides a data processing method for an intermediate neural network layer, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data processing method for the intermediate neural network layer of this application.

[0030] In this embodiment, the data processing method for the intermediate neural network layer includes steps S10 to S30: Step S10: Obtain the output data of the preceding neural network layer and convert the output data of the preceding neural network layer into weighted summation data; It should be noted that this embodiment applies to the intermediate neural network layer of a stochastic binary neural network. This intermediate neural network layer is above the preceding neural network layer and below the following neural network layer. The intermediate neural network layer is the neural network layer located in the middle level of the stochastic binary neural network, usually simply referred to as the intermediate layer. The preceding neural network layer is the neural network layer one level above the intermediate neural network layer, usually simply referred to as the preceding layer. The following neural network layer is the neural network layer one level below the intermediate neural network layer, usually simply referred to as the following layer. The specific functions of the preceding and following neural network layers are not specifically limited in this embodiment.

[0031] Additionally, it's important to note that in a stochastic binary neural network, if information propagates forward, the input information of each neuron is first mapped to a probability value between 0 and 1 using an activation function (such as Sigmoid or Clipped ReLU). Then, a Bernoulli process is performed based on this probability value to obtain a binary (0 or 1) random state as the neuron's output, which is then passed to the next layer. Since the neuron's output is obtained by Bernoulli sampling of a probability value, a single sampling itself has high randomness (variance). This random error accumulates and amplifies as the network propagates layer by layer, especially in deep networks, ultimately leading to a decline in the overall model performance. Therefore, this embodiment improves upon the traditional approach by taking the average of multiple independent samples to obtain a more stable activation output that is closer to the desired value.

[0032] It is understandable that the output data of the preceding neural network layer, i.e. the output of the previous layer, needs to be converted into a form that the intermediate neural network layer can process, i.e., weighted sum data. At this time, the weighted sum data can be regarded as the actual input of the intermediate neural network layer.

[0033] Step S20: Based on a preset number of samplings, perform multiple independent Bernoulli samplings on the weighted summation data to obtain multiple sampling results; It should be noted that the preset number of samplings is the pre-set number of samplings / sample quantity. Multiple independent Bernoulli samplings are performed according to the preset number of samplings. For example, let's assume the preset number of samplings is... Then perform weighted summation on the data. Each Bernoulli sampling is an independent sampling process. Each Bernoulli sampling yields a corresponding result, i.e., a sampling result. Through multiple independent Bernoulli samplings, multiple sampling results can be obtained.

[0034] In one feasible implementation, step S20 may include: mapping the weighted summation data to probability values ​​within a preset interval; performing Bernoulli sampling based on the mapped probability values ​​to obtain a binarized random state, and using the binarized random state as the corresponding sampling result; updating the number of samples already taken; when the number of samples already taken is less than a preset number of samples, returning to the step of performing Bernoulli sampling based on the mapped probability values ​​to obtain a binarized random state, and using the binarized random state as the corresponding sampling result; when the number of samples already taken is equal to the preset number of samples, obtaining multiple sampling results, performing the step of determining target sampling data based on the multiple sampling results, and passing the target sampling data to the subsequent neural network layer for inference.

[0035] It should be noted that the preset interval is [0,1], which maps the input weighted summation data to probability values ​​between 0 and 1. Then, Bernoulli sampling is performed based on this probability value to obtain a binary (0 or 1) random state, i.e., a binary random state, which is used as the sampling result.

[0036] Understandably, from a mathematical perspective, this process is the first in a neural network. Layer nodes The activation function result passed to the next layer is then processed. The process of sub-independent Bernoulli sampling can be represented as:

[0037] In the formula, It is the first Layer nodes The weighted sum of the inputs, An activation function (such as Sigmoid or ClippedReLU) can map the input data to probability values ​​between 0 and 1. Indicates the first The sampling results.

[0038] It should be understood that the number of samples already taken refers to the number of Bernoulli samples that have been performed so far. If the number of samples already taken is less than the preset number of samples, it means that another round of Bernoulli sampling is needed. If the number of samples already taken is equal to the preset number of samples, it means that the sampling is complete. A secondary independent Bernoulli sampling, at this time collecting The sampling results are used to execute step S30.

[0039] Step S30: Based on the multiple sampling results, determine the target sampling data and pass the target sampling data to the subsequent neural network layer for inference.

[0040] It should be noted that for an output probability of... neurons, to perform Sub-independent Bernoulli sampling yields a mean that is proportional to the original probability. An unbiased estimate, whose variance increases with the number of samples. The variance decreases as the variance increases (specifically, the variance decreases to its original value). This averaged output value, compared to the binary result of a single sample, can more accurately and stably represent the activation state of neurons, thus effectively suppressing noise interference in interlayer transmission.

[0041] Understandably, the target sampled data can be considered the output value of an intermediate neural network layer, which needs to be passed to subsequent neural network layers as their input. In this embodiment, the average of multiple sampling results is typically used as the target sampled data. That is, this... The sampling results are statistically averaged to calculate the final output value of the neuron, which is then passed to the next layer (the next layer). (The input of the layer).

[0042] In the specific implementation, instead of performing only one random binarization sampling on the data output from the previous layer as in traditional random binarization algorithms, the activation value is binarized during a single inference process. Second-rate( Independent Bernoulli sampling. Subsequently, this... The binarized results of each sampling are statistically averaged (the mean value is taken), and this average value is used as the final output value of the neuron and passed to the next layer of the network.

[0043] This embodiment provides a data processing method for an intermediate neural network layer. It acquires the output data of the preceding neural network layer and converts it into weighted summation data. Based on a preset number of samplings, it performs multiple independent Bernoulli samplings on the weighted summation data to obtain multiple sampling results. Based on these multiple sampling results, it determines the target sampling data and passes it to the subsequent neural network layer for inference. This embodiment moves the noise suppression mechanism from the output layer to the intermediate layer, introducing a repeated sampling mechanism in the intermediate layer of the neural network. By performing multiple sampling and averaging operations in the intermediate layer, it effectively smooths the instantaneous noise introduced by a single random sampling, significantly reduces the statistical variance of the activation value output of the intermediate layer, greatly improves the stability of signal transmission between neural network layers, effectively suppresses the cascading amplification effect of noise in deep networks, and improves the overall inference accuracy of the neural network model.

[0044] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S30 may include steps S301 to S302: Step S301: Determine the sampling mean based on the multiple sampling results and the preset number of samplings; It should be noted that the sample mean is the average of multiple sampling results calculated according to a preset number of sampling times. For example, assume that the preset number of sampling times is... Then according to The sample mean is calculated in this way.

[0045] It is understandable that for an output probability of... neurons, to perform Sub-independent Bernoulli sampling yields a mean that is proportional to the original probability. An unbiased estimate, whose variance increases with the number of samples. The variance decreases as the variance increases (specifically, the variance decreases to its original value). This averaged output value, compared to the binary result of a single sample, can more accurately and stably represent the activation state of neurons, thus effectively suppressing noise interference in interlayer transmission.

[0046] In practical implementation, this The sampling results were statistically averaged to calculate the sample mean, as shown below:

[0047] In the formula, The sample mean, Indicates the first The sampling results.

[0048] Step S302: The average value of the samples is used as the target sampled data, and the target sampled data is passed to the subsequent neural network layer for inference.

[0049] It is understandable that the calculated sample mean is passed to the next layer (the first layer). (The input of the layer).

[0050] This embodiment provides a data processing method for an intermediate neural network layer. Based on multiple sampling results and a preset number of sampling times, a sampling mean is determined. The sampling mean is used as the target sampling data and passed to subsequent neural network layers for inference. This embodiment moves the noise suppression mechanism from the output layer to the intermediate layer, that is, it introduces a repeated sampling mechanism in the intermediate layer of the neural network. By performing multiple sampling and averaging operations in the intermediate layer, the instantaneous noise introduced by a single random sampling can be effectively smoothed, the statistical variance of the activation value output of the intermediate layer can be significantly reduced, the stability of signal transmission between neural network layers can be greatly improved, the cascading amplification effect of noise in deep networks can be effectively suppressed, and the overall inference accuracy of the neural network model can be improved.

[0051] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data processing method of the intermediate neural network layer of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0052] This application also provides a data processing device for an intermediate neural network layer, as described in the embodiments below. Figure 3 , Figure 3This is a schematic diagram of the structure of the first embodiment of the data processing device for the intermediate neural network layer of this application.

[0053] In this embodiment, the data processing device for the intermediate neural network layer includes a memristor cross array 10 and a repetitive activation sampling control module 20. The intermediate neural network layer is connected to the preceding neural network layer and the following neural network layer in a randomized binary neural network.

[0054] In a specific implementation, a memristor cross array is used to acquire the output data of the preceding neural network layer, convert the output data of the preceding neural network layer into weighted sum data, and pass the weighted sum data to the repeated activation sampling control module. The repeated activation sampling control module is used to perform multiple independent Bernoulli samplings on the weighted sum data based on a preset number of samplings to obtain multiple sampling results. The repeated activation sampling control module is also used to determine the target sampling data based on the multiple sampling results and pass the target sampling data to the subsequent neural network layer for inference.

[0055] It should be noted that the intermediate neural network layer refers to the neural network layer located in the middle level of the stochastic binary neural network, usually simply referred to as the intermediate layer. The pre-order neural network layer refers to the neural network layer above the intermediate neural network layer, usually simply referred to as the upper layer. The post-order neural network layer refers to the neural network layer below the intermediate neural network layer, usually simply referred to as the lower layer. The specific functions of the pre-order and post-order neural network layers are not specifically limited in this embodiment. The output data of the pre-order neural network layer, which is the output of the upper layer, needs to be converted into a form that can be processed by the repetitive activation sampling control module, i.e., weighted sum data. At this time, the weighted sum data can be regarded as the actual input of the intermediate neural network layer.

[0056] Additionally, it should be noted that the preset sampling number is the pre-set number of samplings / sample quantity. Multiple independent Bernoulli samplings are performed according to the preset sampling number. For example, assume the preset sampling number is... Then perform weighted summation on the data. Each Bernoulli sampling is an independent sampling process. Each Bernoulli sampling yields a corresponding result, i.e., a sampling result. Through multiple independent Bernoulli samplings, multiple sampling results can be obtained.

[0057] Understandably, the target sampled data can be considered the output value of an intermediate neural network layer, which needs to be passed to subsequent neural network layers as their input. In this embodiment, the average of multiple sampling results is typically used as the target sampled data. That is, this... The sampling results are statistically averaged to obtain the target sampling data, which is then passed to the next layer (the first layer). (The input of the layer).

[0058] Furthermore, the memristor cross array 10 is also used to perform vector matrix multiplication operations in parallel based on the output data of the preceding neural network layer applied to the rows, generate corresponding analog output currents in the columns, and use the analog output currents as weighted summation data.

[0059] Understandably, the memristor cross array 10, as the core analog computing unit, can be used to store the synaptic weights of the neural network (represented by memristor conductance values). By applying voltages to the memristor array 10 in parallel, it can efficiently perform vector-matrix multiplication (VMM) operations using physical laws (Ohm's law and Kirchhoff's laws), and its output analog current is the weighted sum of the neurons, i.e., the weighted sum data. The output voltage of the previous layer is applied to the rows of the memristor cross array 10, and the memristor cross array 10 performs multiplication and addition operations in parallel, generating analog output currents representing the weighted sum results in the columns.

[0060] In its implementation, this embodiment does not perform only one random binarization sampling on the data output from the previous layer, as in traditional random binarization algorithms. Instead, it performs random binarization sampling on the activation value during a single inference process. Second-rate( Independent Bernoulli sampling. Subsequently, this... The binarized results of each sampling are statistically averaged (the mean value is taken), and this average value is used as the final output value of the neuron and passed to the next layer of the network.

[0061] In one feasible implementation, refer to Figure 4 The repeated activation sampling control module includes a random binarization circuit 201, a timing generator 202, and a sampling output unit 203. The timing generator 201 is used to generate a corresponding number of clock cycles within the repeated sampling period based on the preset number of samplings, and to send a sampling signal to the random binarization circuit 201 within each clock cycle. The random binarization circuit 201 is also used to, upon receiving the sampling signal, map the weighted summation data to probability values ​​within a preset interval, perform Bernoulli sampling based on the mapped probability values ​​to obtain a binarized random state, and use the binarized random state as the sampling result. The sampling output unit 203 is used to determine the target sampling data based on the multiple sampling results output by the random binarization circuit 201 within the repeated sampling period, and to transmit the target sampling data to the subsequent neural network layer for inference.

[0062] It should be noted that the resampling period is the implementation The complete cycle of Bernoulli sampling includes One clock cycle. The timing generator 202 can generate a precise control clock signal, and the instruction randomization binarization circuit 201 executes continuously within a short period of time. The sampling operation means that a sampling signal is sent to the random binarization circuit 201 every clock cycle.

[0063] Understandably, the preset interval is [0,1]. The random binarization circuit 201 receives the analog output current (or its converted voltage) from the memristor cross array 10, that is, it receives the weighted summation data from the memristor cross array 10. Upon receiving the sampling signal, the weighted summation data is mapped to a probability value between 0 and 1, and then Bernoulli sampling is performed based on this probability value to obtain a binarized (0 or 1) random state, that is, a binarized random state, which is used as the sampling result.

[0064] Further reference Figure 4 The sampling output unit 203 includes a counter 2031 and a digital-to-analog converter 2032. The counter 2031 is used to accumulate the sampling results output by the random binarization circuit 201 in real time during the repeated sampling period to obtain a target accumulated value. The digital-to-analog converter 2032 is used to convert the target accumulated value into an analog voltage signal corresponding to the average value of the target accumulated value to obtain a sampling average value, and use the sampling average value as the target sampling data.

[0065] It is understandable that within a complete repetitive sampling period (controlled by timing generator 202) (one clock cycle), continuously outputting the random binarization circuit 201. The binarized results (0 or 1) are accumulated in real time, and the final value is the target accumulated value.

[0066] It should be understood that the digital-to-analog converter 2032 in this embodiment is a low-precision digital-to-analog converter. After all sampling is completed, the target accumulated value (ranging from 0 to 1) stored in counter 2031 Integers between 0 and 1 are directly fed into a low-precision digital-to-analog converter (D / A). This D / A converter converts the average of the target accumulated value into an analog voltage signal, i.e., the sampled average, as the input to the next layer. Where, if Then only a 4-bit digital-to-analog converter (DAC) is needed.

[0067] It should be noted that for an output probability of... neurons, to perform Sub-independent Bernoulli sampling yields a mean that is proportional to the original probability. An unbiased estimate, whose variance increases with the number of samples. The variance decreases as the variance increases (specifically, the variance decreases to 1 / 3 of its original value). This averaged output value, compared to the binary result of a single sample, can more accurately and stably represent the activation state of neurons, thus effectively suppressing noise interference in interlayer transmission.

[0068] It is understood that the counter 2031 is also used to acquire the sampling result output by the random binarization circuit 201, accumulate the sampling result in real time to obtain real-time accumulated data; update the number of samples based on the number of clock cycles generated; wait for the random binarization circuit 201 to output a new sampling result when the number of samples is less than the preset number of samples; and determine the real-time accumulated data as the target accumulated value when the number of samples is equal to the preset number of samples.

[0069] It should be noted that real-time accumulated data refers to the value obtained through real-time accumulation, and the number of samples taken refers to the number of Bernoulli samples that have been performed so far, which can be determined based on the number of clock cycles generated. If the number of samples taken is less than the preset number of samples, it means that another round of Bernoulli sampling is needed; if the number of samples taken is equal to the preset number of samples, it means that the process has been completed. The next independent Bernoulli sampling, at which point the real-time accumulated data is... The cumulative value of the sampling results is the target cumulative value.

[0070] In the specific implementation, refer to Figure 5 The output voltage from the previous layer is applied to the rows of the memristor cross array. The array performs multiplication and addition operations in parallel, generating an analog output current representing the weighted sum in the columns. This analog current is sent to the repetitive sampling control module. The timing generator starts, controlling the random binarization circuit to... Within one clock cycle, the input analog signal is continuously processed. Each independent random binarization operation outputs a digital code stream consisting of "0"s and "1"s. A counter synchronously accumulates this code stream to obtain the final count value. After each cycle, the counter obtains a stable integer accumulated value. This accumulated value is fed into a low-precision DAC. The DAC converts the accumulated value into a corresponding analog voltage, which is then used as the input to the next layer of the network, thus completing a stable and noise-reduced inter-layer signal transfer.

[0071] For example, assuming the analog signal output by the memristor array makes the probability of the random binarization circuit outputting "1" p=0.7, then the circuit needs to be driven to perform 16 consecutive samplings. Ideally, the final value of the counter is 16×0.7≈11. This number "11" is directly fed into a 4-bit DAC and converted into a more accurate analog input voltage required by the next layer.

[0072] Understandably, traditional high-precision solutions require high-power, large-area high-precision ADCs to quantize simulation results. This embodiment completely eliminates the need for an ADC, requiring only a low-precision DAC, significantly reducing hardware power consumption, cost, and area. Traditional random binarization schemes directly transmit a single binary signal (0 or 1) in the intermediate layer. Although this eliminates the need for an ADC and DAC, the signal fluctuations are extremely large, leading to severe accuracy loss. This embodiment introduces a low-precision DAC to transmit a multi-bit, more accurate average signal, achieving a significant improvement in neural network inference accuracy and overall stability at a minimal hardware cost.

[0073] This embodiment provides a data processing device for an intermediate neural network layer. The noise suppression mechanism is moved from the output layer to the intermediate layer. That is, a repeated sampling mechanism is introduced into the intermediate layer of the neural network. By performing multiple sampling and averaging operations in the intermediate layer, the instantaneous noise introduced by a single random sampling can be effectively smoothed, the statistical variance of the activation value output of the intermediate layer can be significantly reduced, the stability of signal transmission between neural network layers can be greatly improved, the cascading amplification effect of noise in deep networks can be effectively suppressed, and the overall inference accuracy of the neural network model can be improved. In addition, it has stronger robustness to the inherent non-ideal characteristics of memristor hardware (such as noise and fluctuations), so that while maintaining the advantages of high-efficiency hardware implementation of the BS algorithm (such as low power consumption and small area), its performance can be significantly improved.

[0074] The data processing apparatus for the intermediate neural network layer provided in this application employs the data processing method for the intermediate neural network layer in the above embodiments. This method can solve the technical problem that traditional random binarization algorithms suffer from noise accumulation and amplification in the intermediate layer due to single sampling, leading to decreased stability of feature representation and affecting the improvement of model inference accuracy. Other technical features in the data processing apparatus for the intermediate neural network layer are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0075] This application also provides a randomized binary neural network, in which the intermediate neural network layer is equipped with the data processing device for the intermediate neural network layer provided in the above embodiments. This solves the technical problem that traditional randomized binary algorithms suffer from noise accumulation and amplification in the intermediate layer due to single sampling, leading to decreased stability of feature representation and affecting the improvement of model inference accuracy. Compared with the prior art, the beneficial effects of the randomized binary neural network provided in this application are the same as those provided in the above embodiments, and other technical features in the randomized binary neural network are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0076] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0077] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0078] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data processing method of the intermediate neural network layer in the above embodiments.

[0079] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0080] The aforementioned computer-readable storage medium may be included in a data processing device for an intermediate neural network layer; or it may exist independently and not be assembled into a data processing device for an intermediate neural network layer.

[0081] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the data processing device of the intermediate neural network layer, the data processing device of the intermediate neural network layer causes the following: to acquire the output data of the preceding neural network layer and convert the output data of the preceding neural network layer into weighted sum data; to perform multiple independent Bernoulli samplings on the weighted sum data based on a preset number of samplings to obtain multiple sampling results; and to determine the target sampling data based on the multiple sampling results and pass the target sampling data to the subsequent neural network layer for inference.

[0082] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0085] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the data processing method of the intermediate neural network layer described above, and is capable of solving the technical problem of data processing in the intermediate neural network layer. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data processing method of the intermediate neural network layer provided in the above embodiments, and will not be repeated here.

[0086] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method of the intermediate neural network layer as described above.

[0087] The computer program product provided in this application can solve the technical problem of data processing in intermediate neural network layers. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the data processing method for intermediate neural network layers provided in the above embodiments, and will not be repeated here.

[0088] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data processing method for an intermediate neural network layer, wherein the intermediate neural network layer is a stochastic binary neural network that connects to a preceding neural network layer and a subsequent neural network layer, characterized in that, The method includes: Obtain the output data of the preceding neural network layer, and convert the output data of the preceding neural network layer into weighted summation data; Based on a preset number of samplings, the weighted summation data is subjected to multiple independent Bernoulli samplings to obtain multiple sampling results. Based on the multiple sampling results, target sampling data is determined and then passed to the subsequent neural network layer for inference.

2. The method as described in claim 1, characterized in that, The step of performing multiple independent Bernoulli samplings on the weighted summation data based on a preset number of samplings to obtain multiple sampling results includes: The weighted summation data is mapped to probability values ​​within a preset interval; Bernoulli sampling is performed based on the probability values ​​obtained from the mapping to obtain a binary random state, and the binary random state is used as the corresponding sampling result. Update the number of samples taken; When the number of samples taken is less than the preset number of samples, return to the step of performing Bernoulli sampling based on the probability value obtained by mapping to obtain a binary random state, and use the binary random state as the corresponding sampling result; When the number of samples is equal to the preset number of samples, multiple sampling results are obtained, and the target sampling data is determined based on the multiple sampling results. The target sampling data is then passed to the subsequent neural network layer for inference.

3. The method as described in claim 1, characterized in that, The step of determining the target sampling data based on the multiple sampling results and passing the target sampling data to the subsequent neural network layer for inference includes: Based on the multiple sampling results and the preset number of samplings, the sampling mean is determined; The average value of the samples is used as the target sample data, and the target sample data is passed to the subsequent neural network layer for inference.

4. A data processing apparatus for an intermediate neural network layer, wherein the intermediate neural network layer is connected to a preceding neural network layer and a subsequent neural network layer in a stochastic binary neural network, characterized in that, The device includes a memristor cross array and a repetitive activation sampling control module; The memristor cross array is used to acquire the output data of the preceding neural network layer, convert the output data of the preceding neural network layer into weighted sum data, and transmit the weighted sum data to the repeated activation sampling control module. The repeated activation sampling control module is used to perform multiple independent Bernoulli samplings on the weighted summation data based on a preset number of samplings to obtain multiple sampling results; The repeated activation sampling control module is further configured to determine target sampling data based on the multiple sampling results, and transmit the target sampling data to the subsequent neural network layer for inference.

5. The apparatus as described in claim 4, characterized in that, The repeated activation sampling control module includes a randomization circuit, a timing generator, and a sampling output unit. The timing generator is used to generate a corresponding number of clock cycles within the repeated sampling period based on the preset number of samplings, and to send a sampling signal to the random binarization circuit within each clock cycle. The random binarization circuit is also used to map the weighted summation data into probability values ​​within a preset interval when a sampling signal is received, perform Bernoulli sampling based on the mapped probability values ​​to obtain a binarized random state, and use the binarized random state as the sampling result. The sampling output unit is used to determine the target sampling data based on the multiple sampling results output by the random binarization circuit within the repeated sampling period, and to pass the target sampling data to the subsequent neural network layer for inference.

6. The apparatus as claimed in claim 5, characterized in that, The sampling output unit includes a counter and a digital-to-analog converter; The counter is used to accumulate the sampling results output by the random binarization circuit in real time during the repeated sampling period to obtain the target accumulated value. The digital-to-analog converter is used to convert the target accumulated value into an analog voltage signal corresponding to the mean of the target accumulated value, to obtain a sampled mean, and to use the sampled mean as the target sampled data.

7. The apparatus as claimed in claim 6, characterized in that, The counter is also used to acquire the sampling result output by the random binarization circuit, and to accumulate the sampling result in real time to obtain real-time accumulated data; Update the number of samples based on the number of clock cycles generated; When the number of samples already taken is less than the preset number of samples, wait for the random binarization circuit to output a new sampling result; When the number of samples taken equals the preset number of samples taken, the real-time accumulated data is determined as the target accumulated value.

8. The apparatus as claimed in claim 4, characterized in that, The memristor cross array is also used to perform vector-matrix multiplication operations in parallel based on the output data of the preceding neural network layer applied to the rows, generate corresponding analog output currents in the columns, and use the analog output currents as weighted summation data.

9. A stochastic binary neural network, characterized in that, The intermediate neural network layer of the randomized binary neural network is provided with a data processing device for the intermediate neural network layer as described in any one of claims 4 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data processing method for the intermediate neural network layer as described in any one of claims 1 to 3.

Images