Nonlinear function operation device, sigmoid function operation device, hyperbolic tangent function operation device, softmax function operation device, and nonlinear function operation method

Abstract

A non-linear function operation device according to an aspect of the present invention includes: a non-linear operation unit in which an internal physical quantity autonomously changes according to a predetermined non-linear characteristic; an input conversion unit that converts an input signal into time information using a parameter representing the predetermined non-linear characteristic; a control unit that, after initializing the non-linear operation unit, starts changing the internal physical quantity of the non-linear operation unit when an input signal is input to the input conversion unit and stops the change of the internal physical quantity of the non-linear operation unit after the time information ends; and an operation result output unit that outputs the internal physical quantity as an arithmetic operation result of a non-linear function having a predetermined non-linear characteristic when the physical quantity stops changing.

Description

Technical Field

[0001] The present invention relates to a non - linear function operation device, a sigmoid function operation device, a hyperbolic tangent function operation device, a softmax function operation device, and a non - linear function operation method. Background Art

[0002] Conventionally, a neuron element constituting a neural network calculates the sum of weighted input signals, and then uses an activation function with this sum as the input to calculate the output signal. The activation function is a function for determining the output format of each neuron element, and a non - linear function is usually used. Typical non - linear activation functions include the sigmoid function, the hyperbolic tangent function, and the softmax function.

[0003] Regarding an operation device for the sigmoid function, the prior art has disclosed a circuit that calculates the sigmoid function by performing a predetermined operation on the output signals of a differential amplifier composed of two transistors (see Patent Document 1).

[0004] [Patent Document 1] Japanese Patent Application Laid - Open No. 9 - 305687 Summary of the Invention

[0005] [Technical Problem to be Solved]

[0006] The present invention is proposed to solve the problems of the prior art.

[0007] [Technical Solution for Solving the Technical Problem]

[0008] A non - linear function operation device according to an aspect of the present invention includes: a non - linear operation unit in which an internal physical quantity changes autonomously according to a predetermined non - linear characteristic; an input conversion unit that converts an input signal into time information using a parameter representing the predetermined non - linear characteristic; a control unit that, after initializing the non - linear operation unit, starts changing the internal physical quantity of the non - linear operation unit when an input signal is input to the input conversion unit, and stops the change of the internal physical quantity of the non - linear operation unit after the time information ends; and an operation result output unit that outputs the internal physical quantity as an arithmetic operation result of a non - linear function having a predetermined non - linear characteristic when the internal physical quantity stops changing.

[0009] As a non - linear function operation method according to another aspect of the present invention, an input conversion unit is used to convert an input signal into time information by using a parameter representing a predetermined non - linear characteristic; a control unit controls the initialization of a non - linear operation unit, which autonomously changes an internal physical quantity according to a predetermined non - linear characteristic. When the input signal is input to the input conversion unit, the change of the internal physical quantity in the non - linear operation unit starts, and when the time information ends, the change of the internal physical quantity in the non - linear operation unit stops; when the change of the internal physical quantity stops, an operation result output unit outputs the internal physical quantity as the operation result of a predetermined non - linear function.

[0010] [Advantageous Effects]

[0011] The present invention can reduce power consumption with a simple circuit structure and stably obtain a high - precision non - linear function operation result. Description of the Drawings

[0012] Figure 1 Block diagram showing the structure of a non - linear function operation device.

[0013] Figure 2 Flowchart showing the operation process of a non - linear function operation device.

[0014] Figure 3 Circuit diagram showing the first embodiment of a non - linear operation unit.

[0015] Figure 4 Schematic diagram showing the transient characteristics of a non - linear operation unit.

[0016] Figure 5 Diagram showing the operation result when a cosine - wave signal is input as an input signal to a non - linear function operation unit.

[0017] Figure 6 Circuit diagram showing the second embodiment of a non - linear operation unit.

[0018] Figure 7 Diagram showing the relationship between the number of switchings n and the normalized output voltage V out / V S of.

[0019] Figure 8 Block diagram showing a sigmoid function operation device.

[0020] Figure 9 Block diagram showing a hyperbolic tangent function operation device.

[0021] Figure 10 Block diagram showing a softmax function operation device.

[0022] ​​​​​​​​​​Figure 11 A circuit diagram showing a sixth embodiment of a non-linear operation unit.

[0023] Figure 12 A schematic diagram showing the oscillation characteristics of a non-linear operation unit.

[0024] Figure 13 A diagram showing the operation result when a cosine wave signal is input as an input signal to a non-linear function operation device. Detailed implementation manners

[0025] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0026] [First Embodiment]

[0027] Figure 1 is a block diagram showing the structure of a non-linear function operation device 1 according to the first embodiment. The non-linear function operation device 1 includes an input conversion unit 11, a control unit 12, a non-linear operation unit 13, and an operation result output unit 14.

[0028] The input conversion unit 11 has a sampler 11-1. In addition, the operation result output unit 14 has a register 14-1 and a sampler 14-2.

[0029] The input conversion unit 11 receives an input signal and converts the input signal into time information. The input signal can be a continuous signal or a discrete signal. When the input signal is a continuous signal, after sampling the input signal into a discrete signal using the sampler 11-1, it is converted into the time information.

[0030] The control unit 12 has an internal timer (not shown), and starts timing from the moment when the input signal is input to the input conversion unit 11 after the initialization of the non-linear operation unit 13 is completed. And, the control unit 12 controls the non-linear operation unit 13 and the operation result output unit 14 according to the time information provided by the input conversion unit 11.

[0031] The non-linear operation unit 13 is initialized based on the control of the control unit 12. After initialization, the non-linear operation unit 13 changes internal physical quantities according to a predetermined non-linear characteristic. In addition, when the internal physical quantities change, when a predetermined control is performed from the control unit 12, the change of the internal physical quantities is stopped.

[0032] ​​​The operation result output unit 14 measures the internal physical quantity of the non-linear operation unit 13 according to the instruction of the control unit 12, and outputs the measurement result as the operation result of a predetermined non-linear function. In addition, as needed, the operation result output unit 14 holds the measurement result for a predetermined time using the register 14-1, or re-samples the measurement result using the sampler 14-2.

[0033] Figure 2 It is a flowchart showing the operation process of the non-linear function operation device 1.

[0034] In step S1, the control unit 12 of the non-linear function operation device 1 initializes the non-linear operation unit 13. Thereby, the non-linear operation unit 13 sets the internal physical quantity to the initial state.

[0035] In step S2, the input conversion unit 11 checks for the presence of an input signal and waits until an input signal is present. If an input signal is present, it proceeds to step S3.

[0036] In step S3, the input conversion unit 11 performs a predetermined conversion process on the input signal, converts the input signal into time information, and provides the time information to the control unit 12. When the input signal is a continuously varying analog signal, the input conversion unit 11 samples the input signal according to a predetermined sampling frequency using the sampler 11-1 and performs the predetermined conversion process.

[0037] In step S4, when the control unit 12 receives the time information from the input conversion unit 11, it instructs the non-linear operation unit 13 to start changing the internal physical quantity. Thereby, the internal physical quantity of the non-linear operation unit 13 changes according to a predetermined non-linear characteristic.

[0038] In step S5, the control unit 12 determines whether to stop the operation of the non-linear operation unit 13 based on the time information provided by the input conversion unit 11, and waits until a positive confirmation is obtained. When the determination is positive, the control unit 12 stops the operation of the non-linear operation unit 13. Thereby, the change in the internal physical quantity of the non-linear operation unit 13 stops.

[0039] In step S6, the control unit 12 causes the operation result output unit 14 to measure the internal physical quantity of the non-linear operation unit 13. The operation result output unit 14 outputs the measurement result as the operation result of a predetermined non-linear function. Then, it returns to step S1.

[0040] In addition, in step S6, the operation result output unit 14 holds the measurement result in the register 14-1 for a predetermined period as needed, and outputs the held measurement result independently of the change in the internal physical quantity of the non-linear operation unit 13. Further, the operation result output unit 14 may also use the sampler 14-2 to sample the measurement result held in the register 14-1 again and output the sampling result.

[0041] Figure 3 is a circuit diagram showing an example of the non-linear operation unit 13. The non-linear operation unit 13 includes a switch SW1, a switch SW2, resistors R1, R2, and a capacitor C m . Further, the resistor R1 is a load resistor, and the resistor R2 is a current limiting resistor.

[0042] One end of the switch SW1 is connected to the power supply V S . The other end of the switch SW1 is connected to the output terminal V via the resistor R2 OUT , one end of the switch SW2 is connected to the positive electrode terminal of the capacitor C m . The other end of the switch SW2 is connected to one end of the resistor R1. The other end of the resistor R1 is grounded. The negative electrode terminal of the capacitor C m is grounded. The switches SW1 and SW2 are turned on / off by the Figure 1 shown control unit 12.

[0043] Figure 3 The non-linear operation unit 13 shown is composed of an RC parallel circuit. The transient characteristics of the non-linear operation unit 13 are shown by Equation (1) and are the inverse function of the natural exponential function.

[0044] V out / V S = exp(-t / (C m R1)) = exp(-x) (1)

[0045] Here, the time information t is obtained from the input signal x and the time constant C m R1 of the RC parallel circuit through Equation (2).

[0046] t = C m R1·x (2)

[0047] Figure 4 is a schematic diagram showing Figure 3 the transient characteristics of the non-linear operation unit 13 shown. Note that the horizontal axis is the input signal x related to the time information t in Equation (2), and the vertical axis is the output voltage (V S normalized by the power supply voltage V out / V S ).

[0048] As an initialization process, the control unit 12 turns on the switch SW1 and turns off the switch SW2. Thus, the power supply V S supplies charge to the capacitor C m via the resistor R2. As a result, the capacitor C m is charged by the power supply V S .

[0049] After detecting the input signal x, the input conversion unit 11 converts the input signal x into time information t using Equation (2).

[0050] Next, after the internal timer set by the control unit 12 starts counting, the switch SW1 is turned off and the switch SW2 is turned on. The charge stored in the capacitor C m is discharged through the resistor R1.

[0051] When the count of the internal timer reaches the time information t obtained in Equation (2), the control unit 12 turns off the switch SW2. At the same time, the switch SW1 remains off.

[0052] The operation result output unit 14 measures the output voltage V out of the nonlinear operation unit 13 and outputs the measured value as the operation result of the nonlinear function based on Equation (1) to the outside. Note that the operation result output unit 14 can also hold the measured value of the output voltage V out in the register 14-1 regardless of the state of the nonlinear operation unit 13. In addition, the operation result output unit 14 can also use the sampler 14-2 to resample the measured value of the output voltage V out held in the register 14-1 based on a predetermined frequency and output it as discretized information.

[0053] Here, it is assumed that the input signal x is equal to or greater than 0. However, if the input signal x is less than 0, the reciprocal of the measured value of the output voltage V out can be calculated.

[0054] Now, for the input signal x equal to or greater than 0, the inverse function e -x of the natural exponential function as the nonlinear function is defined as follows:

[0055] EX(x) = e -x (x ≥ 0) (3)

[0056] In this case, for any input signal x, the inverse function e -x of the natural exponential function is as follows:

[0057] e -x = EX(x) (x ≥ 0)

[0058] = 1 / (EX(-|x|)) (x < 0) (4)

[0059] Figure 5 This is a schematic diagram showing the operation result when a cosine-wave-based signal is input as the input signal x in the non-linear function operation device 1. Specifically, the input signal x is a signal sampled at an arbitrary time interval based on a cosine wave (period T) normalized to the range of -1.0 to 1.0.

[0060] Note that the operation result output unit 14 stores the operation result of the previous input signal x in the register 14-1 until the operation result of the new input signal x is obtained from the non-linear operation unit 13, and re-samples it at a period of T / 16 using the sampler 14-2, and the result is output as the output signal y. In addition, processing delays, such as those caused by parasitic capacitances in the non-linear working section 13, are also ignored.

[0061] At t = 0, the input signal x = 1.0. Thus, the calculation result y = e -1.0 At t = C m It is output at R1, with a delay of C m R1 compared to when the input signal x is input. Then, an initialization process is performed to recharge the capacitor C m .

[0062] At t = 3T / 24, the input signal x = 1 / √2. Thus, the calculation result y = e- 1 / √2 At t = 3T / 24 + C m It is output at R1 / √2, with a delay of C m R1 / √2 compared to when the input signal x is input. Then, an initialization process is performed and the capacitor C m is charged.

[0063] At t = 8T / 24, the input signal x = -0.5. Thus, the calculation result y = e 0.5 At t = 8T / 24 + C m It is output at R1 / 2, with a delay of C m R1 / 2 compared to when the input signal x is input. Then, an initialization process is performed and the capacitor C m is charged.

[0064] At t = 12T / 24, the input signal x = -1.0. Thus, the operation result y = e 1.0 At t = 12T / 24 + C m It is output at R1, with a delay of C m R1 compared to when the input signal x is input. Then, an initialization process is performed and the capacitor C m is charged.

[0065] At t = 15T / 24, the input signal x = -1 / √2. Thus, the calculation result y = e 1 / √2 At t = 15T / 24 + C m Output at R1 / √2, which is delayed by C m R1 / √2 compared to when the input signal x is input. Then, perform the initialization process and charge the capacitor C m Charge.

[0066] When t = 20T / 24, the input signal x = 0.5. Thus, the calculation result y = e -0.5 At t = 20T / 24 + C m Output at R1 / 2, which is delayed by C m R1 / 2 compared to when the input signal x is input. Then, perform the initialization process and charge the capacitor C m Charge.

[0067] At t = 24T / 24, the input signal x = 1.0. Thus, the calculation result y = e -1.0 At t = 24T / 24 + C m Output at R1, which is delayed by C m R1 compared to when the input signal x is input. Then, perform the initialization process and charge the capacitor C m Charge.

[0068] It can be seen from Figure 5 that in the non - linear function operation device 1, when the input signals are signals sampled at any time interval, the input time intervals of these signals need to be long enough, longer than the time constant C m R1 of the non - linear operation unit 13. Similarly, when the input signals are continuous analog signals, the sampling period for sampling the analog signals using the sampler 11 - 1 must also be long enough, longer than the time constant C m R1 of the non - linear operation unit 13.

[0069] Specifically, if the maximum value of the input signal x is x max , and the input signal x is a signal sampled at an arbitrary time interval, the non - linear operation unit 13 does not accept the input of the next signal during the period from the start of operation (discharge start) to C m R1·x max . Therefore, its input time interval needs to be longer than C m R1·x max . If the input signal x is a continuous analog signal, this is equivalent to the sampling frequency f s of the sampler 11 - 1 for the input signal x must satisfy formula (5).

[0070] f s <1 / (Cm R1·x max ) (5)

[0071] In addition, when resampling during the output using sampler 14-2, the resampling frequency f of sampler 14-2 is required to be rs sufficiently greater than the sampling frequency f of the input signal x s . For example, during the period t = 0 to T / 8 in Figure 5 , the calculation result y of the input signal x = 1.0 at t = 0 can only be obtained at t = C m after R1

[0072] On the other hand, when a new input signal is input at t = T / 8, if the input signal is x = 0.0, the calculation result y = 1.0 will be immediately generated. Thus, the arithmetic operation result of the input signal x at t = 0 can only be obtained during the period from t = C m R1 to T / 8. Therefore, in order to obtain the calculation result y of the input signal x at t = 0, at least one resampling needs to be performed within this period

[0073] This can be summarized as follows. If the sampling period (input time interval) of sampler 11-1 for the input signal is set to 1 / f s , within this 1 / fs period, the duration of the first C m R1·x max is occupied by the operation and the result cannot be output, the signal output will be delayed, and the time period for obtaining the actual calculation result is limited to the period from (1 / f s ) to C m R1·x max . Therefore, to enable sampler 14-2 to perform at least one resampling within this period, the resampling can only be completed after this operation occupancy, and the resampling frequency f rs needs to satisfy formula (6).

[0074] f rs > 1 / ((1 / f s ) - C m R1·x max ) (6)

[0075] In addition, in a neural network, when the outputs of all neurons in one layer are aligned, the processing of the next layer of neurons will be performed. In other words, the actions of the neuron elements in each layer are synchronized. Thus, formula (5) is also a constraint condition for using the non-linear function operation device 1 in each neuron element

[0076] [Second Embodiment]

[0077] Next, a second embodiment will be described. Note that the same reference numerals are given to the same parts as in the first embodiment, and their detailed descriptions are omitted. In the second embodiment, the non-linear operation unit 13A shown in Figure 6 is used as another example of the non-linear operation unit 13 shown in Figure 1 .

[0078] Figure 6 is a circuit diagram example showing the non-linear operation unit 13A. The non-linear operation unit 13A is implemented by a switched-capacitor circuit. The non-linear operation unit 13A includes switches SW3, SW4, SW5, a resistor R2, and capacitors C S , C m . In addition, the resistor R2 is a current-limiting resistor.

[0079] One end of the switch SW3 is connected to the power supply V S . The other end of the switch SW3 is connected to the output terminal V out , one end of the switch SW4, and the positive electrode terminal of the capacitor C m via the resistor R2. The other end of the switch SW4 is connected to one end of the switch SW5 and the positive electrode terminal of the capacitor C S . The negative electrode terminals of the capacitors C S , C m and the other end of the switch SW5 are grounded. The switches SW3, SW4, and SW5 are turned on and off by the control unit 12 shown in Figure 1 . In addition, the switch SW4 and the switch SW5 operate mutually exclusively. That is, when the switch SW4 is in the on state, the switch SW5 is in the off state, and when the switch SW4 is in the off state, the switch SW5 is in the on state.

[0080] In the initialization process, the control unit 12 controls the switch SW3 of the non-linear operation unit 13A to be turned on, and controls the switch SW4 to be turned off (the switch SW5 is turned on). Thus, the power supply V S supplies charge to the capacitor C m via the resistor R2 for charging. Then, the power supply V S charges the capacitor C m . In addition, the capacitor C S will be fully discharged.

[0081] When a signal is input to the conversion unit 11, the control unit 12 first turns off the switch SW3 in the state where the switch SW4 (the switch SW5 is turned on) is off, and then continuously switches the switch SW4 and the switch SW5.

[0082] When the switch SW4 is turned on (the switch SW5 is turned off), the charge accumulated in the capacitor is shared between C m , C S , and the capacitor Cm Then, when switch SW4 is turned off (switch SW5 is turned on), capacitor C S When switch SW4 is turned on again (switch SW5 is turned off), the charge remaining in capacitor C m The charge on capacitor C m , C S shared between them, capacitor C m That is, when the switches SW4 and SW5 are switched exclusively and continuously, the capacitor C m The voltage decreases in steps.

[0083] Figure 7 is the switching number n of the switch SW4 and the switch SW5 and the power supply voltage V S Normalized output voltage (V out / V S )’s relationship diagram.

[0084] When a signal is input to the input conversion unit 11 and the nonlinear operation unit 13 starts to operate, the switch SW4 switches to on, off, on, off, etc. On the other hand, the switch SW5 switches to off, on, off, on, etc. in conjunction with the switch SW4.

[0085] When the switch SW4 is turned on for the nth time (the switch SW4 is turned off for the nth time), the normalized output voltage V of the nonlinear operation unit 13 is out / V S As shown in formula (7).

[0086] V out / V S =(C m / (C m +C S )) n

[0087] =(1+C S / C m ) -n

[0088] =e -αn (7)

[0089] Wherein, α is as shown in formula (8).

[0090] α=ln(1+C S / C m ) (8)

[0091] Now, assuming that an input signal x is input, the number of switching times n of switches SW4 and SW5 is as shown in formula (9).

[0092] n = [x / α] (9)

[0093] Note that [z] is the Gaussian symbol, representing the largest integer less than or equal to the real number z.

[0094] In the case of this embodiment, the control unit 12 uses formula (9) to convert the input signal into time information and calculates the number of switching times n of the switches SW4 and SW5.

[0095] Moreover, after the switch SW4 is switched n times in the non-linear operation unit 13A, the operation result output unit 14 measures the output voltage V out of the non-linear operation unit 13A as the operation result of the non-linear function operation device 1.

[0096] Next, consider the case of quickly switching the switches SW4 and SW5 under the condition of C m >>C S (n >> 1).

[0097] At this time, let the number of on / off switching times of the switches SW4 and SW5 per unit time (hereinafter referred to as the switching frequency) be f sw , then substituting α~C S / C m , n = f sw ·t into formula (9), formula (10) is obtained.

[0098] t~C m ·(1 / (C S f sw ))·x (10)

[0099] From the comparison between formula (2) and formula (10), it can be seen that the equivalent resistance corresponding to the reciprocal of the product of the switching capacitor C S and the switching frequency f sw satisfies formula (11).

[0100] R1~1 / (C S f sw ) (11)

[0101] Therefore, after setting the internal timer, the control unit 12 continuously switches the switches SW4 and SW5 through the switching frequency f sw , so that the charge stored in the capacitor C m is discharged periodically. When the internal timer reaches the time information t obtained by formula (10), the control unit 12 stops the switching of the switches SW4 and SW5 and controls the switch SW4 to be off (the switch SW5 is on).

[0102] As described above, switches SW4 and SW5 are switched between on and off starting from the time when the input signal is input to the input conversion unit 11 (operation start point), and the switching between on and off is stopped at the time when the time information t is reached from the operation start point.

[0103] After the switching of switches SW4 and SW5 stops, the operation result output unit 14 measures the output voltage V of the non-linear operation unit 13A. out , and outputs the measured value as the operation result of the non-linear function to the outside. After obtaining the measured value of the output voltage V out , the control unit 12 performs the above-described initialization process of the non-linear operation unit 13A and continues to standby until the input signal is input to the input conversion unit 11 again.

[0104] Here, as a constraint condition of the second embodiment, regarding the input time interval or the sampling period 1 / f of the input signal s , if the maximum value of the input signal is x max , then by substituting Equation (11) into Equation (5), Equation (12) is obtained.

[0105] F s < (C S f sw ) / (C m x max ) (12)

[0106] Regarding the resampling frequency f at the time of output rs , by substituting Equation (11) into Equation (6), Equation (13) is obtained.

[0107] F rs > 1 / ((1 / f s ) - (C m x max ) / (C S f sw )) (13)

[0108] [Third Embodiment]

[0109] In the third embodiment, the sigmoid function operation device is implemented using the above-described non-linear function operation device 1. The sigmoid function (Standard sigmoid function) is an activation function used in the neuron elements of the intermediate layer of the neural network or the output layer of the neural network for binary classification. Specifically, the sigmoid function is defined as follows:

[0110] Sigmoid(x) = 1 / (1 + e -x ) (14)

[0111] If formula (3) is used where x ≥ 0, formula (14) becomes the following:

[0112] Sigmoid(x) = 1 / (1 + EX(x)) (15)

[0113] In addition, according to the properties of the sigmoid function, when x < 0, the following holds:

[0114] Sigmoid(x) = 1 - Sigmoid(|x|) (16)

[0115] Figure 8 is a block diagram showing a sigmoid function operation device 100 that uses a non - linear function operation device 1 when the input signal x ≥ 0. The sigmoid function operation device 100 includes a non - linear function operation device 1, an adder 101, and a divider 102. The non - linear function operation device 1 is configured as in the above - described embodiment, and outputs EX(x) ( = e -x ) shown in formula (3) for x ≥ 0.

[0116] The adder 101 adds the output value "e -x " of the non - linear function operation device 1 and the numerical value "1" as a natural number, and outputs "1 + e -x ". The divider 102 divides the numerical value "1" by the output value "1 + e -x " of the adder 101, and outputs "1 / (1 + e -x )". That is, the divider 102 calculates the reciprocal of the adder 101.

[0117] As described above, by using arithmetic units predetermined with the non - linear function operation device 1, the sigmoid function operation device 100 can be easily implemented.

[0118] [Fourth Embodiment]

[0119] In the fourth embodiment, the above - described non - linear function operation device 1 is used to implement a hyperbolic tangent function calculation device. The hyperbolic tangent function is an activation function used in neuron elements in the intermediate layer of a neural network. Specifically, the hyperbolic tangent function is defined as follows.

[0120] tanh(x) = (e x - e -x ) / (e x + e -x ) (17)

[0121] If formula (3) is used where x ≥ 0, formula (17) is as shown below.

[0122] tanh(x) = (e x - e -x ) / (e x + e -x )

[0123] = (1 - e -2x ) / (1 + e -2x )

[0124] = (1 - EX(2x)) / (1 + EX(2x)) (18)

[0125] In addition, according to the properties of the hyperbolic tangent function, when x < 0, the following equation holds.

[0126] tanh(x) = -tanh(-x)

[0127] = -1 · tanh(|x|) (19)

[0128] Figure 9 It is a block diagram of the hyperbolic tangent function operation device 200 that uses the non - linear function operation device 1 when the input signal x ≥ 0.

[0129] The hyperbolic tangent function operation unit 200 includes an adder 201, a non - linear function operation unit 1, a sign converter 202, adders 203 and 204, and a divider 205. The non - linear function operation device 1 is configured as in the above - mentioned embodiment, and outputs EX(x) (= e -x ) for the output signal x ≥ 0.

[0130] The adder 201 adds the input signal x input to the two input terminals and outputs "2x". The non - linear function operation device 1 outputs "e -2x " for the output value "2x" of the adder 201. The sign converter 202 reverses the sign of the output value "e -2x " of the non - linear function operation device 1 and outputs "-e -2x ".

[0131] The adder 203 adds "1" and the output value "-e -2x " of the sign converter 202 and outputs "1 - e -2x ". The adder 204 adds "1" and the output value "e -2x " of the non - linear function operation device 1 and outputs "1 + e - 2x ". The divider 205 divides the output value "1 - e -2x " of the adder 203 by the output value "1 + e -2x " of the adder 204 and outputs "(1 - e -2x ) / (1 + e -2x)".

[0132] As described above, the hyperbolic tangent function operation device 200 can be easily implemented by using the non-linear function operation device 1 and a predetermined arithmetic unit.

[0133] [Fifth Embodiment]

[0134] In the fifth embodiment, the softmax function is implemented using the above-described non-linear function operation device 1. The Softmax function is an activation function, and is particularly used in neuron elements in the output layer of a neural network for multi-level classification. Specifically, if there are currently m neuron elements in the output layer, the softmax function applied to the i-th neuron element is defined as in Equation (20).

[0135]

[0136] Here, using Equation (3) and setting the maximum input value xi as x max Equation (20) becomes Equation (21).

[0137]

[0138] Note that, as a property of the softmax function, Equation (22) is derived from the definition of Equation (20).

[0139]

[0140] Figure 10 An arbitrary input signal x is shown k (k = 1, 2,..., m) The softmax function operation device 300 using the non-linear function operation device 1. Hereinafter, k represents a natural number from 1 to m.

[0141] The softmax function operation device 300 includes: a maximum value detector 301, m sign converters 302-1, 302-2,..., 302-m, and m adders 303-1, 303-2,..., 303-m, m non-linear function operation units 1-1, 1-2,..., 1-m. Among them, the non-linear function operation device 1-k is the same as the above-described non-linear function operation device 1. In addition, the summation operation device 304 is implemented by combining adders corresponding to the number of input terminals.

[0142] The maximum value detector 301 outputs the maximum value x from the m input signals x k The sign converter 302-k inverts the sign of x max . The sign converter 302-k outputs "-x k " by inverting the sign of xk ".

[0143] The adder 303-k adds the output value x of the maximum value detector 301 to max The output value of the symbol converter 302-k is k "Add, output" x max -x k ". The output value of the nonlinear function operation device 1-k to the adder 303-k is "x max -x k "Output"e -(x max -x k )".

[0144] The summation operator 304 calculates the output values "e" of the nonlinear function operation devices 1-k. -(x max -x k )" and output "∑e -(x max -x k )".

[0145] The divider 305-k converts the output value "e" of the nonlinear function operation device 1-k into -(x max -x k )" divided by the output value of the summation operator 304"∑e -(x max -x k )", output the softmax function operation value Softmax(k,x).

[0146] As described above, the softmax function operation device 300 can be easily realized by using the nonlinear function operation device 1-k and a predetermined operator.

[0147] [Sixth embodiment]

[0148] Next, the sixth embodiment is described. In the sixth embodiment, using Figure 11 The nonlinear operation unit 13B shown in FIG. Figure 1 Another example of the nonlinear operation unit 13 is shown.

[0149] Figure 11It is a circuit diagram showing an example of the non - linear operation unit 13B. The non - linear operation unit 13B is an oscillation circuit for operating the cosine function. The non - linear operation unit 13B includes switches SW6, SW7, resistors R2, R3, capacitor Cn, and inductor L. Additionally, resistor R2 is a current - limiting resistor, and resistor R3 is a shunt resistor.

[0150] The switch SW7 has three terminals T1, T2, and T3. The switch SW7 is connected between terminal T1 and terminal T3, or between terminal T2 and terminal T3, according to Figure 1 the control (selection signal) of the control unit 12 as shown.

[0151] The switch SW6 is turned on and off by Figure 1 the control unit 12 as shown. One end of the switch SW6 is connected to the power supply V S . The other end of the switch SW6 is connected to the output terminal V out , one end of Cn, and terminal T1 of the switch SW7 through the resistor R2 respectively. The other end of the capacitor Cn is grounded. Terminal T2 of the switch SW7 is grounded through the resistor R3. Terminal T3 of the switch SW7 is grounded through the inductor L.

[0152] Here, for the non - linear operation unit 13B as an oscillation circuit, its operating frequency (natural frequency) f LC is as shown in formula (23).

[0153] f LC = 1 / (2π√(LCn)) (23)

[0154] The oscillation operation of the non - linear operation unit 13B is as shown in formula (24).

[0155] V out / V S = cos(2πf LC ·t)= cos(x) (24)

[0156] Therefore, the time information t equivalent to the input signal x is as shown in formula (25).

[0157]

[0158] In formula (25), [z] is the Gaussian symbol, representing the largest integer less than or equal to the real number z.

[0159] Next, the processing flow of the control unit 12 is introduced. As an initialization process, the control unit 12 turns on the switch SW6 of the non - linear operation unit 13B and connects terminal T2 and terminal T3 of the switch SW7. Thus, the power supply V S supplies charge to the capacitor Cn through the resistor R2. As a result, the capacitor Cn By the power supply V S Fully charged, the current in the inductor L is zero.

[0160] After detecting the input signal x, the input conversion unit 11 uses Equation (25) to transform the input signal x into time information t.

[0161] Next, after the control unit 12 sets the internal timer to start counting, it disconnects the switch SW6 and connects the terminals T1 and T3 of the switch SW7. As a result, the capacitor Cn is connected in parallel with the inductor L, and charge moves from the capacitor Cn to the inductor L, and electrical oscillation occurs when the charge moves.

[0162] Figure 12 It is a graph showing the oscillation characteristics when the non-linear operation unit 13B shown in Figure 11 generates electrical oscillation. Among them, the horizontal axis is the input signal x related to the time information t in Equation (25), and the vertical axis is the output voltage (V S normalized by the power supply voltage V out / V S ).

[0163] When the count of the internal timer reaches the time information t in Equation (25), the control unit 12 connects the terminals T2 and T3 of the switch SW7. The switch SW6 remains disconnected. Therefore, the capacitor C n and the inductor L are electrically disconnected, and the current flowing through the inductor L converges to zero via the resistor R3. In this way, the voltage of the capacitor C n at the time of reaching the time information t is retained.

[0164] The operation result output unit 14 measures the output voltage V out of the non-linear operation unit 13, and outputs the measured value as the calculation result of the non-linear function to the outside according to Equation (24). Note that the operation result output unit 14 can also hold the measured value of the output voltage V out in the register 14-1, regardless of the state of the non-linear operation unit 13B. In addition, the operation result output unit 14 can also use the sampler 14-2 to resample the measured value of the output voltage V out held in the register 14-1 based on a predetermined frequency and output it as discretized information.

[0165] Figure 13 It is a diagram showing the calculation result when a cosine wave signal is input as the input signal x to the non-linear function operation device 1. Specifically, the input signal x is a signal sampled at an arbitrary time interval based on a cosine wave (period T) normalized to the range of -1.0 to 1.0.

[0166] The operation result output unit 14 stores the operation result for the previous input signal x in the register 14-1 until the calculation result for the new input signal x is obtained from the non-linear operation unit 13. It will use the sampler 14-2 to resample at a period of T / 16, and the result is output as the output signal y. Additionally, processing delays caused by parasitic capacitances in the non-linear operation unit 13 will be ignored.

[0167] At t = 0, the input signal x = 1.0. Thus, the calculation result y = cos(1.0) is output at t = 1 / (2πf LC ), with a delay of 1 / (2πf) compared to the input signal x. Then the initialization process is executed, and the capacitor C n is charged.

[0168] At t = 3T / 24, the input signal x = 1 / √2. Thus, the calculation result y = cos(1 / √2) is output at t = 3T / 24 + 1 / (2√2·πf LC ), with a delay of 1 / (2√2·πf LC ) compared to the input signal x. Then the initialization process is executed, and the capacitor C n is charged.

[0169] At t = 8T / 24, the input signal x = -0.5. Thus, the calculation result y = cos(0.5) is output at t = 8T / 24 + 1 / (4πf LC ), with a delay of 1 / (4πf LC ) compared to the input signal x. Then the initialization process is executed, and the capacitor C n is charged.

[0170] At t = 12T / 24, the input signal x = -1.0. Thus, the calculation result y = cos(1.0) is output at t = 12T / 24 + 1 / (2πf LC ), with a delay of 1 / (2πf LC ) compared to the input signal x. Then the initialization process is executed, and the capacitor C n is charged.

[0171] At t = 15T / 24, the input signal x = -1 / √2. Thus, the calculation result y = cos(1 / √2) is output at t = 15T / 24 + 1 / (2√2·πf LC ), with a delay of 1 / (2√2·πf LC ) compared to the input signal x. Then the initialization process is executed, and the capacitor C n is charged.

[0172] When t = 20T / 24, the input signal x = 0.5. Thus, the calculation result y = cos(0.5) is output at t = 20T / 24 + 1 / (4πf LC ), with a delay of 1 / (4πf LC ) compared to the input signal x. Then, the initialization process is executed, and the capacitor C n is charged.

[0173] When t = 24T / 24, the input signal x = 1.0. Thus, the calculation result y = cos(1.0) is output at t = 24T / 24 + 1 / (2πf), with a delay of 1 / (2πf) compared to the input signal x. Then, the initialization process is executed, and the capacitor C n is charged.

[0174] From Figure 13 , it can be seen that in the non - linear function operation device 1, when the input signals are signals sampled at arbitrary time intervals, their input time intervals need to be sufficiently longer than the longest operation time 1 / f of the non - linear operation unit 13B LC . Similarly, when the input signal is a continuous analog signal, it is required that the sampling period when using the sampler 11 - 1 to sample the analog signal must be sufficiently longer than the longest operation time 1 / f of the non - linear operation unit 13B LC .

[0175] Specifically, if the input signal x is a signal sampled at any time interval, within the period from the start of operation (start of vibration) up to the longest 1 / f LC , the non - linear operation unit 13B may not be able to accept the input of the next signal. Therefore, its input time interval needs to be longer than 1 / f LC . If the input signal x is a continuous analog signal, this means that the sampling frequency f s of the sampler 11 - 1 with respect to the input signal x must satisfy formula (26).

[0176] f s < f LC (26)

[0177] In addition, when resampling is performed using the sampler 14 - 2 for output, it is required that the resampling frequency f rs of the sampler 14 - 2 is sufficiently greater than the sampling frequency f s of the input signal x. For example, in Figure 13 the period from t = 0 to T / 8, the operation result of the input signal x = 1.0 at t = 0 can only be obtained after t = 1 / (2πf LC ).

[0178] On the other hand, at t = T / 8, a new input signal is input. However, assuming that the input signal x = 0.0, the calculation result y = 1.0 will be obtained immediately without delay. Thus, the calculation result of the input signal x at t = 0 can only be obtained within the range from t = 1 / (2πf LC ) to T / 8. Therefore, in order to obtain the calculation result y of the input signal x at t = 0, it is necessary to perform resampling at least once within this period.

[0179] This can be summarized as follows. If the sampling period (input time interval) of the sampler 11-1 for the input signal is set to 1 / f s , and the operation is being performed within the longest 1 / f LC period within this cycle, the signal output is delayed, and the actual calculation result is only obtained within the period of (1 / f s ) - (1 / f LC ). Therefore, in order to enable the sampler 14-2 to perform resampling at least once within this period, the resampling frequency f rs needs to satisfy formula (27).

[0180] f rs > 1 / ((1 / f s ) - (1 / f LC ))

[0181] = (f LC ·f s ) / (f LC - f s ) (27)

[0182] In addition, the present invention is not limited to the above-described embodiments, and can also be applied to cases where design changes are made within the scope of the matters described in the claims.

[0183] [Symbol Description]

[0184] 1 Nonlinear function operation device

[0185] 11 Input conversion unit

[0186] 12 Control unit

[0187] 13, 13A, 13B Nonlinear operation units

[0188] 14 Operation result output unit

[0189] C m , C S , C n Capacitor

[0190] L Inductor

[0191] R1, R2, R3 Resistors

[0192] SW1, SW2, SW3, SW4, SW5, SW6, and SW7 switches

Claims

1. A non-linear function operation device, comprising: A non-linear operation unit, in which the internal physical quantity changes autonomously according to a predetermined non-linear characteristic; An input conversion unit, which converts an input signal into time information by using a parameter representing the predetermined non-linear characteristic; A control unit, after initializing the non-linear operation unit, starts to change the internal physical quantity of the non-linear operation unit when an input signal is input to the input conversion unit, and stops the change of the internal physical quantity of the non-linear operation unit after the time information ends; An operation result output unit, which outputs the internal physical quantity as the arithmetic operation result of a non-linear function having a predetermined non-linear characteristic when the change of the internal physical quantity stops.

2. The non-linear function operation device according to claim 1, wherein, The operation result output unit includes: A register, which stores the internal physical quantity when the change of the internal physical quantity stops, and A sampler, which samples the internal physical quantity stored in the register and outputs it as the arithmetic operation result of the non-linear function having the predetermined non-linear characteristic.

3. The non-linear function operation device according to claim 1, wherein The predetermined non-linear characteristic is a transient characteristic, and the parameter is the time constant of the transient characteristic.

4. The non-linear function operation device according to claim 3, wherein The non-linear operation unit has a first charge storage unit, and when the stored charge is discharged, its voltage changes according to the transient characteristic; The input conversion unit converts the input signal into the time information by using a time constant proportional to the capacitance of the first charge storage unit; The control unit accumulates charge in the first charge storage unit as initialization, and when the input conversion unit receives an input signal, discharges the charge in the first charge storage unit, starts the voltage change of the first charge storage unit, and stops the charge discharge of the first charge storage unit after the time information has passed, thereby stopping the voltage change of the first charge storage unit; The operation result output unit outputs the voltage of the first charge storage unit as the calculation result of the reciprocal of an exponential function with a predetermined base when the voltage change stops.

5. The non-linear function operation device according to claim 4, wherein The non-linear operation unit further includes a load resistor. When initializing by accumulating charge in the first charge storage unit and when the input signal is input to the input conversion unit, the voltage change of the first charge storage unit is started by connecting the first charge storage unit and the load resistor in parallel, and after the time information has passed, the voltage change of the first charge storage unit is stopped by disconnecting the first charge storage unit and the load resistor.

6. The non-linear function operation device according to claim 5, wherein When the sampling frequency of the input signal is f s , the maximum value of the input signal is x max , the capacitance of the first energy storage unit is C m , when the resistance value of the load resistor is R1 Satisfy f s <1 / (C m ·R1·x max )。 7. The non-linear function operation device according to claim 4, wherein The non-linear operation unit further includes a second charge storage unit. Wherein, as an initialization, charges are accumulated in the first charge storage unit. When an input signal is input to the input conversion unit, the first charge storage unit and the second charge storage unit are connected in parallel to initiate a voltage change in the first charge storage unit. The switching operations of disconnecting the second charge storage unit from the first charge storage unit, discharging the charges stored in the second charge storage unit, and reconnecting the first charge storage unit and the second charge storage unit in parallel are repeatedly executed until the repeated switching operations are stopped when the elapsed time reaches the time information, thereby stopping the voltage change in the first charge storage unit.

8. The non-linear function operation device according to claim 7, wherein When the sampling frequency of the input signal is f s , the maximum value of the input signal is x max , the capacitance of the first energy storage unit is C m , the capacitance of the second energy storage unit is C S , and the switching frequency of the switching operation is f sw then Meet f s <C S ·f sw / (C m ·x max )。 9. The non-linear function operation device according to claim 1, wherein The predetermined non-linear characteristic is an oscillation characteristic, and the parameter is a value related to the natural frequency of the oscillation characteristic.

10. The non-linear function operation device according to claim 9, wherein The non-linear operation unit has an inductor and a capacitor connected in parallel to the inductor through a switching element; The input conversion unit converts the input signal into time information using a value related to the natural frequency; The control unit disconnects the switching element during initialization to accumulate charges in the capacitor. When an input signal is input to the input conversion unit, the voltage of the capacitor is changed by turning on the switching element. After the time information has elapsed, the switching element is disconnected to stop the voltage change of the capacitor; The operation result output unit outputs the voltage of the capacitor at the time point when the voltage change stops as the operation result of a predetermined single oscillation function.

11. A sigmoid function operation device, which includes The non-linear function operation device according to any one of claims 1-8; An adder for adding the output value of the non-linear function operation device to the natural number 1; And A first operation unit for obtaining the sigmoid function arithmetic value by calculating the reciprocal of the output value of the adder.

12. A hyperbolic tangent function operation device, which includes A first operation unit for doubling the input signal; The non-linear function operation device according to any one of claims 1-8, which uses the output value of the first operation unit as the input value; A second operation unit for calculating the value obtained by subtracting the arithmetic value of the non-linear function operation device from the natural number 1; An adder for adding the natural number 1 and the arithmetic value of the non-linear function operation device; A third operation unit for calculating the arithmetic value of the second operation unit divided by the arithmetic value of the adder to obtain the arithmetic operation result of the hyperbolic tangent function.

13. A softmax function operation device, which includes: A maximum detector for detecting the maximum value among m input signals x k ; The first arithmetic unit is used to perform subtraction of each of the m input signals x from the output value of the maximum value detector; k from the output value of the maximum value detector; The non-linear function operation device according to any one of claims 1 to 8, and each of the non-linear function operation devices uses the output value of the corresponding first operation unit as the input value; A summation operation unit for calculating the sum of the m output values of the non-linear function operation devices; A second arithmetic unit that, through calculation, divides each of the m output values of the non-linear function arithmetic device by the arithmetic value of the summing arithmetic unit to obtain m output values of the softmax function.

14. A non-linear function arithmetic method, wherein, an input conversion unit converts an input signal into time information by using a parameter representing a predetermined non-linear characteristic; a control unit controls an initialization of a non-linear arithmetic unit that autonomously changes an internal physical quantity according to a predetermined non-linear characteristic, starts to change the internal physical quantity in the non-linear arithmetic unit when an input signal is input to the input conversion unit, and stops the change of the internal physical quantity in the non-linear arithmetic unit after the time information ends; when the change of the internal physical quantity stops, an arithmetic result output unit outputs the internal physical quantity as an arithmetic result of a predetermined non-linear function.

Images