Device and method for a similarity evaluation

Abstract

The invention relates to a process for determining attention, comprising the following steps: carrying out a first vector-matrix multiplication in such a way that each value of the vector resulting from the vector-matrix multiplication is generated in the form of an electrical signal; converting each generated electrical signal into a pulse, the occurrence of which depends on the magnitude of the respectively generated signal, wherein the larger the signal, the later a pulse occurs; determining attention from the pulses. A second vector-matrix multiplication can be performed. An apparatus for carrying out a method includes a first electrical circuit configured to carry out a vector-matrix multiplication and output the result of the vector-matrix multiplication in the form of electrical signals. The apparatus includes a second electronic circuit configured to convert each electrical signal into a time-dependent electrical pulse. The apparatus includes a third electronic circuit configured to determine attention from the time-dependent electrical pulses. A second vector-matrix multiplication can be carried out with the apparatus.

Description

[0001] Forschungszentrum Jülich GmbH G70313WO / PT 1.3110 PCT Device and method for a similarity assessment Description The invention relates to a device and a method for determining attention. The following equation can be used to determine attention. Q = query, K = key, V = value. QK T This is a vector-matrix multiplication. The result is a first vector. The softmax function is an operation on the first vector to transform it into a second vector containing probabilities. The softmax function divides each value of the first vector by the sum of all values ​​of the first vector and stores the result in the new second vector. A concrete example is shown below: Multiplying the operation by V is another vector-matrix multiplication. The equation for Attention(Q,K,V) performs a similarity assessment between queries (Q) and keys (K) to weight the information of the value (V). K can be sequence elements from the future, present, and past, while Q is current input data. The equation for Attention(Q,K,V) outputs a weighted sum of values, allowing you to focus on the important parts of a query. Attention makes it possible to concentrate on specific information and ignore irrelevant parts. In machine learning, attention is used to focus on important elements of input data to improve task performance. Instead of the softmax function, other activation functions can also be used to determine attention.Determining attention requires very large matrix operations that consume a lot of power. The object of the invention is to enable the determination of attention with minimal effort. This object can be achieved by a method and a device with the features of the first claim and the dependent claim, respectively. Advantageous embodiments are described in the dependent claims.The task can be solved as follows: A procedure for determining an attention with the following steps: Performing an initial vector-matrix multiplication such that each value of the vector resulting from the vector-matrix multiplication is generated in the form of an electrical signal, such as an electric current; Converting each generated electrical signal into a pulse whose occurrence depends on the magnitude of the signal, for example, the electric charge of the electrical signal, whereby a pulse occurs later the larger the electrical signal, for example, the greater the amount of charge; Determining an attention from the pulses. The electrical signal can be a flowing electric current. The electrical signal can be a voltage. The electrical signal can be integrated through a capacitor, thus yielding an amount of charge resulting from the magnitude of the electrical signal.Determining attention from the pulses can involve a second vector-matrix multiplication. Determining attention from the pulses can involve a time-dependent reduction of values. Values ​​generated using the pulses can be reduced over time. The earlier a value is generated, the greater the reduction can be. The time dependence of the reduction can be exponential. The first vector-matrix multiplication can be performed using an analog circuit. Determining attention from the pulses can be performed using an analog circuit. The analog circuit can include memristors.An apparatus for carrying out the method may comprise a first electrical circuit configured to perform vector-matrix multiplication and output the result of the vector-matrix multiplication in the form of electrical signals. The apparatus may comprise a second electronic circuit configured to convert each electrical signal into a time-dependent electrical pulse. The circuit may comprise a third electronic circuit configured to determine an attention from the time-dependent electrical pulses. The first electrical circuit and / or the second electrical circuit and / or the third electrical circuit may include memristors. Circuits with memristors may be configured to perform vector-matrix multiplications.The third circuit can additionally include electrical resistors and capacitors. The components of the third circuit can be configured to determine values ​​from the input values. The electrical resistors and capacitors of the third circuit can be designed to reduce values ​​resulting from vector-matrix multiplications over time. These values ​​can be reduced exponentially. They can be values ​​resulting from pulses of the second circuit. The second electrical circuit can include a capacitor for integrating an electrical signal. The second circuit can include a capacitor discharge circuit. The second circuit can include comparators. The second circuit can include transistors.The components of the second circuit can be configured to generate a time-dependent pulse from the magnitude of a time-dependent signal. The second circuit can include an operational amplifier. The second circuit can reduce a value, for example, linearly. This can be achieved by a charged capacitor that is discharged through an electrical resistor with a constant current. This can be done digitally, either alternatively or additionally. Depending on the linear reduction, the aforementioned exponential decay can be selected to appropriately determine the desired level. If the linear discharge is fast, then the exponential decay is fast. If the linear discharge is slow, then the exponential decay is slow. The method can therefore involve selecting the rate of exponential decay based on the linear decay rate.Conversely, a value can also be exponentially reduced by the second circuit. Subsequently, a linear reduction can be performed by the third circuit. Further details are described below. To solve the problem, a procedure for determining an attention can include the following steps: Performing an initial vector-matrix multiplication. The initial vector-matrix multiplication can be performed digitally or analogously. Each value of the vector resulting from the initial vector-matrix multiplication can be generated in the form of an electrical signal. The generated electrical signal can be an electric current. The amount of charge in the generated electric current can be a measure of the value of the resulting vector. The greater the amount of charge, the greater the value can be. Each generated value can be converted into an electrical pulse.The timing of an electrically generated pulse can depend on the generated value. Specifically, the timing of an electrically generated pulse can depend on the amount of charge in the respective current, if the amount of charge in a generated electric current is a measure of the value of a generated vector. A pulse can occur later the larger the value. Therefore, the timing of an electrical signal is a measure of the value of a generated vector. Attention can be determined from the pulses. This means that attention is determined from the times at which pulses occur.The invention is based on the idea of ​​representing a vector using electrical signals, the timing of which is a measure of the vector's values. Determining attention from the pulses can include a second vector-matrix multiplication. This second vector-matrix multiplication can be performed analogously or digitally. Determining attention from the electrical pulses can also include time-dependent value reduction, which can be performed digitally or analogously. Values ​​generated using the pulses can be reduced in a time-dependent manner. A value generated as a function of the timing of an electrical pulse can subsequently be reduced, again depending on its timing.The earlier an electrical pulse occurs, the more a value determined using that pulse can be reduced. In other words, the earlier a value is generated, the greater the reduction can be. The time dependence of the reduction can be exponential. A value determined very early can be reduced very significantly. A value determined somewhat later can be reduced considerably less. Values ​​occurring late in the process can hardly be reduced at all. Values ​​occurring late in the process can be reduced only minimally. In this sense, the time dependence of the reduction can correspond to an exponential decay. The first vector-matrix multiplication can be performed using an analog circuit.Compared to a digitally performed vector-matrix multiplication, the problem of the invention can be solved more effectively in this way. The analog circuit can include memristors to perform a first vector-matrix multiplication. The values ​​of a matrix K. TThe resistance values ​​of the memristors can be programmed. Voltages applied to the analog circuit can represent the values ​​of a vector Q. A separate input can be provided for each value of a vector. Each input can be electrically connected to a plurality of memristors. A first input can be electrically connected to the first memristors of the circuit for multiplication. A second input can be electrically connected to other second memristors for multiplication, and so on. A resulting current, which depends on the voltage applied to the associated input and the resistance value of a memristor, is then a measure of the result of multiplying a value of a vector by a value of a matrix.The analog circuit designed to perform a first vector-matrix multiplication can have a separate output for each value of a vector resulting from the first vector-matrix multiplication. Each output can be electrically connected to a plurality of memristors to add currents. A first output can be electrically connected to a plurality of first memristors to add currents. A second output can be electrically connected to a plurality of second, other memristors to add currents, and so on. Electrical signals generated at the outputs of the analog circuit can therefore be the result of a vector-matrix multiplication. These can be analog electrical signals. An analog electrical signal can represent a quantity of charge. Determining an attention from the pulses can be accomplished using an analog circuit.Compared to a purely digital method of determining attention from pulses, this improved solution to the problem of the invention is possible. The circuit for determining attention can include an analog circuit that can operate like the analog circuit for determining a first vector-matrix multiplication. The circuit for determining attention can also include memristors so that the values ​​of a matrix V can be programmed as resistance values. The analog circuit for determining attention can have a separate input for each pulse. The circuit for determining attention from the pulses performs a multiplication as soon as a pulse arrives. Since the pulses arrive at different times, multiplications are performed at different times.The results of multiplications are thus available at different times. To reduce the number of multiplication results depending on when they occur, the result of a multiplication can be reduced by a preset factor for each unit of time. For example, this factor could be 0.9. The result of a multiplication will then be multiplied by 0.9 very often if the result is available very early. The later the result of a multiplication is available, the less often it is multiplied by 0.9. In this way, the number of multiplication results can be reduced as a function of time, according to an exponential decay.If the values ​​M1 and M2 are the results of initial multiplications obtained early due to early pulses, and M3 and M4 are values ​​of multiplication obtained later due to late pulses, then the following value of a vector could have been obtained by reducing and adding: M1·0.9. 9 + M2·0.9 8 + M3·0.9 2+ M4·0.9. The values ​​M1 and M2 have therefore been greatly reduced due to their early occurrence. The values ​​M3 and M4 have only been slightly reduced due to their late occurrence. The invention enables in-memory computing. In-memory computing refers to the practice of processing data directly in memory. This leads to faster application processing, which is particularly advantageous for data-intensive applications. The invention is explained in more detail below with reference to the figures. Figure 1 shows a circuit for performing a vector-matrix multiplication; Figure 2 shows a circuit for generating time-dependent electrical signals; Figure 3 shows a circuit for determining attention; Figure 4 shows an example of a circuit for integrating a time-dependent electrical signal; and Figure 5 shows a delay circuit.Figure 1 shows a circuit 1 with which vector-matrix multiplication can be performed. The circuit 1 comprises memristors r11 to r34 and electrical terminals v11 to v14 for applying electrical voltages. The values ​​of a matrix K can be entered into the memristors r11 to r34. TThe values ​​of a vector Q can be stored in the circuit. The values ​​of a vector Q can be applied to terminals V11 to V14 in the form of voltages. When corresponding voltages are applied to terminals V11 to V14, currents flow through memristors R11 to R34, the values ​​of which depend on the programmed resistances. The electrical currents flowing through memristors R11 to R14 flow via a common electrical line 2 to a first output of circuit 1 and are thus summed. The electrical currents flowing through memristors R21 to R24 flow via a common electrical line 3 to a second output of circuit 1 and are thus summed. The electrical currents flowing through memristors R31 to R34 flow via a common electrical line 4 to a third output of circuit 1 and are thus summed. The added electric currents thus represent the result of a vector-matrix multiplication.The summed electrical currents that have been routed to the outputs can be converted into electrical voltages by converters 5. These electrical voltages can charge capacitors 6 to temporarily store the result of the vector addition. A transistor 7 can be connected downstream of each memristor r11 to r34 to switch the function of each memristor r11 to r34 via terminals 8. The circuit 1 shown in Figure 1 allows for energy-efficient vector-matrix multiplication with minimal technical effort, for example, to perform a vector-matrix multiplication QK. T to be able to perform this. However, it is not absolutely necessary to use such an analog circuit 1. A vector-matrix multiplication, such as the matrix multiplication QK TAlternatively, the process can be performed digitally. The result of a digitally performed vector matrix multiplication can be converted into analog electrical signals using digital-to-analog converters. Figure 2 illustrates how a time-dependent voltage pulse is generated from the charge of an electric current. An electric current reaching the circuit shown in Figure 2 via an input IN is integrated by an integrator 9. The integrator 9 can include a capacitor for integration. The integrator can also include only one capacitor. Therefore, a capacitor can be the integrator itself, which is particularly preferable. A capacitor can be a capacitor 6 from Figure 1. Once the electric current has been integrated, the integrator 9 can then be discharged, for example, linearly. The resulting linearly decreasing voltage can be applied to the first inputs of two differential stages 10.The differential stages 10 can be comparators. A first reference voltage Vref1 can be applied to the second input of one differential stage 10, or one comparator, and a second reference voltage Vref2 can be applied to the second input of the other differential stage 10, or the second comparator. These two reference voltages, Vref1 and Vref2, define a voltage window. If the voltage applied to the first inputs of the two comparators 10 reaches a value within this window due to the discharge of the integrator 9, then two electrical switches 11 and 12, controlled by the differential stages 10, are open. An electric current can then flow through the two switches 11 and 12. If the voltage is outside the window, then at least one switch 11 or 12 is open. In this case, no current can flow through the two switches 11 and 12. The two switches 11 and 12 can be transistors.Once the integrator 9 has discharged sufficiently to leave the window defined by the reference voltages Vref1 and Vref2, a switch 11, 12 opens again, and no more current flows through the two switches 11, 12. The current flowing through the two switches 11, 12 during this time is therefore an electrical signal that was generated at a time dependent on the amount of charge integrated by the integrator. Instead of two comparators, one comparator and a delay circuit can be used, as this has proven to be more robust against process variations and mismatches. A delay circuit (also called a delay circuit) is an electronic circuit that deliberately delays a signal; that is, the output signal occurs with a specific time delay after the input signal.The circuit can include a discharge device 13, through which the integrator 9 can be discharged linearly, for example, at a desired time. The discharge device 13 can include a transistor as a switch to enable the discharge to be performed at a desired time. The discharge device 13 can include an electrical resistor through which the integrator 9 can be discharged. The value of the electrical resistor determines how quickly the integrator 9 can discharge. Instead of a resistor, a current mirror can be used for discharge. A current mirror is an electronic circuit that ensures that a specific electric current is constantly duplicated or "mirrored."**A current mirror typically consists of two (or more) transistors, usually bipolar transistors or MOSFETs, connected in such a way that the current through one transistor (reference current) is replicated by one or more other transistors. A current mirror allows for particularly precise control of the discharge. The circuit sketched in Figure 2 is provided for each value of a vector previously generated by vector-matrix multiplication. A third circuit, shown in Figure 3, can be used to determine an attention. This third circuit can perform a second vector-matrix multiplication and includes a circuit 1 for this purpose. Voltage pulses Vp1 to Vp4 can be applied to circuit 1 as soon as they occur. In addition to circuit 1, there is a circuit 14 that reduces the values ​​at the output of circuit 1 over time.The values ​​can be advantageously reduced exponentially to suitably determine the level of attention. Capacitors 16 integrate the currents generated by pulses. The capacitors 16 are discharged via the resistors 15. This generates voltage signals at the outputs of the circuit 14 that decay exponentially over time. Figure 4 shows an example of a circuit with one input IN and one output OUT for integrating incoming currents. The circuit comprises three ohmic resistors R1 to R3, two transistors T1 and T2, an operational amplifier OP, and a capacitor C. The charge of a current arriving via the input IN is integrated through the capacitor C. Discharge occurs as soon as transistor T1 is switched on. A current then flows through resistor R2.To ensure correct vector-matrix multiplication, a self-feedback operational amplifier (OP) can be included to provide a virtual ground. Switching on transistor T2 and resistor R3 ensures that the capacitor can be fully discharged initially. The circuit shown in Figure 4 can integrate an electrical signal. An embodiment of a circuit for generating time-dependent electrical signals is shown in Figure 5. The circuit shown can be used for each output of the circuit 1 (array) shown in Figure 3.This embodiment of a circuit for generating time-dependent electrical signals, shown in Figure 5, can connect a first such circuit 1 with a second such circuit 1 by applying pulses of constant length to the second circuit 1. The delay of these pulses is inversely linear and dependent on the magnitude of the respective output current of the first circuit 1 during a charging process. The circuit for generating time-dependent electrical signals can, as shown in Figure 5, comprise an input 17, a comparator 18, a buffer 19, an inverter delay 20, a current mirror 21, an AND gate 22, a pulse gate 23, and / or an output 24. In the default state, "charge" 25 is connected to ground (GND), and any currents at input 17 are routed to ground.The voltage “Vcap” across the integration capacitor is at GND potential. When “charge” 25 is pulled HIGH (step 1), currents flowing into input 17 are integrated across the integration capacitor, resulting in an increase in “V. cap This leads to the following: These currents are the result of the vector matrix multiplication of the first resistive array, i.e., circuit 1 in Fig. 3. After charging, “charge” 25 is pulled back to GND and “Vcap” is isolated from input 17. Now, a current “I” is applied to all parallel circuits simultaneously. discharge “applied to the current mirror. This is also applied to the node “V” by the current mirror. cap “Applies and discharges this linearly. Depending on the charge applied to the integration capacitor according to step 1, the switching voltage “V” is thThe comparator's switching point is reached at a different time. This point in time is later the more charge was present after step 1. This charge, in turn, depends on the current flowing through input 17 during the charging process. The switching point of comparator 18 is thus inversely linearly dependent on the current. A digital buffer 19 amplifies the comparator signal. The buffered comparator signal is delayed by an inverting inverter delay 20, which can be implemented, for example, as an inverter delay with additional delay resistors in the GND or VSS path. A digital AND gate 22 with three inputs ("3AND") controls, via the pulse gate 23, when a pulse is sent to the subsequent circuit 1 ("array"). As long as all three inputs to the AND gate 22 are HIGH, a pulse is applied.This is the case when comparator 18 has switched to HIGH, but this switching has not yet been propagated by inverter delay 20. Simultaneously, an "enable" signal is HIGH. The circuit shown in Figure 5 represents an alternative to the circuit shown in Figure 2.

Claims

Forschungszentrum Jülich GmbH G70313WO / PT 1.3110 PCTA Claims 1. Method for determining attention comprising the steps: performing a first vector-matrix multiplication such that each value of the vector resulting from the vector-matrix multiplication is generated in the form of an electrical signal; converting each generated electrical signal into a pulse whose occurrence depends on the magnitude of the electrical signal, wherein a pulse occurs later the larger the electrical signal; determining attention using the pulses.

2. Method according to claim 1, characterized in that determining attention from the pulses comprises a second vector-matrix multiplication.

3. Method according to any of the preceding claims, characterized in that determining attention from the pulses comprises a time-dependent reduction of values. 4.

5. A method according to any one of the preceding claims, characterized in that values ​​generated by the pulses are reduced in a time-dependent manner.

6. A method according to any one of the preceding claims, characterized in that the earlier a value was generated, the greater the reduction.

7. A method according to any one of the preceding claims, characterized in that the time dependence of the reduction is exponential.

8. A method according to any one of the preceding claims, characterized in that the first vector-matrix multiplication is performed using an analog circuit.

9. A method according to any one of the preceding claims, characterized in that the second vector-matrix multiplication is performed using an analog circuit. 2 9. A method according to one of the preceding claims, characterized in that the determination of attention from the pulses is carried out using an analog circuit.

10. A method according to one of the two preceding claims, characterized in that the analog circuit comprises memristors.

11. A device for carrying out a method according to one of the preceding claims, comprising a first electrical circuit (1) configured to perform vector-matrix multiplication and output the result of the vector-matrix multiplication in the form of electrical signals, a second electronic circuit configured to convert each electrical signal into a time-dependent electrical pulse, and a third electronic circuit (1, 14) configured to determine attention from the time-dependent electrical pulses. 12.Device according to the preceding claim, characterized in that the first electrical circuit and / or the second electrical circuit comprises memristors (r12 to r34).

13. Device according to one of the two preceding claims, characterized in that the second electrical circuit comprises a capacitor (C) for integrating an electrical signal.

14. Device according to the preceding claim, characterized in that the second circuit comprises a discharge device (13) for the capacitor (C).

15. Device according to one of the two preceding claims, characterized in that the second circuit comprises an operational amplifier (op).

16. Device according to one of the two preceding claims, characterized in that the third circuit (14) comprises an electrical resistor (15) and a capacitor (16).

Images