Speech recognition system and method based on lightweight Transformer network

Abstract

The present invention relates to speech recognition technology in the field of artificial intelligence, and specifically discloses a speech recognition system and method based on a lightweight Transformer network. The system includes a preprocessing module, a first forward propagation module, a multi-head attention module, a convolution module, a second forward propagation module and a fully connected layer module. The system is designed through modular hardware and some parameters are implemented using a memristor cross array. The system not only has a lighter network model, but also has good recognition accuracy under different hardware scales, can effectively reduce the energy consumption and area overhead of the circuit, can realize speech recognition tasks on edge devices with low power consumption and light structure, and has good stability and noise resistance.

Description

Technical Field

[0001] The present invention relates to speech recognition technology in the field of artificial intelligence, and more specifically, to a speech recognition system and method based on a lightweight Transformer network. Background Art

[0002] With the development of artificial intelligence, deep neural networks have been widely used, especially in complex tasks such as image processing, speech recognition, and machine translation. As the challenges of the tasks gradually increase, the industry has also proposed many more complex network models, among which the Transformer network (TN for short) is a very typical deep neural network model and has attracted widespread attention in the industry. At the same time, many improvement measures based on the classic TN model have gradually been proposed.

[0003] Traditional TN models such as Figure 1 As shown, the core idea is to rely on encoder and decoder architectures, which are mainly based on Figure 2 The attention mechanism shown in Figure 2 It can be seen that the attention mechanism is actually through the query matrix W Q , key matrix W K Sum matrix W V Calculate the single-head attention of the input information feature X, and the multi-head attention is to connect multiple single-head attentions in series, using the weight matrix W Z Map it to the final output. With the development of TN, its training cost has become unaffordable for researchers, and even caused a lot of natural resource overhead. For example, in order to translate a sentence of only 30 words, a TN model often requires more than 10G of cumulative calculation parameters. To improve this problem, many improved models have introduced convolutional layers, including depthwise separable convolutions, lightweight convolutions, and GLUs (gated linear units), which alleviate the problem of the large number of parameters caused by pure attention mechanisms, while reducing the number of parameters while maintaining or even exceeding the previous accuracy. Despite this, deploying large-size TN network models to edge devices requires sacrificing a lot of hardware resources and is still challenging.

[0004] Memristor is a nonlinear and nonvolatile device with memory function, which makes it an ideal device for artificial intelligence with integrated storage and computing functions. In addition, since the separation of memory and computing unit, the classic von Neumann computing architecture (such as CPU, GPU) still has a great impact on system power consumption and latency. Therefore, a new neural network computing architecture is urgently needed to alleviate the impact of the separation of storage and computing. Memristor-based neural network computing can generate large-scale accumulative multiplication circuits while consuming very little power, which makes memristor-based neural network computing a reliable choice for implementing various neural networks and algorithms.

[0005] Therefore, more and more researchers are trying to deploy TN networks on memristor-based memory computing hardware for practical applications. However, due to the complex structure and large-scale parameters of TN networks, existing memristor-based circuit implementations are still too cumbersome for TN applications and cannot be applied to edge devices. At the same time, no memristor-based TN network has been found to complete speech recognition-related work, so it is very necessary to provide a reliable and efficient hardware implementation solution on the memristor neuromorphic computing framework to achieve speech recognition tasks. Summary of the invention

[0006] In view of this, the present invention provides a speech recognition system based on a lightweight Transformer network. Aiming at the application requirements of speech recognition, the lightweight convolution module circuit is integrated into the memristor-based TN network to achieve a significant reduction in the number of system parameters, so that it meets the edge device deployment requirements and ensures its feasibility in practical applications.

[0007] To achieve the above purpose, the specific technical solutions adopted by the present invention are as follows:

[0008] A speech recognition system based on a lightweight Transformer network includes a preprocessing module, a first forward propagation module, a multi-head attention module, a convolution module, a second forward propagation module and a fully connected layer module, wherein:

[0009] The preprocessing module is used to collect voice data and extract feature parameters in the voice data to form training samples, verification samples and test samples;

[0010] The first forward propagation module includes a first layer regularization circuit, a first linearization circuit, a first Swish activation function circuit, a second linearization circuit and a first addition circuit. The first layer regularization circuit is used to perform regularization processing on each feature parameter extracted by the preprocessing module. The regularized data is processed by the first linearization circuit, the first Swish activation function circuit and the second linearization circuit in sequence, and then the first addition circuit implements residual connection and transmits it to the multi-head attention module; at least part of the parameters in the first forward propagation module are implemented by a memristor cross array circuit;

[0011] The multi-head attention module includes a second-layer regularization circuit, a query matrix circuit, a key matrix circuit, a value matrix circuit, a first storage module, a first multiplication-accumulation circuit, a SoftMax activation function circuit, a second multiplication-accumulation circuit, a second addition circuit and a second storage module; the second-layer regularization circuit performs regularization processing on each signal output by the first forward propagation module, the query matrix circuit, the key matrix circuit and the value matrix circuit are implemented by a memristor cross array circuit, and are used to calculate the single-head attention of the input information feature, the first storage array is used to implement the cache of multiple single-head attentions, and is selectively output to the first multiplication-accumulation circuit under the drive of the control signal to obtain the product of the output voltage of each row of the query matrix and the output voltage of each column of the key matrix, each output of the first multiplication-accumulation circuit is processed by the SoftMax activation function circuit, and then the second multiplication-accumulation circuit is used to implement the product operation with the output voltage of each column of the value matrix, and finally the second addition circuit is used to perform residual connection and store the calculation result in the second storage module;

[0012] The convolution module includes a third layer regularization circuit, a first point-wise convolution module, a gated linear unit, a depth convolution circuit and a second point-wise convolution module;

[0013] The circuit structure of the second forward propagation module is connected to the circuit structure of the first forward propagation module, and is used to forward propagate the output of the convolution module, and finally obtain the final output through the fully connected layer module;

[0014] The parameters in the first forward propagation module, the multi-head attention module, the convolution module, the second forward propagation module and the fully connected layer module are trained through training samples, and the trained system circuit is verified using verification samples. Finally, the trained system circuit is used as a speech recognition system to perform speech recognition on test samples.

[0015] Optionally, the preprocessing module preprocesses the acquired speech data using Mel-spectrogram coefficients and extracts 20 features from each frame of speech.

[0016] Optionally, a signal selection switch is provided on each input end of the first layer regularization circuit in the first forward propagation module to realize the selective input of each signal feature.

[0017] Optionally, the first layer regularization circuit is based on Regularize each input signal feature, where V x Represents the input signal characteristics, R f represents the preset feedback resistor in the first layer regularization circuit, M1 is the first weight coefficient matrix, V bias1 It is a first bias voltage matrix, and the first weight coefficient matrix and the first bias voltage matrix are implemented by a first memristor cross array circuit.

[0018] Optionally, the first Swish activation function circuit is used to implement: The signal conversion of V d is the input signal of the first Swish activation function circuit, Vsh is the output signal of the first Swish activation function circuit, β is a preset hyperparameter, V bias3 is the bias voltage.

[0019] Optionally, the second-layer regularization circuit has the same structure as the first-layer regularization circuit, and in the finally trained system circuit, the first multiplication-accumulation circuit and the second multiplication-accumulation circuit are implemented using a memristor crossbar array.

[0020] Optionally, the structure of the third-layer regularization circuit is the same as that of the first-layer regularization circuit, a third storage module is arranged at the output end of the gated linear unit, and a batch regularization circuit, a second Swish activation function circuit and a fourth storage module are arranged at the output end of the deep convolution circuit. The second Swish activation function circuit has the same structure as the first Swish activation function circuit. The third storage module is used to send multiple row voltages processed by the third-layer regularization circuit to the deep convolution circuit. The output signal of the deep convolution circuit passes through the batch regularization circuit and the second Swish activation function circuit and is sent to the second point-to-point convolution module by the fourth storage module. Finally, the third addition circuit is used to perform residual connection, wherein the weight of the convolution kernel in the deep convolution circuit is realized by a memristor cross array.

[0021] Optionally, the kernel size of the first point-wise convolution module and the second point-wise convolution module is 1×1, and the kernel size of the deep convolution circuit is C1×C1, where C1 is the number of signal columns output by the multi-head attention module.

[0022] Optionally, the SoftMax activation function circuit is provided with an exponential function circuit, two transistors, an operational amplifier and a voltage divider, for implementing signal processing, where V in Vst is the multi-channel input signal of the SoftMax activation function circuit. m is the output signal of the SoftMax activation function circuit, is a preset proportional coefficient, and m is the total number of input signal paths.

[0023] Based on the above system, the present invention also provides a speech recognition method based on a lightweight Transformer network, the key of which is to construct the speech recognition system described above, determine the values ​​of each parameter through pre-training, and then use a memristor cross array to implement the corresponding parameter configuration, thereby forming a speech recognition model for speech recognition.

[0024] The significant effect of the present invention is: by introducing a lightweight convolution module circuit into the traditional TN network model, the system is designed through modular hardware and some parameters are implemented using a memristor cross array. It not only has a lighter network model, but also has good recognition accuracy under different hardware scales, can effectively reduce the energy consumption and area overhead of the circuit, can realize speech recognition tasks on edge devices with low power consumption and light structure, and has good stability and noise resistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 This is a diagram of a classic TN network architecture in the prior art;

[0027] Figure 2 This is a diagram of the logical architecture of the attention mechanism in the prior art;

[0028] Figure 3 This is a diagram of the MLTN network architecture proposed by the present invention;

[0029] Figure 4 This is a block diagram of the MLTN network circuit module proposed by the present invention;

[0030] Figure 5 It is a schematic diagram of a forward propagation module and an addition circuit in a specific embodiment of the present invention;

[0031] Figure 6 It is a circuit diagram of a multi-head attention module in a specific embodiment of the present invention;

[0032] Figure 7 is a circuit schematic diagram of a convolution module in a specific embodiment of the present invention;

[0033] Figure 8 It is a schematic diagram of the execution principle of the memristor crossbar array in a specific embodiment of the present invention;

[0034] Fig. 9 It is a circuit schematic diagram of each basic functional module in a specific embodiment of the present invention;

[0035] Fig.10 The architecture diagram of the speech recognition system based on the lightweight Transformer network provided by the present invention;

[0036] Fig.11 It is a schematic diagram of characteristic information of a voice command after preprocessing in a specific embodiment of the present invention;

[0037] Fig.12 It is a schematic diagram of MLTN network size parameter setting in a specific embodiment of the present invention;

[0038] Fig.13 It is the system training loss curve in a specific embodiment of the present invention;

[0039] Fig.14 The signal control clock curve in the specific embodiment of the present invention;

[0040] Fig.15 Schematic diagram of weight distribution after adding positive and negative pulses to the memristor crossbar array;

[0041] Fig.16 Schematic diagram of simulation verification results of a specific embodiment;

[0042] Fig.17 It is a schematic diagram of the simulation results of the functional module circuit at different temperatures;

[0043] Fig.18 The changes in input and output signals when white noise is added to the spoken word "four". DETAILED DESCRIPTION

[0044] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] like Figure 3 and Figure 4 As shown, this embodiment first provides a speech recognition system based on a lightweight Transformer network, whose network model is defined as an MLTN model, including a preprocessing module, a first forward propagation module, a multi-head attention module, a convolution module, a second forward propagation module and a fully connected layer module, wherein:

[0046] The preprocessing module is used to collect voice data and extract feature parameters in the voice data to form training samples, verification samples and test samples;

[0047] The first forward propagation module (FNN1 module for short) includes a first regularization circuit, a first linearization circuit, a first Swish activation function circuit, a second linearization circuit and a first addition circuit. The first regularization circuit is used to perform regularization processing on each feature parameter extracted by the preprocessing module. The regularized data is processed by the first linearization circuit, the first Swish activation function circuit and the second linearization circuit in sequence, and then the first addition circuit realizes residual connection and transmits it to the multi-head attention module; at least part of the parameters in the first forward propagation module are realized by a memristor cross array circuit;

[0048] The multi-head attention module includes a second-layer regularization circuit, a query matrix circuit, a key matrix circuit, a value matrix circuit, a first storage module, a first multiplication-accumulation circuit, a SoftMax activation function circuit, a second multiplication-accumulation circuit, a second addition circuit and a second storage module; the second-layer regularization circuit performs regularization processing on each signal output by the first forward propagation module, the query matrix circuit, the key matrix circuit and the value matrix circuit are implemented by a memristor cross array circuit, and are used to calculate the single-head attention of the input information feature, the first storage array is used to implement the cache of multiple single-head attentions, and is selectively output to the first multiplication-accumulation circuit under the drive of the control signal to obtain the product of the output voltage of each row of the query matrix and the output voltage of each column of the key matrix, each output of the first multiplication-accumulation circuit is processed by the SoftMax activation function circuit, and then the second multiplication-accumulation circuit is used to implement the product operation with the output voltage of each column of the value matrix, and finally the second addition circuit is used to perform residual connection and store the calculation result in the second storage module;

[0049] The convolution module includes a third layer regularization circuit, a first point-wise convolution module, a gated linear unit, a depth convolution circuit and a second point-wise convolution module;

[0050] The circuit structure of the second forward propagation module (FFN2 for short) is similar to that of the first forward propagation module, and is used to forward propagate the output of the convolution module, and finally obtain the final output through the fully connected layer module;

[0051] The parameters in the first forward propagation module, the multi-head attention module, the convolution module, the second forward propagation module and the fully connected layer module are trained through training samples, and the trained system circuit is verified using verification samples. Finally, the trained system circuit is used as a speech recognition system to perform speech recognition on test samples.

[0052] In the specific implementation, the MLTN circuit adds residual connections in the forward propagation process, so that the network can obtain more global information. In order to further understand the working principle of the first forward propagation module (FFN1) and the second forward propagation module (FFN2), combined with Figure 5 It can be seen that the traditional ReLu activation function is replaced by the Swish activation function, and two forward propagation modules are used to clamp the multi-head attention module and the convolution module. A signal selection switch is set on each input end of the first layer of the regularization circuit in the first forward propagation module to realize the selective input of each signal feature. Suppose that the speech information features output by the preprocessing module have n items, recorded as VX 1,m , m = 1 ~ n, the routing switch signal SW0 controls VX 1,m The input is passed to the layer regularization circuit, and the specific implementation scheme of the layer regularization circuit is given by Fig. 9 It can be seen that the layer regularization circuit follows: Regularize each input signal feature, and the voltage V after layer regularization is norm is input in parallel to the first linearization circuit, where the weight matrix W of the linear layer N =[W1,W2,...W n ], bias matrix W b =[W b1 ,W b2 ,...W bn ], we realize the mapping of linearization layer parameters through a memristor crossbar array. The two linearization layers after linear mapping can be represented by memristor values ​​M1 = [R11, R12, ... R1 n ] and memristor M2=[R21,R22,...R2 n ], therefore, the circuit expression of the two linearization layers can be expressed as: VL1 m =M1·V norm m , VFFN1 1,m =M2·V norm m In addition, the specific circuit implementation of the Swish function is also shown in Fig. 9 In the Swish function, the input voltage V d =VL1 m , combined with Fig. 9 It can be seen that the Swish activation function circuit is used to achieve: signal conversion, β is preset to 1, V bias3 is the bias voltage. After forward propagation, the linearized output is residually connected and passed to the next layer. The residual connection subcircuit is shown in Figure 5At the bottom, it is implemented based on the addition circuit. The addition circuit can realize the residual connection of each link in this system. After the first feature information is input, the V input in the addition circuit is m =VFFN1 1,m , then the final output of the FFN block is: VFFN1 1,m =VFFN1 1,m +VX 1,m .

[0053] For the multi-head attention module, we integrate the memristor crossbar array with various basic module circuits to realize the multi-head attention module, as shown below: Figure 6 As shown, Figure 6 The calculation process of the attention module is given, and the corresponding output voltage VAtten is obtained m,n Specifically, first, the memristor crossbar array W Q , W K and W V Process the output voltage VFFN1 of a row of FFN1 modules respectively 1,n And get the output voltage V qC1 , V kC2 and V vC2 In addition, the control signals (LR1, LR2...LR m ) controls the plurality of memory units (MUs) in the first memory module to temporarily store the intermediate output, at the input voltage VFFN1 of all rows m,n After being converted, SW1 controls the intermediate voltage V qC1 and V kC2 The output of the first multiplication and accumulation circuit calculates the product of the two to obtain the voltage Vqk C2,C1 Since the transpose of K is multiplied by Q, the first multiplication-accumulation circuit is used to multiply each row voltage of Q and each column voltage of K. Finally, after the SoftMax operation, V vC2 and Vst C2,C1 are simultaneously sent to the second multiplication and accumulation circuit to calculate the product of the two. SW2 is used to control the storage V vC2 The switch of MU shows that the final output voltage Vatten of the single-head attention mechanism m,C1 It is expressed as:

[0054] Vatten m,C1 =Vst C2,C1 ·V vC2 ,where Vst C2,C1 =Soft Max(Vqk C2,C1 )

[0055] In specific implementation, the SoftMax activation function circuit is provided with an exponential function circuit, two transistors, an operational amplifier and a voltage divider for realizing signal processing, where V in Vst is the multi-channel input signal of the SoftMax activation function circuit. m is the output signal of the SoftMax activation function circuit, is a preset proportional coefficient, and m is the total number of input signal paths.

[0056] It is worth mentioning that in the intermediate process, multipliers can be used instead of memristor cross arrays to implement multiplication and accumulation calculations. The reason is that the use of memristor cross arrays only performs weight mapping once, but due to the changes in matrix elements, multiple weight mappings are required here. In addition, if the weights of the memristors are updated online, additional programming circuits and control circuits are required, which greatly increases the area and power consumption of the circuit. Therefore, when the present invention is implemented, each corresponding pair of elements in the two matrices are multiplied by a multiplier, and then the outputs of the two operational amplifiers are added to obtain the output voltage of each column of the multiplier. In addition, before performing the residual connection, the output channel of the attention block is expanded to the same size as the input feature, and finally a residual connection is performed. The final output of the attention layer can be expressed as: V Atten m,n =V atten m,n +FFN m,n , through the scheduling signal SW2, the output of the attention module is stored in Mus in turn. In addition, the signal (LC1, LC2...LC n ) controls the output of MUs from the 1st to the nth columns.

[0057] Combination Figure 7 It can be seen that in the specific implementation, a third storage module is provided at the output end of the gated linear unit, and a batch regularization circuit, a second Swish activation function circuit and a fourth storage module are provided at the output end of the deep convolution circuit. The second Swish activation function circuit has the same structure as the first Swish activation function circuit. The row voltages processed by the third layer regularization circuit are sent to the deep convolution circuit through the third storage module. The output signal of the deep convolution circuit passes through the batch regularization circuit and the second Swish activation function circuit, and is sent to the second point-to-point convolution module by the fourth storage module. Finally, the third addition circuit is used for residual connection, wherein the weight of the convolution kernel in the deep convolution circuit is realized by a memristor cross array.

[0058] Assume that the kernel size of the first point-wise convolution module and the second point-wise convolution module is 1×1, and the kernel size of the deep convolution circuit is C1×C1, where C1 is the number of signal columns output by the multi-head attention module.m,n It is input column by column into the first point-by-point convolution module to obtain VP1 with twice the number of channels C1. 2C1 Then, each column voltage passes through GLU to obtain a column voltage VG with a size equal to half of the original dimension m,C1 , in LG1,...LG m Further, the stored C1×C1 row voltages VG C1,C1 are simultaneously input into the deep convolution circuit module to obtain the corresponding output voltage VD . Finally, after all the depthwise convolutions are completed, the voltages stored in all MUs are input to the second point-wise convolution layer through the control of SW3. Specifically, GLU is implemented based on multiple Sigmoid circuit modules. Because the GLU operation needs to calculate multiple inputs at the same time, it should be noted that the point-to-point product (Hadamard product) of the elements only multiplies the elements at the corresponding positions in the input matrix, and does not accumulate the row and column vectors of the input matrix. Therefore, refer to Figure 6 As shown in the figure, we add a number of multiply-accumulate modules (MACs) in one column equal to C1 The output of the point-to-point product of the elements is expressed as: B∈[VP C1+1 ,VP 2C1 ]

[0059] Where M = m-C1 2 +1 is the number of output channels of the depth convolution layer, and the final output Vconv m,o As shown below:

[0060] The number of output channels o is equal to the number of dimensions d, and W P1 ,W P2 ,W D1 They represent the weights of the convolution kernels respectively. The weights of the same convolution kernels can be realized by linear mapping of the memristor cross array, where: Then Vconv m,o It is sent to the second forward propagation module (FFN2) and finally connected to the fully connected layer (FC) with a prediction channel number of Z, where Z is equal to the number of categories of the speech recognition task.

[0061] In the specific implementation, in addition to the memristor crossbar array, the functional circuit also includes normalization and activation function modules (SoftMax, Sigmoid and Swish), all of which can be directly used in the circuit implementation of other neural networks. The widely used memristor crossbar arrays mainly include 1M, 2M, 2M, 1T1R, and 1T1M. In this embodiment, the 1T1M structure is used to implement VMM and convolution operations. In addition, a pair of memristors can be regarded as a neuron. Therefore, the memristor crossbar array can realize linear mapping and MAC operations. Figure 8 The specific circuit implementation steps of the memristor crossbar array are shown. The input two-dimensional feature information X m,n is converted line by line into V m,n , and then fed into the memristor crossbar array. Since memristors cannot store negative values, a typical two-column frame is used to represent a column of weights.

[0062] Based on the above description, this embodiment also provides a speech recognition method based on a lightweight Transformer network, constructs the speech recognition system mentioned above, determines the values ​​of each parameter through pre-training, and then uses a memristor cross array to implement the corresponding parameter configuration, thereby forming a speech recognition model for speech recognition.

[0063] Combination Fig.10 As shown, the MLTN model proposed in the present invention is first trained on the PyTorch platform, and then the reasoning of the speech recognition task is implemented on the proposed hardware circuit. Fig.10 The working process of speech recognition is shown. It is worth noting that in order to achieve the lightweight design of the proposed model, the depth and size of the network are reduced as much as possible, making it possible for the designed MLTN to be used in edge applications.

[0064] In the specific implementation, the voice command dataset provided by the Google platform was used for network training, and three types of commands were selected, including "forward", "four", and "off". All samples in the training set included test and validation samples. More than 7,000 samples were used to complete the training of the network model. The voice data was preprocessed using Mel-spectral coefficients (MFCC) to convert the voice wave from the time domain to the frequency domain, and 20 features were obtained in each frame of voice. The feature information of the voice command after preprocessing is as follows: Fig.11 As shown, one frame of feature information is marked with a colored rectangle, and the horizontal axis represents the number of frames corresponding to a complete speech wave. All of them are then input into the network model frame by frame for training.

[0065] Fig.12The specific dimensions of MLTN are described, in which a single-head attention mechanism is used instead of multi-head attention, which significantly reduces the model size. In addition, the size scaling process of the intermediate forward propagation layer is removed, and the original input size is maintained throughout the inference process. The convolutional layer first expands the incoming attention to twice the input dimension through point convolution, and then halves the input feature information dimension through GLU. After completing all the forward propagation layers, the model finally divides the output features into three categories. This embodiment uses the cross entropy loss function and the Adam method to train the entire model. Fig.13 The loss curve of MLTN after training for 300 epochs is shown, which indicates that the proposed MLTN model can be used to complete speech recognition tasks after training. In addition, once the weights of the trained network are loaded into the circuit module, the complete circuit implementation of MLTN can be achieved.

[0066] Combined with the circuit described above, during the design process, R f =1KΩ, marked as R f A multiple of means that the resistor is replaced by a resistor of the corresponding size, and all bias voltages are equal to 1V. However, when the trained weights are loaded into the memristor, all memristors should be fixed. Therefore, the memristors are controlled to work within the threshold voltage range, and before simulation, their input voltage is according to the formula: V x =V x ×10 -3 (V) scaling is performed so that the inputs remain in the mV level.

[0067] Table 1 lists the specific parameters of the hardware circuit used for speech recognition tasks. In addition, the hardware circuit of the proposed MLTN is implemented according to the circuit configuration of Table 1.

[0068] Table 1: Circuit configuration for speech recognition

[0069]

[0070] Fig.14 The time periods of the control signals SW, LR and LC are shown, where SW plays a decisive role in the entire circuit. Each circuit module requires at least one time period t = 10 μs to provide a stable output. SW0 determines the input of the preprocessed feature information, and the total time to process all features is T1 = 5m×t μs. In addition, SW2 determines the order of the input voltage of the attention module, and the time cost is T3 = (m+3)×t μs. In addition, SW3, LO M and LG m The total time period of the control convolution module is T4 = M × t μs. In addition, T2 and T5 are fixed time periods equal to 2t, which is independent of the number of input frames m.

[0071] Fig.15 Figure (a) shows the change curve of the memristor value with the pulse voltage when the pulse voltage is higher than the threshold voltage. The figure shows the effect of adding 0.5s negative pulse (-1V, -1.5V, -2V) and 0.5s positive pulse (1V, 1.5V, 2V) to the memristor model. The MLTN weight mapping to the memristor crossbar array mainly uses a proportional mapping scheme, in which the conductivity is proportional to the synaptic weight, because it minimizes the complexity of the circuit required for mapping. When determining the discrete level, we map the continuous weights to these levels by rounding, that is, mapping them to the closest discrete level during the quantization process. In this embodiment, the level is defined as 16. Fig.15 The weight distribution corresponding to the memristor crossbar array in the attention module is shown in (b). In addition, the weights are orthogonally mapped to the range of memristor values ​​[5KΩ, 80KΩ].

[0072] In order to further understand the effect of the present invention, the following circuit simulation is used to verify that Fig.16 It can be seen that Fig.16 (a) is an example of voltage conversion for the phonetic word ‘forward’. Fig.16 (b) Comparison of the input and output of the SoftMax function circuit in the attention module. Fig.16 (c) Comparison of simulation and inference results in the attention module circuit. Fig.16 (d) is the simulation result of speech recognition implemented by MLTN circuit. Fig.16 (a)- Fig.16 (c) illustrates the resulting changes of the attention module and shows the details of the voltage conversion process. The voltage corresponding to the normalized feature information of one frame is used as the input of the attention module. The simulation output of the SoftMax function module stabilizes after 20μs, and the sum of all 32 intermediate outputs of the attention calculation is always equal to 1V; the attention module fully uses 10 clocks to maintain a stable state. In addition, the difference between the circuit simulation and software inference of the attention module does not exceed 0.2mV. In addition, three types of speech are imported into the MLTN circuit in the order of ‘forward’, ‘off’ and ‘four’ to achieve a 3-label speech recognition task. Fig.16(d) shows the corresponding single distribution of three predictions, where the dashed line indicates that the simulation result is equal to the inference value. Specifically, the first prediction result 'forward' is consistent with the actual input, and its circuit simulation result is almost consistent with the inference result. It is worth mentioning that the prediction results are distributed on both sides of the dashed line, mainly due to the non-ideal characteristics of the memristor distribution. Although the error between the output value and the expected value is caused by the approximate operation in the mapping process, it does not affect the final prediction result.

[0073] As device size shrinks and chip power consumption increases, temperature is one of the main issues that cause transistor performance degradation in addition to the natural aging of the device. In the embodiment, we assume that the memristor operates in a stable state unless the operating voltage exceeds the threshold voltage of the memristor. In addition, a transistor is included in the designed activation function circuit to generate an analog voltage signal in the form of an exponential. Fig.17 The test results of the functional module circuit at 7 different temperature levels are shown. Fig.17 (a) is the result of the SoftMax module circuit when the input voltage is four degrees (2, 1, -1, -2); the simulation time is equal to 20μs to maintain a stable state. Fig.17 (b) and Fig.17 (c) The results of the Sigmoid and Swish module circuits when the input voltage is equal to 2mV. Fig.17 (d) shows the change in the test accuracy of the functional module circuit at different temperatures.

[0074] On this basis, the output accuracy of the module circuit due to temperature changes is calculated. Although the voltage input to the SoftMax module circuit is always changing, the sum of the SoftMax circuit is always equal to 1V, which can be expressed as Where i is the channel of the output voltage. In addition, it takes 20μs for the SoftMax module circuit to maintain a stable state. Due to the introduction of the temperature compensation circuit, the error is very small when working at different temperatures, less than 0.5%.

[0075] Memristor is a nonlinear device, so in practical applications, it is also necessary to consider the characteristics of the memristor model, such as retention, variability, and durability. In the simulation experiment, the ideal characteristics of the membrane resistor are simulated by quantizing and grading the membrane resistance; that is, when the threshold voltage is not exceeded, the conductivity of the membrane resistor is in a stable state. In addition, by adding white noise, the anti-interference property of the memristor crossbar array is explored. Fig.18The input-output changes after adding white noise with an intensity of 0.5 to the input features are shown. It shows that adding white noise with an intensity of 0.5 has a great impact on both the input and output voltage amplitudes, however the final test result (one-hot) remains unaffected. Therefore, the output of the circuit can be smoother when the white noise input intensity is lower.

[0076] The analysis of circuit resource consumption is as follows: In this embodiment, each pair of memristors corresponds to a weight parameter, which reduces the impact of the non-ideal characteristics of the memristors. At the same time, due to the 2M frame of the memristor cross array, the area overhead increases. Table 2 shows the power consumption of the main module when m feature frames are input, and also shows its corresponding area overhead. The input voice command 'forward' is used as a benchmark to calculate the circuit source overhead of the designed circuit. Specifically, the corresponding number of input frames is m=127. At the same time, all input voltages converted by MFCC are limited to the voltage range [-0.1V, 0.1V], so the maximum energy overhead of a neuron can be estimated to be (0.1V) 2 ×2 / 5KΩ=4μW. In addition, since the main energy consumption in the proposed circuit comes from the memristor crossbar array, it is feasible to evaluate the estimated energy consumption by calculating the power consumption of the memristor crossbar array. Q , W K and W V The sizes of are 20×4, 20×8, and 20×8 respectively. Therefore, the maximum power consumption of the attention module can be estimated as P q =40.64mW and P kv =81.28mW. The memristor crossbar array in the convolution module W P1 , W P2 and W D1 The sizes of are 20×4, 127×20 and 4×16 respectively, so their maximum power consumption is estimated to be P p1 =40.64mW and P p2 =10.16mW. It is worth noting that in order to improve the processing efficiency of deep convolution, 20 identical memristor crossbar arrays are used to implement W D1 , whose power consumption is evaluated as P d =5mW. In addition, P F =409.84mW represents the power consumption of all forward propagation realized by the memristor crossbar array.

[0077] Table 2: Main power consumption and area overhead of MLTN circuits

[0078]

[0079] In this paper, in addition to the time cost from the memristor crossbar array, the scheduling time cycle and the time of the intermediate module circuit are also taken into account. In integrated circuits, MOSFETs, transistors, resistors, and other basic components are all nanometer-scale. Therefore, when calculating the overall circuit module area overhead, the size of each component is set to; an operational amplifier consists of 8 MOSFETs. Table 2 provides a comparison of the MLTN circuit with other hardware baselines. In the 3-label speech recognition task benchmark, the area and power consumption of MLTN are much smaller than the overhead of the hardware baseline of edge devices implemented based on other computing architectures. Although the circuit area overhead increases due to parallel operations on all module circuits, the multipliers in the MAC and GLU modules can be completed by timing to reuse one multiplier. Therefore, the main area overhead can be evaluated by the on-chip area of ​​the memristor crossbar array.

[0080] However, the on-chip area is also related to the parameters of the model, which means that when the model is larger, more features need to be processed in parallel, resulting in an increasing on-chip area overhead. For example, when the hardware implements a traditional TN with 65M parameters, its area overhead is as high as 1.3×10 -2 mm 2 Therefore, in order to compare with the most advanced models, a speech recognition task with 35 labels was implemented on the Speech Command dataset. In addition, in order to simplify the model for easy deployment, the model size was reduced to 18.03K. After simulation testing, the speech recognition accuracy reached 95.03%, which is sufficient to correctly implement the speech recognition task on hardware.

[0081] In summary, it can be seen that the speech recognition system and method based on the lightweight Transformer network provided by the present invention introduces a lightweight convolution module circuit into the traditional TN circuit, which not only makes the TN lighter, but also effectively reduces the energy and area overhead of its circuit. In addition, the characteristics of the circuit are analyzed and verified by simulation on LTSPICE. By comparing with ARM, CPU and GPU, the results show that the MLTN circuit designed by the present invention can realize speech recognition tasks on edge devices with low power consumption and light structure.

[0082] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A speech recognition system based on a lightweight Transformer network, characterized in that: It includes a preprocessing module, a first forward propagation module, a multi-head attention module, a convolution module, a second forward propagation module and a fully connected layer module, where: The preprocessing module is used to collect voice data and extract characteristic parameters in the voice data to form training samples, verification samples and test samples; The first forward propagation module includes a first layer regularization circuit, a first linearization circuit, a first Swish activation function circuit, a second linearization circuit and a first addition circuit. The first layer regularization circuit is used to perform regularization processing on each feature parameter extracted by the preprocessing module. The regularized data is processed by the first linearization circuit, the first Swish activation function circuit and the second linearization circuit in sequence, and then the first addition circuit implements residual connection and transmits it to the multi-head attention module; at least part of the parameters in the first forward propagation module are implemented by a memristor cross array circuit; The multi-head attention module includes a second-layer regularization circuit, a query matrix circuit, a key matrix circuit, a value matrix circuit, a first storage module, a first multiplication-accumulation circuit, a SoftMax activation function circuit, a second multiplication-accumulation circuit, a second addition circuit and a second storage module; the second-layer regularization circuit performs regularization processing on each signal output by the first forward propagation module, the query matrix circuit, the key matrix circuit and the value matrix circuit are implemented by a memristor cross array circuit, and are used to calculate the single-head attention of the input information feature, the first storage array is used to implement the cache of multiple single-head attentions, and is selectively output to the first multiplication-accumulation circuit under the drive of the control signal to obtain the product of the output voltage of each row of the query matrix and the output voltage of each column of the key matrix, each output of the first multiplication-accumulation circuit is processed by the SoftMax activation function circuit, and then the second multiplication-accumulation circuit is used to implement the product operation with the output voltage of each column of the value matrix, and finally the second addition circuit is used to perform residual connection and store the calculation result in the second storage module; The convolution module includes a third layer regularization circuit, a first point-wise convolution module, a gated linear unit, a depth convolution circuit and a second point-wise convolution module; The circuit structure of the second forward propagation module is the same as that of the first forward propagation module, and is used to forward propagate the output of the convolution module, and finally obtain the final output through the fully connected layer module; The parameters in the first forward propagation module, the multi-head attention module, the convolution module, the second forward propagation module and the fully connected layer module are trained by using training samples, and the trained system circuit is verified by using verification samples. Finally, the trained system circuit is used as a speech recognition system to perform speech recognition of the test sample; The first linearization circuit and the second linearization circuit implement mapping of linearization layer parameters through a memristor cross array; The first Swish activation function circuit is used to implement: The signal conversion of V d is the input signal of the first Swish activation function circuit, Vsh is the output signal of the first Swish activation function circuit, β is a preset hyperparameter, V bias3 is the bias voltage; The kernel size of the first point-wise convolution module and the second point-wise convolution module is 1×1, and the kernel size of the deep convolution circuit is C1×C1, where C1 is the number of signal columns output by the multi-head attention module.

2. The speech recognition system based on the lightweight Transformer network according to claim 1, characterized in that: The preprocessing module preprocesses the acquired speech data using Mel-spectrogram coefficients and extracts 20 features from each frame of speech.

3. The speech recognition system based on the lightweight Transformer network according to claim 1 or 2, characterized in that: A signal selection switch is provided on each input end of the first layer regularization circuit in the first forward propagation module, so as to realize the selective input of each signal feature.

4. The speech recognition system based on a lightweight Transformer network according to claim 1, characterized in that: The first layer regularization circuit is based on Regularize each input signal feature, where V x Represents the input signal characteristics, R f represents the preset feedback resistor in the first layer regularization circuit, M1 is the first weight coefficient matrix, V bias1 It is a first bias voltage matrix, and the first weight coefficient matrix and the first bias voltage matrix are implemented by a first memristor cross array circuit.

5. The speech recognition system based on the lightweight Transformer network according to claim 1 or 4, characterized in that: The second-layer regularization circuit has the same structure as the first-layer regularization circuit, and in the finally trained system circuit, the first multiplication-accumulation circuit and the second multiplication-accumulation circuit are implemented using a memristor cross array.

6. The speech recognition system based on a lightweight Transformer network according to claim 5, characterized in that: The structure of the third-layer regularization circuit is the same as that of the first-layer regularization circuit. A third storage module is arranged at the output end of the gated linear unit. A batch regularization circuit, a second Swish activation function circuit and a fourth storage module are arranged at the output end of the deep convolution circuit. The second Swish activation function circuit has the same structure as the first Swish activation function circuit. The row voltages processed by the third-layer regularization circuit are sent to the deep convolution circuit through the third storage module. The output signal of the deep convolution circuit passes through the batch regularization circuit and the second Swish activation function circuit and is sent to the second point convolution module by the fourth storage module. Finally, the third addition circuit is used for residual connection. The weight of the convolution kernel in the deep convolution circuit is realized by a memristor cross array.

7. The speech recognition system based on a lightweight Transformer network according to claim 1, characterized in that: The SoftMax activation function circuit is provided with an exponential function circuit, two transistors, an operational amplifier and a voltage divider for realizing signal processing, where V in Vst is the multi-channel input signal of the SoftMax activation function circuit. m is the output signal of the SoftMax activation function circuit, is a preset proportional coefficient, and m is the total number of input signal paths.

8. A speech recognition method based on a lightweight Transformer network, characterized in that: Construct a speech recognition system as described in any one of claims 1 to 7, determine the values ​​of various parameters through pre-training, and then use a memristor cross array to implement corresponding parameter configuration, thereby forming a speech recognition model for speech recognition.

Images