Keyword speech recognition method and device based on magnetic tunnel junction array

Abstract

The embodiment of the invention provides a keyword speech recognition method and device based on a magnetic tunnel junction array, and the method comprises the steps: training a neural network through employing a statistical perception training method, obtaining the weight data of the neural network, mapping the weight data into the high and low resistance state data of the magnetic tunnel junction array, and carrying out the recognition of the high and low resistance state data of the magnetic tunnel junction array. Selecting a magnetic tunnel junction array of a corresponding size according to an actual network structure of the neural network, and pre-writing magnetic tunnel junctions into corresponding resistance states; extracting Mel-frequency cepstrum coefficient (MFCC) features from the voice keyword signals; when voice keyword recognition is carried out, binarized voice MFCC features are continuously input into a magnetic tunnel junction array in a voltage amplitude form, a multiply-accumulate calculation result is obtained by measuring the output current of a target column of the magnetic tunnel junction array, and a hardware keyword recognition result is finally output through normalization and layer-by-layer reasoning.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a keyword speech recognition method and device based on a magnetic tunnel junction array. Background Art

[0002] With the continuous development of artificial intelligence (AI), image and speech recognition and processing technology, as an important foundation for human-computer interaction, is one of the important research directions in the field of artificial intelligence. Speech keyword recognition (KWS) is the core step in speech information processing. In edge computing scenarios with limited hardware resources, the accuracy of speech keyword recognition and hardware resource usage are the most concerned performance indicators. Researching speech keyword recognition methods with lower hardware resource overhead while ensuring accuracy is a key direction to improve speech recognition performance and expand speech recognition application scenarios.

[0003] Convolutional neural networks (CNNs) can automatically extract features through the convolution process and have excellent performance such as accuracy. They are widely used in the field of neural network target recognition. However, in order to meet complex task requirements, traditional convolutional neural networks have complex structures, large parameters, and require a large amount of computing and storage resources. They are difficult to apply to edge computing scenarios where hardware resources are extremely limited. The problems of storage wall and power consumption wall limit the large-scale integration of convolutional neural networks in computing hardware platforms based on traditional von Neumann architectures and their application in edge computing scenarios. As the key dimensions of semiconductor devices gradually enter the node below 3nm and artificial intelligence technology develops rapidly, in-memory computing architecture has attracted much attention as a new architecture that breaks through the limitations of von Neumann architecture. Magnetic random access memory has the advantages of simple reading and writing, low reading and writing power consumption, high stability, high storage density and data non-volatility. It is fully compatible with CMOS technology and is a strong competitor to mainstream in-memory computing devices. It has been widely studied in fields such as neuromorphic computing. At the same time, the deep separable neural network (DS-CNN) reduces the dimensionality of the three-dimensional convolution kernel of the traditional convolutional neural network step by step, which greatly simplifies the network structure and reduces the computing and storage overhead while basically keeping the accuracy of the neural network unchanged. It has great potential in hardware applications in the field of edge computing.

[0004] The prior art provides a binary speech keyword recognition neural network based on MFCC feature extraction, which can be deployed on FPGA for hardware acceleration. However, the network model is a traditional deep neural network (DNN), the number of weights and storage overhead are greater than the present invention, and FPGA cannot achieve a storage-computing integrated architecture compared to a magnetic tunnel junction array, and has obvious disadvantages in terms of energy efficiency, read-write speed, etc.

[0005] The prior art also provides a small-occupancy speech keyword recognition technology based on a convolutional recurrent neural network (CRNN), which can recognize and classify input speech keyword information. However, the network structure still requires the use of ~230K network weights, occupies a large amount of storage, and has not been implemented in hardware. Summary of the invention

[0006] The purpose of the present invention is to provide a keyword speech recognition method and device based on a magnetic tunnel junction array, aiming to solve the above-mentioned problems in the prior art.

[0007] The present invention provides a keyword speech recognition method based on a magnetic tunnel junction array, comprising:

[0008] Using a statistical perception training method to train a neural network, obtaining weight data of the neural network, mapping the weight data to high and low resistance state data of a magnetic tunnel junction array, selecting a magnetic tunnel junction array of a corresponding size according to an actual network structure of the neural network, and pre-writing the magnetic tunnel junction into a corresponding resistance state;

[0009] Extract Mel-frequency cepstral coefficient MFCC features from speech keyword signals;

[0010] When performing speech keyword recognition, the binary speech MFCC features are continuously input into the magnetic tunnel junction array in the form of voltage amplitude. The multiplication and accumulation calculation results are obtained by measuring the output current of the target column of the magnetic tunnel junction array. After normalization and layer-by-layer reasoning, the hardware keyword recognition results are finally output.

[0011] The present invention provides a keyword speech recognition device based on a magnetic tunnel junction array, comprising:

[0012] A weight mapping module is used to train the neural network using a statistical perception training method to obtain weight data of the neural network, map the weight data to high and low resistance state data of the magnetic tunnel junction array, select a magnetic tunnel junction array of corresponding size according to the actual network structure of the neural network, and pre-write the magnetic tunnel junction into the corresponding resistance state;

[0013] A speech feature extraction module is used to extract Mel-frequency cepstral coefficient (MFCC) features from speech keyword signals;

[0014] The speech keyword recognition module is used to continuously input the binary speech MFCC features into the magnetic tunnel junction array in the form of voltage amplitude when performing speech keyword recognition, obtain the multiplication and accumulation calculation results by measuring the output current of the target column of the magnetic tunnel junction array, and finally output the hardware keyword recognition results after normalization and layer-by-layer reasoning.

[0015] An embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-mentioned keyword speech recognition method based on a magnetic tunnel junction array.

[0016] An embodiment of the present invention further provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the above-mentioned keyword speech recognition method based on a magnetic tunnel junction array are implemented.

[0017] The embodiment of the present invention is compatible with the existing CMOS integration process based on the magnetic tunnel junction core structure, which is conducive to large-scale preparation and thus helps to realize the integrated application of in-memory computing and neuromorphic computing such as high-speed, high-energy efficiency, voice data processing in edge computing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a keyword speech recognition method based on a magnetic tunnel junction array according to an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of the training process of obtaining DS-CNN network weights by the statistical perception training method of an embodiment of the present invention and the performance indicators of DS-CNN in a host computer simulation experiment;

[0021] Figure 3 is a schematic diagram of a neural network structure of a DS-CNN according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a defect matrix generation method applied to statistical perception training according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of a neural network weight mapping rule according to an embodiment of the present invention;

[0024] Figure 6 Schematic diagram of a process of performing MFCC feature extraction on a speech keyword signal according to an embodiment of the present invention;

[0025] Figure 7is a schematic diagram of the principle of implementing a convolution operation based on a magnetic tunnel junction array according to an embodiment of the present invention;

[0026] Figure 8 Schematic diagram of hardware implementation of finally obtaining keyword recognition results in the speech keyword recognition stage of an embodiment of the present invention;

[0027] Fig. 9 is a schematic diagram of a speech keyword recognition process based on a magnetic tunnel junction array according to an embodiment of the present invention;

[0028] Fig.10 is a schematic diagram of a keyword speech recognition device based on a magnetic tunnel junction array according to an embodiment of the present invention;

[0029] Fig.11 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0031] Method Embodiment

[0032] According to an embodiment of the present invention, a keyword speech recognition method based on a magnetic tunnel junction array is provided. Figure 1 is a flow chart of a keyword speech recognition method based on a magnetic tunnel junction array according to an embodiment of the present invention, such as Figure 1 As shown, the keyword speech recognition method based on the magnetic tunnel junction array according to an embodiment of the present invention specifically includes:

[0033] Step S101, using a statistical perception training method to train a neural network, obtaining weight data of the neural network, mapping the weight data to high and low resistance state data of a magnetic tunnel junction array, selecting a magnetic tunnel junction array of a corresponding size according to the actual network structure of the neural network, and pre-writing the magnetic tunnel junction into a corresponding resistance state; specifically comprising:

[0034] Using a statistically aware training method, the weight data of a binary deep separable neural network that is robust to non-ideal hardware is obtained, and the trained weight matrix is ​​mapped to a magnetic tunnel junction array. A single magnetic tunnel structure is used to form a single weight, and different weights are mapped to different resistance state configurations of the magnetic tunnel junction. According to the structure of the neural network and the number of weights, a magnetic tunnel junction array of corresponding size is selected, and the magnetic tunnel junction is written into the corresponding resistance state through spin-transfer torque or spin-orbit torque, where the magnetic tunnel junction is in a parallel state and mapped to a weight of "1"; the magnetic tunnel junction is in an anti-parallel state and mapped to a weight of "0".

[0035] The magnetic tunnel junction array is at least one of the following: a passive STT-MRAM array, an active STT-MRAM array, an active SOT-MRAM array, and a three-dimensional stacked SOT-MRAM array.

[0036] Step S102, extracting Mel-frequency cepstral coefficient (MFCC) features from the speech keyword signal; specifically comprising:

[0037] The Mel-Freguency CeptraICoefficients (MFCC) features are extracted from the input speech keyword signal. The window length, window step size, and number of MFCC feature channels in the MFCC processing process are selected according to the network structure of the neural network and actual needs to adjust the size of the MFCC two-dimensional feature matrix that is finally input into the neural network.

[0038] Step S103, when performing speech keyword recognition, the binary speech MFCC features are continuously input into the magnetic tunnel junction array in the form of voltage amplitude, and the multiplication and accumulation calculation results are obtained by measuring the output current of the target column of the magnetic tunnel junction array, and finally the hardware keyword recognition results are output after normalization and layer-by-layer reasoning. Specifically including:

[0039] The speech MFCC feature matrix P is mapped to a read voltage matrix V, wherein each column of the read voltage matrix V is sequentially input into the column where the corresponding convolution kernel of the corresponding magnetic tunnel junction array is located;

[0040] According to Ohm's law and Kirchhoff's law, the convolution operation is equivalent to the reading operation of the magnetic tunnel junction array, the convolution result is summed up with the output current of each column of the magnetic tunnel junction array, the output current of each column of the magnetic tunnel junction under the corresponding read voltage combination is calculated, and the intermediate characteristic output matrix is ​​generated;

[0041] The intermediate feature output matrix is ​​binarized to form a new input matrix P1, which is mapped into a read voltage matrix V1. The read voltage input and current accumulation reading operations are performed again on the subsequent columns of the magnetic tunnel junction array to obtain the intermediate results P2 and V2 until the reasoning of each layer of the network is completed, and finally the speech keyword recognition results are output.

[0042] As can be seen from the above description, the neural network is first trained using a statistical perception training method to map the neural network weights to the high and low resistance configurations of the magnetic tunnel junction. Then, the Mel-frequency cepstral coefficient features are extracted from the speech keyword signal and input into the magnetic tunnel junction array in the form of voltage amplitude to map to the read voltage amplitude. According to the actual network structure, a magnetic tunnel junction array of the corresponding size is selected, and the magnetic tunnel junction is pre-written into the corresponding resistance state. When performing the formal speech keyword recognition task, the binary speech MFCC features are continuously input into the magnetic tunnel junction array in the form of voltage amplitude, and the multiplication and accumulation calculation results are obtained by measuring the output current of the target column of the magnetic tunnel junction array. After normalization and layer-by-layer reasoning, the hardware keyword recognition results are finally output. This technical solution is based on the magnetic tunnel junction core structure and is compatible with the existing CMOS integration process, which is conducive to large-scale preparation and thus helps to realize the integrated application of in-memory computing and neuromorphic computing such as voice data processing in high-speed, high-energy-efficiency, and edge computing scenarios.

[0043] The above technical solution of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings.

[0044] The technical solution of the embodiment of the present invention specifically includes: a weight mapping stage, a speech feature extraction stage, and a speech keyword recognition stage. In the weight mapping stage, the weight data of the neural network is obtained through a statistical perception training method and mapped to high and low resistance data of the magnetic tunnel junction array; in the speech feature extraction stage, the MFCC feature of the speech keyword signal is extracted to generate an input feature matrix; in the speech keyword recognition stage, the convolution operation of the weight matrix and the input speech feature data is replaced by the reading operation of the magnetic tunnel junction, and the sum of the read currents of each column of the magnetic tunnel junction array is the new intermediate feature value and recognition result obtained by the convolution.

[0045] First, in the weight mapping stage, a statistically aware training method is used to obtain binary deep separable neural network weights that are robust to non-ideal hardware, and then the trained weight matrix is ​​mapped to the magnetic tunnel junction array. A single magnetic tunnel structure is used to form a single-bit weight. Different weights are mapped to different resistance state configurations of the magnetic tunnel junction. Specifically, the magnetic tunnel junction is in a parallel state (P, low resistance state), mapped to a weight of "1"; the magnetic tunnel junction is in an antiparallel state (AP, high resistance state), mapped to a weight of "0". According to the structure of the neural network and the number of weights, a magnetic tunnel junction array of corresponding size is selected, and the magnetic tunnel junction is written into the corresponding resistance state through spin transfer torque or spin-orbit torque.

[0046] Secondly, in the speech feature extraction stage, MFCC feature extraction is performed on the input speech keyword signal. According to the network structure and actual needs, the window length, window step, number of MFCC feature channels and other parameters in the MFCC processing process are selected to adjust the size of the MFCC two-dimensional feature matrix that is finally input into the neural network.

[0047] In the speech keyword recognition stage, the speech MFCC feature matrix P is mapped to the read voltage matrix V, and each column of the read voltage matrix V is sequentially input into the column where the corresponding convolution kernel of the corresponding magnetic tunnel junction array is located. According to Ohm's law and Kirchhoff's law, the convolution operation, that is, the multiplication and addition operation, is equivalent to the reading operation of the magnetic tunnel junction array, and the convolution result corresponds to the sum of the output currents of each column of the magnetic tunnel junction array I out =∑ i,j V ij *G ij . Measure the output current of each column of the magnetic tunnel junction under the corresponding read voltage combination to generate an intermediate feature output matrix. Re-binarize the intermediate feature output matrix to form a new input matrix P1, which is mapped into a read voltage matrix V1. Perform the read voltage input and current accumulation reading operations on the subsequent columns of the magnetic tunnel junction array again to obtain intermediate results such as P2 and V2 until the reasoning of each layer of the network is completed and the array finally outputs the speech keyword recognition result.

[0048] In practical applications, the embodiments of the present invention can implement different network structures; use different speech feature extraction methods; use different magnetic tunnel junction arrays; and use different read voltage mapping ranges and step sizes.

[0049] Figure 2 The training process of obtaining the DS-CNN network weights by the statistical perception training method in the embodiment of the present invention and the performance indicators of DS-CNN in the host computer simulation experiment are demonstrated. The MFCC feature matrix training data after Wsat processing with different defect degrees is input, and the neural network is trained by back propagation using the gradient descent method to obtain network weights that are robust to hardware non-idealities. The array return result is the output value of the fully connected layer, which represents the return value of the target word and the non-target word respectively. After regularization, it is converted into the discrimination probability of the target word and the non-target word, and finally the recognition result is obtained. In this example, the same non-target keyword group is used for training different target keywords, and the recognition accuracy of different target keywords is shown in the figure.

[0050] Figure 3 The neural network structure of DS-CNN in the embodiment of the present invention is described. DS-CNN consists of several convolutional layers, depthwise separable convolutional layers, pooling layers, and fully connected layers. The number of network layers and their distribution, the size and step size of operators in each layer, and the network connection method are adjusted according to specific application requirements. In this specific embodiment, Figure 2 The network structure shown consists of a convolutional layer, a separable convolutional layer, a pooling layer and a fully connected layer. The illustrated structure requires a total of 790 binary weights.

[0051] Figure 4The defect matrix generation method used in statistical perception training in the embodiment of the present invention is described. When training the neural network weights, a random defect matrix within the upper limit of the defect number Wsat is generated, and the training data set is randomly defected to simulate non-ideal factors such as random single device failure in the hardware array. By generating multiple batches of defect matrices with different Wsat and training repeatedly, the robustness of the neural network weights to hardware non-ideal factors is improved.

[0052] Figure 5 The neural network weight mapping rules of the embodiment of the present invention are described. When the weights are hardwareized, a single magnetic tunnel junction is formed into a single weight, and different weights are mapped to different resistance state configurations of the magnetic tunnel junction. Specifically, the magnetic tunnel junction is in a parallel state (P, low resistance state), mapped to a weight of "1"; the magnetic tunnel junction is in an antiparallel state (AP, high resistance state), mapped to a weight of "0". In this example, Figure 2 Each weight of DS-CNN occupies 1 bit of storage resources, that is, a magnetic tunnel junction, and the weight of the complete neural network occupies less than 1Kbit. Figure 3 The method shown writes the weights of each layer of the DS-CNN in the embodiment into a 32×32 magnetic tunnel junction array.

[0053] Figure 6 The process of extracting MFCC features from speech keyword signals according to an embodiment of the present invention is described. The MFCC feature extraction of speech signals is divided into the steps of pre-emphasis, framing, windowing, frequency domain transformation, Mel cepstral filtering, taking log-DCT values, and seeking first-order and second-order differences. The engineering parameters in the MFC feature extraction process can be modified according to the characteristics of the speech keyword signal such as the duration and sampling rate. The MFCC feature extraction process finally outputs an N×D MFCC feature matrix, where N is the number of frames after the signal is framed, and D is the dimension of the Mel cepstral filter. In this specific embodiment, a speech keyword signal with a length of 1s and a sampling rate of 16000Hz is framed, the number of sampling points per frame is 1024, the step size is 512, and N=30; the Mel cepstral filter dimension D=40, and a 30×40 MFCC feature matrix is ​​obtained. In order to simplify the neural network task requirements and reduce the hardware resource overhead, the first 10 dimensional features are taken for input, that is, a 30×10 MFCC feature matrix.

[0054] Figure 7 The principle of implementing convolution operation based on magnetic tunnel junction array in the embodiment of the present invention is described. In this specific embodiment, taking the first layer of traditional convolution as an example, its convolution kernel is a 3×3×10 three-dimensional operator, and a single dimension is a 3×3 convolution operator. It can be represented by the following variables:

[0055]

[0056] According to actual needs, the size of each layer of convolution kernel and the specific weight value can be adjusted, so the scope of this patent also includes different convolution kernels and neural network weight values.

[0057] According to the neural network weight mapping rule of the present invention, the 3×3 single-dimensional convolution weights are mapped to the corresponding resistance states of the magnetic tunnel junction array. First, the weights are reduced to 1 dimension:

[0058]

[0059] The conductivity matrix corresponding to the magnetic tunnel junction is:

[0060]

[0061] As mentioned above, the weight values ​​w in the weight matrix S ij The 1 corresponds to the conductivity matrix G P (low impedance state), w ij The 0 of the derivative matrix G AP (High impedance state). Figure 2 As shown, the voice keyword MFCC feature matrix P is mapped to the read voltage matrix V. According to Ohm's law and Kirchhoff's law, the convolution operation of the MFCC feature matrix P and the one-dimensional convolution kernel is directly equivalent to the reading operation of the magnetic tunnel junction array under the corresponding read voltage, I out =∑ i,j V ij *G ij , input the voltage of each column of the read voltage matrix in turn, read the magnetic tunnel junction column current corresponding to the current convolution kernel, and get the result of this convolution as the intermediate input feature of the next convolution layer.

[0062] Figure 8The hardware implementation diagram of the output of intermediate features by the magnetic tunnel junction array in the speech keyword recognition stage of the embodiment of the present invention, layer-by-layer reasoning, and finally obtaining the keyword recognition result is described. The input MFCC feature matrix P is mapped to a read voltage matrix V, and the intermediate result is output after the convolution operation of the corresponding layer operator of the neural network on the magnetic tunnel junction array. In this specific embodiment, the FPGA controls the peripheral circuit digital-to-analog converter, analog-to-digital converter and other components to perform read and write operations on the magnetic tunnel junction. The 30×10 MFCC feature matrix is ​​input column by column in the form of a read voltage matrix into the magnetic tunnel junction array after weight mapping. In the first layer of traditional convolution process, a column of read voltage is applied multiple times, and the accumulated results of each column current of the magnetic tunnel junction mapped with the first layer of convolution kernel are read respectively to obtain the intermediate features; repeat this operation for each column of the voltage matrix to obtain the intermediate feature matrix P1, map the intermediate feature matrix P1 to a new read voltage matrix V1, and re-input it into the magnetic tunnel junction array, read the column current corresponding to the second layer of depth-separable convolution kernel for each column voltage, and obtain the intermediate feature matrix P2. And so on, after the pooling layer and the fully connected layer respectively, the output of the fully connected layer is the final result of speech keyword recognition.

[0063] Fig. 9 The flow chart of speech keyword recognition based on magnetic tunnel junction array in the embodiment of the present invention is described. The whole process is divided into weight mapping stage, speech feature extraction stage and speech keyword recognition stage. In the weight mapping stage, according to Figure 4 Generate a defect matrix for statistical perception training as shown, and then follow Figure 5 The weight mapping principle shown in the figure maps the DS-CNN weights to the magnetic tunnel junction array and writes the corresponding resistance state; in the speech feature extraction stage, according to Figure 6 The MFCC speech feature extraction process shown in the figure extracts the input speech keyword signal into a 30×10 MFCC feature matrix; in the speech keyword recognition stage, according to Figure 7 The convolution principle shown in FIG. 1 inputs the MFCC feature matrix into the magnetic tunnel junction array in the form of read voltage mapping; according to Figure 8 The hardware structure shown performs multi-layer neural network convolution operations and ultimately obtains speech keyword recognition results.

[0064] The beneficial effects of the embodiments of the present invention are as follows:

[0065] In terms of hardware, the convolution operation of the neural network is hardware-based through the current reading operation of the magnetic tunnel junction array. A convolution operation can be completed in a single clock cycle, which improves the parallelism and convolution efficiency. The single recognition delay is <…μs and the energy consumption is <…J.

[0066] In terms of software, the binary DS-CNN neural network model is used to optimize the number of network weights to <1k, and the network weight storage occupies <1k bits. This greatly reduces the hardware resource overhead of speech keyword recognition under the benchmark of recognition accuracy >90%;

[0067] During the network training process, by applying statistically aware training methods, the robustness of the network weights to array physical defects and the versatility of the weights to different defective devices are enhanced from the software level.

[0068] Device Example 1

[0069] According to an embodiment of the present invention, a keyword speech recognition device based on a magnetic tunnel junction array is provided. Fig.10 is a schematic diagram of a keyword speech recognition device based on a magnetic tunnel junction array according to an embodiment of the present invention, such as Fig.10 As shown, the keyword speech recognition device based on the magnetic tunnel junction array according to the embodiment of the present invention specifically includes:

[0070] The weight mapping module 100 is used to train the neural network using a statistical perception training method to obtain weight data of the neural network, map the weight data to high and low resistance state data of the magnetic tunnel junction array, select a magnetic tunnel junction array of corresponding size according to the actual network structure of the neural network, and pre-write the magnetic tunnel junction into the corresponding resistance state; specifically used for:

[0071] Using the statistical perception training method, the weight data of the binary deep separable neural network that is robust to non-ideal hardware is obtained, and the trained weight matrix is ​​mapped to the magnetic tunnel junction array. A single magnetic tunnel structure is used to form a single weight. Different weights are mapped to different resistance state configurations of the magnetic tunnel junction. According to the structure of the neural network and the number of weights, a magnetic tunnel junction array of corresponding size is selected, and the magnetic tunnel junction is written into the corresponding resistance state through spin transfer torque or spin-orbit torque. The magnetic tunnel junction is in a parallel state and mapped to a weight of "1"; the magnetic tunnel junction is in an anti-parallel state and mapped to a weight of "0".

[0072] The speech feature extraction module 102 is used to extract Mel-frequency cepstral coefficient (MFCC) features from the speech keyword signal; specifically used for:

[0073] Perform MFCC feature extraction on the input speech keyword signal, select the window length, window step length, and number of MFCC feature channels in the MFCC processing process according to the network structure of the neural network and actual needs, so as to adjust the size of the MFCC two-dimensional feature matrix that is finally input into the neural network;

[0074] The speech keyword recognition module 104 is used to continuously input the binary speech MFCC features in the form of voltage amplitude into the magnetic tunnel junction array when performing speech keyword recognition, obtain the multiplication and accumulation calculation results by measuring the output current of the target column of the magnetic tunnel junction array, and finally output the hardware keyword recognition results after normalization and layer-by-layer reasoning. Specifically used for:

[0075] The speech MFCC feature matrix P is mapped to a read voltage matrix V, wherein each column of the read voltage matrix V is sequentially input into the column where the corresponding convolution kernel of the corresponding magnetic tunnel junction array is located;

[0076] According to Ohm's law and Kirchhoff's law, the convolution operation is equivalent to the reading operation of the magnetic tunnel junction array, the convolution result is summed up with the output current of each column of the magnetic tunnel junction array, the output current of each column of the magnetic tunnel junction under the corresponding read voltage combination is calculated, and the intermediate characteristic output matrix is ​​generated;

[0077] The intermediate feature output matrix is ​​binarized to form a new input matrix P1, which is mapped into a read voltage matrix V1. The read voltage input and current accumulation reading operations are performed again on the subsequent columns of the magnetic tunnel junction array to obtain the intermediate results P2 and V2 until the reasoning of each layer of the network is completed, and finally the speech keyword recognition results are output.

[0078] The magnetic tunnel junction array is at least one of the following: a passive STT-MRAM array, an active STT-MRAM array, an active SOT-MRAM array, and a three-dimensional stacked SOT-MRAM array.

[0079] The embodiment of the present invention is a device embodiment corresponding to the above method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0080] Device Example 2

[0081] An embodiment of the present invention provides an electronic device, such as Fig.11 As shown, it includes: a memory 110, a processor 112, and a computer program stored in the memory 110 and executable on the processor 112, and when the computer program is executed by the processor 112, the steps described in the method embodiment are implemented.

[0082] Device Example 3

[0083] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by the processor 112, the steps described in the method embodiment are implemented.

[0084] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A keyword speech recognition method based on a magnetic tunnel junction array, characterized in that: include: Using a statistical perception training method to train a neural network, obtaining weight data of the neural network, mapping the weight data to high and low resistance state data of a magnetic tunnel junction array, selecting a magnetic tunnel junction array of a corresponding size according to an actual network structure of the neural network, and pre-writing the magnetic tunnel junction into a corresponding resistance state; Extract Mel-frequency cepstral coefficient MFCC features from speech keyword signals; When performing speech keyword recognition, the binary speech MFCC features are continuously input into the magnetic tunnel junction array in the form of voltage amplitude. The multiplication and accumulation calculation results are obtained by measuring the output current of the target column of the magnetic tunnel junction array. After normalization and layer-by-layer reasoning, the hardware keyword recognition results are finally output.

2. The method according to claim 1, characterized in that Using a statistical perception training method to train a neural network, obtaining weight data of the neural network, mapping the weight data to high and low resistance state data of a magnetic tunnel junction array, selecting a magnetic tunnel junction array of a corresponding size according to the actual network structure of the neural network, and pre-writing the magnetic tunnel junction into a corresponding resistance state specifically include: Using a statistically aware training method, the weight data of a binary deep separable neural network that is robust to non-ideal hardware is obtained. The trained weight matrix is ​​mapped to a magnetic tunnel junction array, and a single magnetic tunnel structure is used to form a single weight. Different weights are mapped to different resistance state configurations of the magnetic tunnel junction. According to the structure of the neural network and the number of weights, a magnetic tunnel junction array of corresponding size is selected, and the magnetic tunnel junction is written into the corresponding resistance state through spin-transfer torque or spin-orbit torque, where the magnetic tunnel junction is in a parallel state and mapped to a weight of "1"; the magnetic tunnel junction is in an anti-parallel state and mapped to a weight of "0".

3. The method according to claim 1, characterized in that Extracting Mel cepstral coefficient features from the speech keyword signal and inputting them into the magnetic tunnel junction array in the form of voltage amplitude to map them into read voltage amplitude specifically includes: MFCC feature extraction is performed on the input speech keyword signal. The window length, window step size, and number of MFCC feature channels in the MFCC processing process are selected according to the network structure of the neural network and actual needs to adjust the size of the MFCC two-dimensional feature matrix that is finally input into the neural network.

4. The method according to claim 1, characterized in that: When performing speech keyword recognition, the binary speech MFCC features are continuously input into the magnetic tunnel junction array in the form of voltage amplitude, and the multiplication and accumulation calculation results are obtained by measuring the output current of the target column of the magnetic tunnel junction array. After normalization and layer-by-layer reasoning, the final output hardware keyword recognition results include: The speech MFCC feature matrix P is mapped to a read voltage matrix V, wherein each column of the read voltage matrix V is sequentially input into the column where the corresponding convolution kernel of the corresponding magnetic tunnel junction array is located; According to Ohm's law and Kirchhoff's law, the convolution operation is equivalent to the reading operation of the magnetic tunnel junction array, the convolution result is summed up with the output current of each column of the magnetic tunnel junction array, the output current of each column of the magnetic tunnel junction under the corresponding read voltage combination is calculated, and the intermediate characteristic output matrix is ​​generated; The intermediate feature output matrix is ​​binarized to form a new input matrix P1, which is mapped into a read voltage matrix V1. The read voltage input and current accumulation reading operations are performed again on the subsequent columns of the magnetic tunnel junction array to obtain the intermediate results P2 and V2 until the reasoning of each layer of the network is completed, and finally the speech keyword recognition results are output.

5. The method according to claim 1, characterized in that The magnetic tunnel junction array is at least one of the following: a passive STT-MRAM array, an active STT-MRAM array, an active SOT-MRAM array, and a three-dimensional stacked SOT-MRAM array.

6. A keyword speech recognition device based on a magnetic tunnel junction array, characterized in that: include: A weight mapping module is used to train the neural network using a statistical perception training method to obtain weight data of the neural network, map the weight data to high and low resistance state data of the magnetic tunnel junction array, select a magnetic tunnel junction array of corresponding size according to the actual network structure of the neural network, and pre-write the magnetic tunnel junction into the corresponding resistance state; A speech feature extraction module is used to extract Mel-frequency cepstral coefficient (MFCC) features from speech keyword signals; The speech keyword recognition module is used to continuously input the binary speech MFCC features into the magnetic tunnel junction array in the form of voltage amplitude when performing speech keyword recognition, obtain the multiplication and accumulation calculation results by measuring the output current of the target column of the magnetic tunnel junction array, and finally output the hardware keyword recognition results after normalization and layer-by-layer reasoning.

7. The device according to claim 6, characterized in that The weight mapping module is specifically used for: Using the statistical perception training method, the weight data of the binary deep separable neural network that is robust to non-ideal hardware is obtained, and the trained weight matrix is ​​mapped to the magnetic tunnel junction array. A single magnetic tunnel structure is used to form a single weight. Different weights are mapped to different resistance state configurations of the magnetic tunnel junction. According to the structure of the neural network and the number of weights, a magnetic tunnel junction array of corresponding size is selected, and the magnetic tunnel junction is written into the corresponding resistance state through spin transfer torque or spin-orbit torque. Among them, the magnetic tunnel junction is in a parallel state and mapped to a weight of "1"; the magnetic tunnel junction is in an anti-parallel state and mapped to a weight of "0". The speech feature extraction module is specifically used for: Perform MFCC feature extraction on the input speech keyword signal, select the window length, window step length, and number of MFCC feature channels in the MFCC processing process according to the network structure of the neural network and actual needs, so as to adjust the size of the MFCC two-dimensional feature matrix that is finally input into the neural network; The speech keyword recognition module is specifically used for: The speech MFCC feature matrix P is mapped to a read voltage matrix V, wherein each column of the read voltage matrix V is sequentially input into the column where the corresponding convolution kernel of the corresponding magnetic tunnel junction array is located; According to Ohm's law and Kirchhoff's law, the convolution operation is equivalent to the reading operation of the magnetic tunnel junction array, the convolution result is summed up with the output current of each column of the magnetic tunnel junction array, the output current of each column of the magnetic tunnel junction under the corresponding read voltage combination is calculated, and the intermediate characteristic output matrix is ​​generated; The intermediate feature output matrix is ​​binarized to form a new input matrix P1, which is mapped into a read voltage matrix V1. The read voltage input and current accumulation reading operations are performed again on the subsequent columns of the magnetic tunnel junction array to obtain the intermediate results P2 and V2 until the reasoning of each layer of the network is completed, and finally the speech keyword recognition results are output.

8. The device according to claim 6, characterized in that The magnetic tunnel junction array is at least one of the following: a passive STT-MRAM array, an active STT-MRAM array, an active SOT-MRAM array, and a three-dimensional stacked SOT-MRAM array.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the keyword speech recognition method based on a magnetic tunnel junction array as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the keyword speech recognition method based on a magnetic tunnel junction array as described in any one of claims 1 to 4 are implemented.

Images