Speech recognition method based on photon pulse neural network

Abstract

This invention discloses a speech recognition method based on a photonic spiking neural network, comprising: preprocessing the original speech dataset to obtain FBank feature values ​​corresponding to the training set and the test set, respectively; training a convolutional spiking neural network using the FBank feature values ​​corresponding to the training set; processing the FBank feature values ​​corresponding to the training set and the test set using the trained convolutional spiking neural network to obtain high-dimensional features corresponding to the training set and the test set; training the photonic spiking neural network using the high-dimensional features corresponding to the training set; and processing the high-dimensional features corresponding to the test set using the trained photonic spiking neural network to obtain the speech recognition result. This invention provides several speech recognition methods based on photonic spiking neural networks that not only have the advantages of low power consumption, high speed, and short latency, but also have high network complexity, enabling classification and recognition of large-scale standard datasets, and are suitable for large-scale networks.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech recognition, and in particular relates to a speech recognition method based on a photon pulse neural network. Background Art

[0002] The spiking neural network is a new generation of artificial neural network model inspired by biology. It mainly uses spiking neurons as basic units and has a strong biological basis. It is a process that is very close to the processing of brain neural signals. It is an effective tool for complex spatiotemporal information processing and has better biological feasibility than traditional ANN. At the same time, it is based on sparse pulse coding, making it hardware-friendly and energy-saving.

[0003] For speech recognition tasks, most of the current solutions are designed based on spiking neural networks. For example, Method 1 (Dennis J, Tran HD, Chng ES. Overlapping sound event recognition using local spectrogram features and the generalised hough transform [J]. Pattern Recognition Letters, 2013, 34 (9): 1085-1093) proposed to directly extract local features LSF (Local Spectrogram Feature) from the spectrogram, and convert the time and frequency two-dimensional information of LSF into the corresponding ignition information in the spiking neural network, and then connect it to the voting system for classification and discrimination; Method 2 (Dennis J, Yu Q, Tang H, et al. Temporal coding of local spectrogram features for robust sound recognition [C] / / 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013: 803-807) based on Method 1, first injects the extracted LSF features into the SOM network (Self Organising Network) Maps, self-organizing map network), the output of SOM is used as the input information of the spiking neural network, and then the Tempotron learning algorithm is used for learning and training.Another type of solution is to treat the spectrogram as a special kind of image and draw on the processing method of convolutional neural networks. For example, Method 3 (Dong M, Huang X, Xu B. Unsupervised speech recognition through spike-timing-dependent plasticity in a convolutional spiking neural network [J]. PloS one, 2018, 13 (11): e0204596) proposed using a convolutional spiking neural network as a front-end network to extract features and use softmax for classification and recognition; Method 4 (Zhang Z, Liu Q. Spike-Event-Driven Deep Spiking Neural Network With Temporal Encoding [J]. IEEE Signal Processing Letters, 2021, 28: 484-488) also proposed using a convolutional spiking neural network as a front-end to extract features, but it used three different sizes of convolution kernels for convolution, passed the extracted features into the spiking neural network, and used the STDBP algorithm for learning.

[0004] However, the above-mentioned speech recognition algorithms mainly use simple and ideal pulse neuron models in terms of model usage, and their pulses are represented by impulse functions. This representation method lacks the internal mechanism of actually generating pulses, and cannot effectively and accurately simulate the generation and transmission process of pulses in biological neural networks, as well as the absolute refractory period and relative refractory period of pulse neurons, which greatly reduces the complexity of the network and is not applicable to large-scale networks. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a speech recognition method based on photon pulse neural network. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0006] A speech recognition method based on photon pulse neural network, comprising:

[0007] Step 1: preprocess the original speech data set to obtain FBank feature values; wherein the original speech data set includes a training set and a test set, and the obtained FBank feature values ​​include the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set;

[0008] Step 2: Construct a convolutional spiking neural network, and train it using the FBank feature values ​​corresponding to the training set to obtain a trained convolutional spiking neural network;

[0009] Step 3: Use the trained convolutional spike neural network to process the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set to obtain the high-dimensional feature Feature corresponding to the training set and the high-dimensional feature Feature corresponding to the test set;

[0010] Step 4: construct a photon pulse neural network, and use the high-dimensional feature Feature corresponding to the training set for training to obtain a trained photon pulse neural network;

[0011] Step 5: Use the trained photon pulse neural network to process the high-dimensional feature Feature corresponding to the test set to obtain the speech recognition result.

[0012] In one embodiment of the present invention, in step 1, the original speech data set is preprocessed to obtain FBank feature values, including:

[0013] Extracting valid speech segments from the original speech data set using endpoint detection technology;

[0014] Feature extraction is performed on the valid speech segment to obtain FBank feature values.

[0015] In one embodiment of the present invention, step 2 comprises:

[0016] 21) constructing a convolutional spike neural network including a convolutional layer and a pooling layer, and temporally encoding the FBank feature values ​​corresponding to the training set to convert them into firing times of IF neurons in the convolutional layer;

[0017] 22) obtaining the firing information of each IF neuron in the convolutional layer;

[0018] 23) Setting inhibition strategies to determine the IF neurons that need to update weights;

[0019] 24) Based on the ignition information and the inhibition strategy, the weights of the IF neurons are updated using the STDP algorithm to train the network;

[0020] 25) Repeat steps 23)-24) until the preset maximum number of training times is reached to obtain a trained convolutional spike neural network.

[0021] In one embodiment of the present invention, step 23) comprises:

[0022] To set the ignition inhibition strategy:

[0023] For the same position in different feature maps, the IF neuron with the earliest firing time is retained; among them, if there are IF neurons with the same firing time, the IF neuron with the largest membrane voltage corresponding to the firing time is selected and retained;

[0024] To set the update suppression policy:

[0025] For the same feature map, if there are multiple IF neurons firing at adjacent positions, the earliest IF neuron firing is determined, and the weight of the IF neuron is updated, while the remaining adjacent IF neurons are not updated; if the firing time of multiple IF neurons at adjacent positions is the same, the IF neuron with the largest membrane voltage is selected for weight update, and the remaining adjacent IF neurons are not updated.

[0026] In one embodiment of the present invention, in step 24), the formula for updating the weight of the IF neuron using the STDP algorithm is as follows:

[0027]

[0028] Among them, t i represents the firing time of input neuron i, t j represents the firing time of output neuron j; w ij represents the connection weight between neuron i and neuron j; Δw ij Represents the weight update between neuron i and neuron j; α + Indicates the learning rate of the update expression when the firing time of neuron i is earlier than the firing time of neuron j; α - It indicates the learning rate of the update expression when the firing time of neuron i is earlier than the firing time of neuron j.

[0029] In one embodiment of the present invention, step 4 comprises:

[0030] 41) Construct a photon pulse neural network including an input layer, an output layer, and a discrimination layer;

[0031] 42) Initializing the photon pulse neural network, and performing time encoding on the high-dimensional feature Feature corresponding to the training set to convert it into the firing time of the VCSEL neuron in the output layer;

[0032] 43) Obtaining the firing time of the output layer VCSEL neuron;

[0033] 44) Setting a discrimination method, and updating the weights of the VCSEL neurons in the output layer based on the weights of time to train the network;

[0034] 45) Repeat steps 42)-44) until the preset maximum number of training times is reached to obtain a trained photon pulse neural network.

[0035] In one embodiment of the present invention, in step 44), setting the determination method includes:

[0036] Each output VCSEL neuron is set to correspond to a sample. When the input is a certain sample, the output VCSEL neuron corresponding to the sample is fired first, and the remaining VCSEL neurons are fired after the neuron or remain silent.

[0037] In one embodiment of the present invention, in step 44), updating the weight of the output layer VCSEL neuron based on the time weight includes:

[0038] Output VCSEL neuron n corresponding to the current input sample ref , if and only if the neuron is not firing, update the neuron n according to the following formula ref The corresponding weight size:

[0039] Δw=α1K(t max -t i -t delay ), t i <t max

[0040] Among them, Δw represents the weight increment, α1 represents the corresponding positive learning rate, and t max represents the time when the output power of the VCSEL neuron in the output layer reaches the maximum value within the simulation cutoff time, t i represents the firing time of the VCSEL neuron in the input layer, t delay Represents the delay of the VCSEL neuron, and the K function corresponds to the STDP curve.

[0041] In one embodiment of the present invention, in step 44), updating the weight of the output layer VCSEL neuron based on the time weight further includes:

[0042] For all output VCSEL neurons n except the one corresponding to the current input sample ref Other VCSEL neurons o :

[0043] If the ignition time t o Earlier than the output VCSEL neuron n corresponding to the current input sample ref Ignition time t ref , update neuron n according to the following formula ref The corresponding weight size:

[0044] Δw=α1K(tmax -t i -t delay ), t i <t max

[0045] If o Later than t ref , and the time difference between the two does not exceed the set time threshold, then update the neuron n according to the following formula o The corresponding weight size:

[0046]

[0047] Among them, α2 represents the negative constant learning rate, t thre Indicates the set time threshold;

[0048] If t o Later than t ref , and the time difference between the two is greater than the time threshold t thre , the weights are not updated.

[0049] In one embodiment of the present invention, step 5 comprises:

[0050] Inputting the high-dimensional feature Feature corresponding to the test set into the trained photon pulse neural network for processing;

[0051] Calculating the firing status of VCSEL neurons in the output layer of the photon pulse neural network;

[0052] The predicted label of the current input sample is determined according to the firing status of the VCSEL neurons in the output layer to complete speech recognition.

[0053] Beneficial effects of the present invention:

[0054] 1. The present invention provides multiple speech recognition methods based on photon pulse neural networks. First, the convolutional pulse neural network is used to extract the features of speech data to ensure that the feature values ​​extracted by the network are discrete; then the photon pulse neural network is used to perform encoding recognition processing to obtain speech recognition results. This method not only has the advantages of low power consumption, high speed, and short delay, but also has a high network complexity, can classify and recognize large-scale standard data sets, and is suitable for large-scale networks;

[0055] 2. The speech recognition method based on photon pulse neural network provided by the present invention is based on the actual laser neuron model when designing the photon pulse neural network structure, and takes into account various constraints in practice, so it is suitable for hardware reasoning.

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of a speech recognition method based on a photon pulse neural network provided by an embodiment of the present invention;

[0058] Figure 2 It is an algorithm framework diagram of a speech recognition method based on a photon pulse neural network provided by an embodiment of the present invention;

[0059] Figure 3 This is the result obtained by using the speech recognition method based on photon pulse neural network provided by the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0061] Embodiment 1

[0062] The present invention proposes a learning algorithm based on photon pulse neural network for speech recognition tasks. First, the convolutional pulse neural network is used to extract the features of the speech signal to ensure that the feature values ​​extracted by the network are discrete, which is conducive to the encoding of the photon pulse neural network; in the photon pulse neural network, a new judgment standard is proposed, and the weight update algorithm is designed based on this standard. The combination of the two realizes the application of photon pulse neural network in speech recognition, enabling it to classify and recognize large-scale standard data sets.

[0063] For details, please refer to Figure 1-2 , Figure 1 is a schematic diagram of a speech recognition method based on a photon pulse neural network provided by an embodiment of the present invention, Figure 2 It is an algorithm framework diagram of a speech recognition method based on a photon pulse neural network provided in an embodiment of the present invention.

[0064] First of all, it should be noted that the network architecture provided by the present invention requires the use of a GPU for network training and test results. Therefore, the host used must be equipped with an NVIDIA series GPU device. When specifically implemented, the original speech data set can be placed in the current working directory, and the data save path can be set in the Result folder.

[0065] Specifically, the implementation steps of the present invention include:

[0066] Step 1: Preprocess the original speech dataset to obtain FBank feature values.

[0067] Among them, the original speech data set includes a training set and a test set, and the obtained FBank feature values ​​include the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set.

[0068] In the present embodiment, with respect to the speech signal preprocessing part, key parameters are first set, and an endpoint detection technique is used to extract valid speech segments from the original speech data set.

[0069] For example, the classic double threshold detection method is used for the original speech dataset file in .wav format to extract the valid speech segments and remove the invalid speech segments.

[0070] Then, feature extraction is performed on the valid speech segment to obtain FBank feature values.

[0071] Specifically, the FBank feature value can be extracted by setting the number of frames Frames and the number of filters Mels in the Mel filter bank.

[0072] For the valid speech segment obtained, after pre-emphasis, framing, windowing, Fourier transform, and Mel filter bank filtering, the FBank eigenvalue matrix Arr_feature is obtained. Arr_feature is represented as a spectrogram and passed into the convolutional spike neural network as a special grayscale image. Frames and Mels are the corresponding number of frames and the number of filters in the Mel filter bank in the framing operation and the Mel filter bank filtering operation, respectively. The size of Arr_feature is {Frames, Mels}.

[0073] The above operation is performed on all speech data sets including the training set and the test set to obtain the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set.

[0074] Step 2: Construct a convolutional spiking neural network, and use the FBank feature values ​​corresponding to the training set for training to obtain a trained convolutional spiking neural network.

[0075] 21) constructing a convolutional spike neural network including a convolutional layer and a pooling layer; and time-encoding the FBank feature values ​​corresponding to the training set to convert them into the firing time of the IF neurons in the convolutional layer.

[0076] Specifically, the convolutional pulse neural network constructed in this embodiment includes a convolution layer and a pooling layer. The convolution layer is used to perform convolution processing on the FBank feature values, and the pooling layer is used to obtain higher-dimensional features Feature obtained after the convolution processing.

[0077] The training of convolutional spike neural networks is mainly aimed at determining the weights of each neuron in the convolutional layer.

[0078] Before training, it is necessary to time-code the Arr_feature corresponding to the training set obtained in step 1, and convert the Arr_feature inversely proportionally into the firing time of the IF neuron in the input layer.

[0079] 22) Obtain the firing information of each IF neuron in the convolutional layer.

[0080] In this embodiment, the firing information of each IF neuron in the convolutional layer is obtained, and the convolution characteristic settings including local connection and local weight sharing are included, as follows:

[0081] For the convolutional layer IF neurons, the calculation expression of the membrane voltage and the judgment of the firing time are as follows:

[0082]

[0083] t conv =t and v conv =V(t),when V(t)≥IF_threshold

[0084] Among them, s i and w i They represent the input layer IF neuron i at t i The pulse amplitude and corresponding connection weight generated at the moment, among which the pulse amplitude s generated by the input layer IF neuron i Set to 1.

[0085] According to the above formula, combined with the local connection and local weight sharing strategy, when the membrane voltage of the IF neuron in the convolutional layer exceeds the firing threshold IF_threshold, its firing time and membrane voltage value {t conv ,v conv}, and no pulses are generated after ignition.

[0086] Among them, local connections and local weight sharing are quite different from traditional convolutional neural networks. The specific contents are as follows:

[0087] Local connection: Combined with the harmonic correlation of the spectrogram, the convolution kernel window includes all frequency segments on the frequency axis (Mels), but retains the local connection characteristics on the time axis (Frames). The size of the convolution kernel window can be expressed as {Δt,f}. Therefore, unlike CNN, the feature map obtained after the spectrogram is convolved with the convolution kernel is one-dimensional, representing the local features extracted in different time periods.

[0088] Local weight sharing: Combined with local connection, because the feature map represents the local features in different time periods, it is considered to divide the feature map into multiple sections corresponding to different time periods. Different sections of the same feature map correspond to different convolution kernels. Neurons in the same section use the same convolution kernel to achieve local weight sharing.

[0089] 23) Set the inhibition strategy to determine the IF neurons that need to update their weights.

[0090] After obtaining the firing information of the IF neuron in the convolutional layer [t conv , v conv ], it is necessary to set the inhibition strategy to determine the neurons that need to update their weights, as follows:

[0091] First, set the ignition suppression strategy: for the same position of different feature maps, only one ignition neuron can exist, and the rest are suppressed. The specific ignition suppression strategy can be described as:

[0092] For the same position in different feature maps, the IF neuron with the earliest firing time is retained; among them, if there are IF neurons with the same firing time, the IF neuron with the largest membrane voltage corresponding to the firing time is selected and retained;

[0093] Then, set the update suppression policy:

[0094] For the same feature map, if there are multiple IF neurons firing at adjacent positions, the earliest IF neuron firing is determined, and the weight of the IF neuron is updated, while the remaining adjacent IF neurons are not updated; if the firing time of multiple IF neurons at adjacent positions is the same, the IF neuron with the largest membrane voltage is selected for weight update, and the remaining adjacent IF neurons are not updated.

[0095] 24) Based on the ignition information and the inhibition strategy, the weights of the IF neurons are updated using the STDP algorithm to train the network.

[0096] Specifically, after determining the neurons whose weights need to be updated, the STDP algorithm is used to update them, and the update expression is:

[0097]

[0098] Among them, t i represents the firing time of input neuron i, t j represents the firing time of output neuron j; w ij represents the connection weight between neuron i and neuron j; Δw ij Represents the weight update between neuron i and neuron j; α +Indicates the learning rate of the update expression when the firing time of neuron i is earlier than the firing time of neuron j; α - It indicates the learning rate of the update expression when the firing time of neuron i is earlier than the firing time of neuron j.

[0099] 25) Repeat steps 23)-24) until the preset maximum number of training times is reached to obtain a trained convolutional spike neural network.

[0100] At this point, the weights of the convolutional spike neural network have been determined.

[0101] Step 3: Use the trained convolutional spike neural network to process the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set to obtain the high-dimensional feature Feature corresponding to the training set and the high-dimensional feature Feature corresponding to the test set.

[0102] 31) First, initialization is performed, that is, the trained weights obtained in step 2 are read, and the FBank feature value data corresponding to the training set and the test set are respectively passed into the network.

[0103] 32) Then, a pooling layer is introduced into the convolutional spike neural network to extract higher dimensional features.

[0104] Specifically, the firing status of the neurons in the convolutional layer is obtained in the same way as in step 2 (here only firing suppression is considered, and no update operation is performed). A pooling layer is added after the convolutional layer, and a pooling operation (statistical operation) is performed on the neurons in the convolutional layer window. The details are as follows:

[0105] According to the internal segments of the feature map divided by local weight sharing in step 22), they are connected to the corresponding pooling layer IF neurons, and statistical pooling operations are performed to calculate the membrane voltage of the pooling layer IF neurons, where the pulse firing intensity of the convolution layer IF neurons is 1, and the weight connecting the convolution layer and the pooling layer is equal to 1. Therefore, calculating the membrane voltage of the pooling layer IF neurons is equivalent to calculating the number of convolution layer neurons fired in the corresponding time period.

[0106] Through the processing of convolutional layers and pooling layers, the high-dimensional feature Features corresponding to the training set and the high-dimensional feature Features corresponding to the test set are obtained.

[0107] Step 4: Construct a photon pulse neural network, and use the high-dimensional feature Feature corresponding to the training set for training to obtain a trained photon pulse neural network.

[0108] 41) Construct a photon pulse neural network including an input layer, an output layer, and a discrimination layer.

[0109] 42) Initializing the photon pulse neural network and time-encoding the high-dimensional feature Feature corresponding to the training set to convert it into the firing time of the output layer VCSEL neuron.

[0110] In this embodiment, the photon pulse neural network is initialized, including the scale of the input layer and the output layer, the initialization of the training weights, and the time encoding for higher dimensional features.

[0111] Specifically, the output layer size is determined according to the data set used, the input layer size is determined according to the dimension of the Feature extracted in step 3, and the network weights are initialized.

[0112] At the same time, due to the statistical pooling operation in step 32), the obtained feature values ​​are {f1,f2,…,f n}, so the corresponding eigenvalues ​​need to be converted into ignition times within the simulation cutoff time T.

[0113] 43) Obtain the firing time of the output layer VCSEL neuron.

[0114] Specifically, according to the VCSEL neuron calculation expression, the output power P of the output layer VCSEL neuron is obtained. out , according to P out Whether the threshold is exceeded determines whether the neuron fires.

[0115] However, regardless of whether the neuron fires or not, P can be obtained within the simulation deadline T. out The time t corresponding to the maximum value max , which is calculated as:

[0116] t max =Max(P out )

[0117] Therefore, the firing time t of the VCSEL neuron out It can be expressed as:

[0118]

[0119] 44) Setting a discrimination method, and updating the weights of the VCSEL neurons in the output layer based on the weights of time to train the network.

[0120] First, set the discrimination method.

[0121] Since the size of the output layer is the same as the sample type of the data set, each output VCSEL neuron is set to correspond to one sample. When the input is a certain sample, the output VCSEL neuron corresponding to the sample is fired first, and the remaining VCSEL neurons are fired after the neuron or remain silent.

[0122] Then, update the algorithm.

[0123] In this embodiment, the output VCSEL neuron n corresponding to the current input sample ref and neuron n ref Other VCSEL neurons o The updates are discussed separately.

[0124] Case 1: Output VCSEL neuron n corresponding to the current input sample ref , if and only if the neuron is not firing, its corresponding weight needs to be increased, then the neuron n is updated according to the following formula ref The corresponding weight size:

[0125] Δw=α1K(t max -t i -t delay ), t i <t max

[0126] Among them, Δw represents the weight increment, α1 represents the corresponding positive learning rate, and t max represents the time when the output power of the VCSEL neuron in the output layer reaches the maximum value within the simulation cutoff time, t i represents the firing time of the VCSEL neuron in the input layer, t delay Represents the delay of the VCSEL neuron, and the K function corresponds to the STDP curve.

[0127] Specifically, the K function represents mapping part of the STDP curve to [0,t thre ] time period, update the expression for Δw=K(Δt) to obtain higher resolution.

[0128] Case 2: For output VCSEL neurons n except the one corresponding to the current input sample ref Other VCSEL neurons o :

[0129] If the ignition time t o Earlier than the output VCSEL neuron n corresponding to the current input sample ref Ignition time t ref , we still need to increase the number of neurons n refThe corresponding weight size is still updated according to the method in case 1.

[0130] If t o Later than t ref , and the time difference between the two does not exceed the set time threshold, then update the neuron n according to the following formula o The corresponding weight size:

[0131]

[0132] Among them, α2 represents the negative constant learning rate, t thre Represents the set time threshold; it can be seen that its update amplitude decreases as the time difference increases.

[0133] If t o Later than t ref , and the time difference between the two is greater than the time threshold t thre , it is believed that this situation will not affect the judgment, and the weight will not be updated.

[0134] 45) Repeat steps 42)-44) until the preset maximum number of training times is reached to obtain a trained photon pulse neural network.

[0135] At this point, the weights of the photon pulse neural network have been determined.

[0136] Step 5: Use the trained photon pulse neural network to process the high-dimensional feature Feature corresponding to the test set to obtain the speech recognition result.

[0137] 51) Inputting the high-dimensional feature Feature corresponding to the test set into the trained photon pulse neural network for processing;

[0138] 52) Calculating the firing status of VCSEL neurons in the output layer of the photon pulse neural network;

[0139] 52) According to the firing status of the VCSEL neurons in the output layer, combined with the discrimination method proposed in step 44), the predicted label of the current input sample is determined to complete the speech recognition.

[0140] The present invention provides a plurality of speech recognition methods based on photon pulse neural networks. First, the convolutional pulse neural network is used to extract the features of speech data to ensure that the feature values ​​extracted by the network are discrete; then the photon pulse neural network is used to perform coding recognition processing to obtain speech recognition results. This method not only has the advantages of low power consumption, high speed, and short delay, but also has a high network complexity, and can classify and recognize large-scale standard data sets, and is suitable for large-scale networks. At the same time, since the present invention is based on the actual laser neuron model when designing the photon pulse neural network structure, it takes into account various constraints in practice and is therefore suitable for hardware reasoning.

[0141] Embodiment 2

[0142] The speech recognition method based on photon pulse neural network provided by the present invention is illustrated by a specific example below.

[0143] Step 1: Preprocess the original speech dataset

[0144] Assume that the original speech data set includes 1600 training samples and 400 test samples, and save the speech data processed by the double threshold detection method.

[0145] To obtain the Fbank eigenvalue, set the number of sub-frames Frames = 41, the number of filters in the Mel filter bank Mels = 40, and obtain the FBank eigenvalue matrix Arr_feature, sizeof(Arr_feature) = {40,41}. Transform Arr_feature into a one-dimensional vector and store it in a .csv file, and store the voice file name at the same time to obtain label information.

[0146] Step 2: Convolutional Spiking Neural Network Training

[0147] First, the Arr_feature corresponding to each voice data is time-encoded;

[0148] Then, train some network parameters:

[0149] Combined with the local connection characteristics, the convolution kernel size is set to {40×6}, 6 represents the local connection on the time axis, and the stride is 1, so the size of the feature map obtained is {36×1}.

[0150] Combined with the local weight sharing feature, the size of the segment in the feature map is set to Local_weight_sharing_size = 4. Neurons in the same segment share one Kernel, and neurons in different segments (in the same feature map) use different Kers.

[0151] Set the firing threshold of the convolutional layer IF neuron IF_threshold = 33, the number of feature maps Feature_map = 50, and calculate the membrane voltage of the convolutional layer IF neuron. Therefore, in the convolutional layer, matrices Arr_t and Arr_v with dimensions {36×50} are obtained, storing the firing information and the membrane voltage at the corresponding firing time.

[0152] According to the inhibition strategy, determine the neurons {t conv , v conv} that need weight update based on Arr_t and Arr_v, and update according to the STDP update expression.

[0153] Set the number of training times of the convolutional spiking neural network to 6, repeat the above steps before the end, and save the trained w weights.

[0154] Step 3: Use the trained convolutional neural network to obtain higher-dimensional features Feature

[0155] Among them, introduce a pooling layer to perform statistical pooling operations on the convolutional layer. The size of the pooling window is the same as Local_weight_sharing_size, which is 4. Therefore, the size of the pooling layer is {9×50}.

[0156] According to the trained weights, extract the FBank feature values of the training set and the test set to obtain high-dimensional feature values Feature, and convert {9×50} into a one-dimensional vector form {1×450} for storage.

[0157] Step 4: Train the photonic spiking neural network

[0158] According to the statistical pooling operation and Local_weight_sharing_size in the convolutional spiking neural network, it can be known that the value range of the Feature feature values is discrete and finite, and its possible values are {0, 1, 2, 3, 4}. Perform inverse proportional time encoding on it, and the converted firing time is {T, 9ns, 8ns, 7ns, 6ns}, where the simulation cut-off time T = 15ns. Therefore, calculate the firing time t < T of the output neuron, and set t = 1s if the neuron does not fire.

[0159] Set the input layer size of the photonic spiking neural network to 450, corresponding to the dimension of Feature; the output layer size to 10, corresponding to the number of sample types; initialize the size of the weights to {450x10};

[0160] Assume that the current input sample label = 0. After obtaining the firing situations {t0, t1, …, t9} of the 10 output neurons, it is necessary to update the neurons corresponding to {t0} and {t1, t2, …, t9} respectively.

[0161] For {t0}, its weight needs to be updated if and only if t0 = 1 (not fired). Regardless of whether the neuron is fired or not, the t ref If the ignition is not on, ref =t max .

[0162] For {t1, t2, …, t9}, an update is required if and only if the corresponding neuron fires, according to its firing time t o , t of {t0} ref and the effective time difference t thre =4ns, the weights of the corresponding neurons are updated.

[0163] Set the number of training times of the photon pulse neural network to 50 times, repeat the above steps before the end, and save the trained w weight.

[0164] Step 5: Use the trained network for speech recognition

[0165] Specifically, input the test set sample data, import the trained weights, and infer the sample categories. Figure 3 , Figure 3 This is the result obtained by using the speech recognition method based on photon pulse neural network provided by the present invention.

[0166] from Figure 3 It can be seen that the recognition accuracy of the method of the present invention can reach 93.3%, which is in line with expectations. However, a key factor restricting the recognition accuracy of the photon pulse neural network is that the VCSEL neurons in the output layer do not generate pulses, resulting in the inability to distinguish the sample type (the predicted type corresponds to Figure 3 The impact of this problem is far greater than the impact of misjudgment, which is why photon pulse neural networks are difficult to train.

[0167] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A speech recognition method based on photon pulse neural network, characterized in that: include: Step 1: preprocess the original speech data set to obtain FBank feature values; wherein the original speech data set includes a training set and a test set, and the obtained FBank feature values ​​include the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set; Step 2: Construct a convolutional spiking neural network, and train it using the FBank feature values ​​corresponding to the training set to obtain a trained convolutional spiking neural network; Step 3: Use the trained convolutional spike neural network to process the FBank feature values ​​corresponding to the training set and the FBank feature values ​​corresponding to the test set to obtain the high-dimensional feature Feature corresponding to the training set and the high-dimensional feature Feature corresponding to the test set; Step 4: construct a photon pulse neural network, and use the high-dimensional feature Feature corresponding to the training set for training to obtain a trained photon pulse neural network; Step 5: Use the trained photon pulse neural network to process the high-dimensional feature Feature corresponding to the test set to obtain a speech recognition result; Wherein, step 4 comprises: 41) Construct a photon pulse neural network including an input layer, an output layer, and a discrimination layer; 42) Initializing the photon pulse neural network, and performing time encoding on the high-dimensional feature Feature corresponding to the training set to convert it into the firing time of the VCSEL neuron in the output layer; 43) Obtaining the firing time of the output layer VCSEL neuron; 44) Setting a discrimination method, and updating the weights of the VCSEL neurons in the output layer based on the weights of time to train the network; 45) Repeat steps 42)-44) until the preset maximum number of training times is reached to obtain a trained photon pulse neural network.

2. The speech recognition method based on photon pulse neural network according to claim 1 is characterized in that: In step 1, the original speech data set is preprocessed to obtain FBank feature values, including: Extracting valid speech segments from the original speech data set using endpoint detection technology; Feature extraction is performed on the valid speech segment to obtain FBank feature values.

3. The speech recognition method based on photon pulse neural network according to claim 1 is characterized in that: Step 2 includes: 21) constructing a convolutional spike neural network including a convolutional layer and a pooling layer, and temporally encoding the FBank feature values ​​corresponding to the training set to convert them into firing times of IF neurons in the convolutional layer; 22) obtaining the firing information of each IF neuron in the convolutional layer; 23) Setting inhibition strategies to determine the IF neurons that need to update weights; 24) Based on the ignition information and the inhibition strategy, the weights of the IF neurons are updated using the STDP algorithm to train the network; 25) Repeat steps 23)-24) until the preset maximum number of training times is reached to obtain a trained convolutional spike neural network.

4. The speech recognition method based on photon pulse neural network according to claim 3 is characterized in that: Step 23) includes: To set the ignition inhibition strategy: For the same position in different feature maps, the IF neuron with the earliest firing time is retained; among them, if there are IF neurons with the same firing time, the IF neuron with the largest membrane voltage corresponding to the firing time is selected and retained; To set the update suppression policy: For the same feature map, if there are multiple IF neurons firing at adjacent positions, the earliest IF neuron firing is determined, and the weight of the IF neuron is updated, while the remaining adjacent IF neurons are not updated; if the firing time of multiple IF neurons at adjacent positions is the same, the IF neuron with the largest membrane voltage is selected for weight update, and the remaining adjacent IF neurons are not updated.

5. The method for speech recognition based on photon pulse neural network according to claim 3, characterized in that: In step 24), the formula for updating the weight of the IF neuron using the STDP algorithm is as follows: Among them, t i represents the firing time of input neuron i, t j represents the firing time of output neuron j; w ij represents the connection weight between neuron i and neuron j; Δw ij Represents the weight update between neuron i and neuron j; α + Indicates the learning rate of the update expression when the firing time of neuron i is earlier than the firing time of neuron j; α - Represents the learning rate of the updated expression when the firing time of neuron i is equal to or later than the firing time of neuron j.

6. The method for speech recognition based on photon pulse neural network according to claim 1, characterized in that: In step 44), setting the determination method includes: Each output VCSEL neuron is set to correspond to a sample. When the input is a certain sample, the output VCSEL neuron corresponding to the sample is fired first, and the remaining VCSEL neurons are fired after the neuron or remain silent.

7. The speech recognition method based on photon pulse neural network according to claim 6 is characterized in that: In step 44), updating the weight of the output layer VCSEL neuron based on the time weight includes: Output VCSEL neuron n corresponding to the current input sample ref , if and only if the neuron is not firing, update the neuron n according to the following formula ref The corresponding weight size: Δw=α1K(t max -t i -t delay ),t i <t max Among them, Δw represents the weight increment, α1 represents the corresponding positive learning rate, and t max represents the time when the output power of the VCSEL neuron in the output layer reaches the maximum value within the simulation cutoff time, t i represents the firing time of the VCSEL neuron in the input layer, t delay Represents the delay of the VCSEL neuron, and the K function corresponds to the STDP curve.

8. The speech recognition method based on photon pulse neural network according to claim 7 is characterized in that: In step 44), updating the weight of the output layer VCSEL neuron based on the time weight also includes: For all output VCSEL neurons n except the one corresponding to the current input sample ref Other VCSEL neurons o : If the ignition time t o Earlier than the output VCSEL neuron n corresponding to the current input sample ref Ignition time t ref , update neuron n according to the following formula ref The corresponding weight size: Δw=α1K(t max -t i -t delay ),t i <t max If o Later than t ref , and the time difference between the two does not exceed the set time threshold, then update the neuron n according to the following formula o The corresponding weight size: Among them, α2 represents the negative constant learning rate, t thre Indicates the set time threshold; If o Later than t ref , and the time difference between the two is greater than the time threshold t thre , the weights are not updated.

9. The method for speech recognition based on photon pulse neural network according to claim 1, characterized in that: Step 5 includes: Inputting the high-dimensional feature Feature corresponding to the test set into the trained photon pulse neural network for processing; Calculating the firing status of VCSEL neurons in the output layer of the photon pulse neural network; The predicted label of the current input sample is determined according to the firing status of the VCSEL neurons in the output layer to complete speech recognition.

Images