An end-to-end speech recognition method and system based on spiking neural network

Abstract

A method for end-to-end speech recognition based on spiking neural networks (SNNs) involves three steps: preprocessing speech data, training a convolutional spiking neural network (CNN) with self-attention, and then inputting the preprocessed speech data into the trained CNN for recognition. The spiking neuron model uses a multi-time-step LIF neuron model and proposes a convolutional spiking neural network architecture with self-attention. This method leverages biological properties to effectively reduce the computational burden of the neural network and achieve more accurate text recognition in large-scale speech datasets.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and speech recognition technology, and in particular relates to an end-to-end speech recognition method and system based on a spiking neural network. Background Art

[0002] Mobile and portable computing platforms have become essential in daily work, learning, and research, and simpler, more effective interaction methods are becoming widely used in real life. Speech recognition technology converts human speech into computer-readable text, a process known as speech-to-text. Traditional speech recognition first converts speech content into phonemes using an acoustic model, and then converts it into text using a language model. End-to-end speech recognition frameworks can directly convert speech into text and are easily trainable. Most electronic products are equipped with speech recognition interaction features, allowing people to directly speak what they want to do or what they want to search for, prompting the computer to execute the corresponding program. This is more user-friendly, efficient, and convenient than interaction methods such as keyboards, mice, and touchscreens. With smart home appliances such as smart speakers, smart TVs, and smart refrigerators appearing in more and more homes, speech recognition is expected to be one of the most promising human-computer interaction interfaces of the future. However, speech recognition technology still has shortcomings, especially in scenarios with limited computing resources and large vocabulary.

[0003] The Transformer architecture, which takes into account biological attention mechanisms, was first applied in natural language processing, significantly improving the performance of neural network models at the time. Subsequently, researchers in other fields also began using the Transformer model for research and proposed a series of corresponding optimization methods. Taking advantage of the Transformer's global modeling properties, international researchers combined the local modeling properties of convolution with the Transformer to propose the Conformer architecture, which simultaneously considers both global and local features. This architecture has been applied to speech recognition, achieving high recognition accuracy, especially with large datasets. However, the Conformer model has too many layers and a large number of model parameters, making training costly and difficult.

[0004] Spiking neural networks, hailed as the third generation of neural networks, are designed with a focus on biomimetic principles, demonstrating stronger biological similarity than traditional artificial neural networks. They are commonly used in brain-inspired simulation research. Due to their energy-saving and high-efficiency advantages, a growing number of researchers are exploring replacing traditional artificial neural networks with spiking neural networks for certain pattern recognition tasks. However, existing spiking neural networks are still inferior to artificial neural networks in areas such as image and speech recognition. In particular, in the field of speech recognition, spiking neural networks have only been applied to small datasets, and their effectiveness on large-scale speech datasets needs to be improved. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an end-to-end speech recognition method and system based on a pulse neural network, which fully utilizes the high efficiency and energy-saving characteristics of the pulse neural network, solves the application of the pulse neural network in large-scale speech recognition scenarios, and greatly improves the speed and accuracy of end-to-end speech recognition.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] An end-to-end speech recognition method based on a spiking neural network comprises the following steps:

[0008] Step 1: Speech data preprocessing, including feature extraction and feature enhancement of the original speech data, upsampling, and finally relative position encoding;

[0009] Step 2: Use the dataset consisting of preprocessed speech data to train a convolutional spiking neural network with self-attention;

[0010] Step 3: After preprocessing the speech data to be recognized, input it into the trained pulse neural network for recognition to achieve speech-to-text conversion.

[0011] In one embodiment, the process of step 1, pre-processing the voice data is as follows:

[0012] Step 1.1: Speech feature extraction: First, resample the original speech data according to a uniform sampling rate, then create a spectrogram from the audio signal and convert it into a mel-domain spectrum. Finally, normalize the feature strengths according to their mean and standard deviation to generate a normalized speech feature sequence.

[0013] Step 1.2: Speech feature enhancement: first perform frequency domain masking on the speech feature sequence processed in step 1.1, and then perform time domain masking;

[0014] Step 1.3: 2D convolution upsampling: squeeze the speech feature sequence processed in step 1.2 out of the speech channel dimension, then pass it through two layers of 2D convolutional spike neurons with multiple time steps. After that, the channel dimension and feature dimension of the output feature sequence are multiplied to form a new feature dimension.

[0015] Step 1.4: Relative position encoding: generate a new unique position for each index token in the feature sequence processed in step 1.3 through relative position encoding, and then sum it with the feature sequence before encoding.

[0016] In one embodiment, in step 2, the convolutional pulse neural network with self-attention includes a convolutional pulse encoder with self-attention and a linear pulse decoder;

[0017] The convolutional pulse encoder with self-attention includes a linear pulse neuron layer, a dropout layer and a self-attention convolution block composed of m pulse neurons connected in sequence, wherein the activation function is replaced by a LIF pulse neuron for excitation in the self-attention convolution block, and residual calculation is performed between the input and output of each module of the self-attention convolution block;

[0018] The linear pulse decoder is a layer of linear pulse neurons, which takes the output data of the convolutional pulse encoder with self-attention as input.

[0019] In one embodiment, the modules in the self-attention convolution block include a first half-step feedforward module, a multi-head self-attention module, a convolution module, a second half-step feedforward module, and a normalization module connected in sequence;

[0020] The first half-step feedforward module and the second half-step feedforward module have the same structure, and both include a first normalization module, a first linear pulse neuron layer A, a first LIF pulse neuron layer, a first Dropout layer A, a first linear pulse neuron layer B, and a first Dropout layer B connected in sequence, wherein the output data of the first Dropout layer B is divided by 2 and the residual is calculated with the input data of the first normalization module as the final output of the first half-step feedforward module;

[0021] The multi-head self-attention module includes a second normalization module, h self-attention layers, and a second Dropout layer connected in sequence. The second Dropout layer sums the input data of the multi-head self-attention module through the results of each self-attention layer, and the summation result is calculated with the input data of the second normalization module as the final output of the multi-head self-attention module;

[0022] The convolution module includes a third normalization module, a point-by-point convolution layer A, a second LIF pulse neuron layer A, a one-dimensional depth convolution layer, a batch normalization layer, a second linear pulse neuron layer, a second LIF pulse neuron layer B, a point-by-point convolution layer B and a third Dropout layer connected in sequence.

[0023] The one-dimensional deep convolution layer and the batch normalization layer multiply the data after the LIF pulse neuron is excited with the data before the excitation, and the output data of the third Dropout layer is subjected to residual calculation with the input data of the third normalization module as the final output of the convolution module.

[0024] In one embodiment, the first LIF spiking neuron layer, the second LIF spiking neuron layer A, and the second LIF spiking neuron layer B all use multi-step LIF spiking neurons.

[0025] In one embodiment, the training process in step 2 is as follows:

[0026] Step 2.1: Set network training parameters;

[0027] Step 2.2: Use the pseudo BP method to perform back propagation of the network. Use the sigmoid gradient function to replace the non-differentiable part of the spiking neuron in the BP process. When the membrane potential of the spiking neuron exceeds the threshold, it outputs 1 to the downstream neuron and then resets it to the resting potential; otherwise, it outputs 0 to the downstream neuron.

[0028] Step 2.3: Repeat forward propagation, loss function calculation, backward propagation, and network weight update until the loss function on the validation dataset no longer decreases after n consecutive training rounds, and the training ends.

[0029] In one embodiment, in step 3, the output of the linear pulse decoder is normalized and then logarithm is taken to speed up the decoding speed and improve the decoding stability; a greedy search strategy is used to convert the output content into text, and each decoding selects the predicted label with the highest current probability, and the continuous decoding output is the text obtained by speech recognition.

[0030] Another aspect of the present invention provides a computer program product, comprising a computer program, which implements the steps of the method when executed by a processing unit.

[0031] Another aspect of the present invention provides a computing device, including a storage unit and a processing unit, wherein the storage unit stores a computer program, and the processing unit implements the steps of the method when executing the computer program.

[0032] Another aspect of the present invention further provides an end-to-end speech recognition system based on a spiking neural network, comprising a language acquisition device, a display device, and the computing device described above;

[0033] The language acquisition device acquires voice data to be recognized;

[0034] The computing device, wherein the processing unit includes a preprocessing module for performing preprocessing, a training module for performing network training, and an operation module for performing conversion; the preprocessing module takes the speech data to be recognized as input, and the operation module takes the preprocessed speech data to be recognized as input and outputs the obtained text;

[0035] The display device displays the content of the text.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. A set of network construction and training models based on bionic design can be effectively applied to end-to-end speech recognition problems.

[0038] 2. Based on the deep network structure, the self-attention modeling of global information and the convolution modeling of local information are retained, achieving the advantages of both global and local considerations, and the network's performance on large-scale speech datasets has been improved.

[0039] 3. Taking advantage of the high efficiency and energy-saving characteristics of pulse neurons, the network parameters and training costs are smaller than those of traditional deep self-attention convolutional deep neural models.

[0040] 4. Since the pulse neurons are sparsely excited and the amount of calculation is small, the overall calculation speed is accelerated, and the text results can be converted faster and more accurately in end-to-end speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the overall flow chart of the speech recognition method based on pulse neural network proposed in the present invention.

[0042] Figure 2 This is the structural diagram of the convolutional pulse neural network with self-attention proposed in the present invention.

[0043] Figure 3 This is the structure diagram of the feedforward module.

[0044] Figure 4 This is the structure diagram of the multi-head self-attention module.

[0045] Figure 5 It is a convolution module structure. DETAILED DESCRIPTION

[0046] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0047] A method for end-to-end speech recognition based on spiking neural networks (SNNs) involves three steps: preprocessing speech data, training a convolutional spiking neural network (CNN) with self-attention, and then inputting the preprocessed speech data into the trained CNN for recognition. The spiking neuron model uses a multi-time-step LIF neuron model and proposes a convolutional spiking neural network architecture with self-attention. This method leverages biological properties to effectively reduce the computational burden of the neural network and achieve more accurate text recognition in large-scale speech datasets.

[0048] like Figure 1 As shown, the present invention provides an end-to-end speech recognition method based on a pulse neural network, comprising the following steps:

[0049] Step 1: Speech data preprocessing, including feature extraction and feature enhancement of the original speech data, upsampling, and finally relative position encoding.

[0050] Step 2: Use the data set consisting of the preprocessed speech data to train a convolutional pulse neural network with self-attention. The convolutional pulse neural network mainly includes a convolutional pulse encoder with self-attention and a linear pulse decoder.

[0051] Among them, the convolutional pulse encoder with self-attention includes a linear pulse neuron layer, a Dropout layer and m self-attention convolution blocks composed of pulse neurons connected in sequence. In the self-attention convolution block, LIF pulse neurons are used to replace the activation function for excitation, and the input and output between each module of the self-attention convolution block are calculated as residuals.

[0052] The output data of the convolutional pulse encoder with self-attention is the input of the linear pulse decoder. In the present invention, the linear pulse decoder is a layer of linear pulse neurons.

[0053] Step 3: After preprocessing the speech data to be recognized, input it into the trained pulse neural network for recognition to achieve speech-to-text conversion.

[0054] In the embodiment of the present invention, the speech data set used is the libirspeech data set, and the speech data preprocessing process is as follows:

[0055] Step 1.1: Speech feature extraction. First, resample the original speech data, i.e., the libirspeech dataset, to a uniform sampling rate (for example, 16000). Create a spectrogram from the audio signal. For example, set the FFT frame size to 512, the window length to 25 milliseconds, and the interval between windows to 10 milliseconds. Convert the spectrogram to a Mel-domain spectrum. For example, use 80 Mel-domain filter banks to convert the spectrogram to a Mel-domain spectrum. Finally, normalize the features according to the mean and standard deviation of the feature strength to generate a normalized speech feature sequence. For example, set the mean to 5.6501 and the standard deviation to 4.2280.

[0056] In this step, the speech data after feature extraction has 80 feature dimensions, which removes redundant information in the original speech data that is useless for speech recognition and retains information that can reflect the essential characteristics of the speech.

[0057] Step 1.2: Speech feature enhancement: First, perform frequency domain masking on the speech feature sequence processed in step 1.1. For example, the number of masks is 2, and the maximum mask size is 27. Then perform time domain masking. For example, the number of masks is between [5-10], and the maximum mask size is 0.05.

[0058] In this step, the enhanced speech features are robust to deformation in the time direction, partial loss of frequency information, and partial loss of small segments of speech.

[0059] Step 1.3: Perform 2D convolution upsampling to squeeze the speech feature sequence processed in step 1.2 out of the speech channel dimension. Then, pass it through two layers of 2D convolutional spike neurons with multiple time steps. For example, the number of time steps is 100 and the convolution kernel size is 3. The channel dimension and feature dimension of the output feature sequence are then multiplied together to form the new feature dimension.

[0060] In this step, the upsampling operation can amplify the features in the feature sequence, making it easier for the subsequent model to better capture key features.

[0061] Step 1.4: Relative Position Encoding: Each indexed token in the feature sequence processed in Step 1.3 is then relative-position encoded to generate a new unique position. This position is then summed with the pre-encoded feature sequence. For example, relative position encoding involves transforming the token using the trigonometric functions sin and cos to generate a new unique position code.

[0062] In this step, the speech feature sequence after relative position encoding can distinguish the same features at different positions, which can help the model better understand the semantics.

[0063] Each speech data item in the libirspeech dataset is an audio segment in .flac format. After preprocessing in step 1, the speech data has 80 feature dimensions. This removes redundant information in the original speech data that is useless for speech recognition (such as the blank speech between two words), making the information within each time interval more discriminative and retaining information that reflects the essential characteristics of the speech.

[0064] In some embodiments of the present invention, the modules in the self-attention convolution block include a first half-step feedforward module, a multi-head self-attention module, a convolution module, a second half-step feedforward module, and a normalization module connected in sequence. The convolutional spiking neural network is constructed by the following steps:

[0065] Step 2.1: Build a spike encoder with self-attention convolutional blocks.

[0066] like Figure 2 As shown in the figure, a linear pulse neuron layer and a Dropout layer are first constructed, and then m self-attention convolution blocks are connected. For example, m=16.

[0067] Step 2.1 specifically includes the following steps:

[0068] Step 2.1.1: Construct a linear spiking neuron layer and a dropout layer. The linear spiking neuron layer provides a linear change to the input data, mapping it to the attention computation space. The dropout layer randomly sets some elements in the input data to 0 to prevent overfitting.

[0069] Step 2.1.2: Construct a half-step feed-forward module.

[0070] In the present invention, the first half-step feedforward module and the second half-step feedforward module have the same structure, and both include a first normalization module, a first linear pulse neuron layer A, a first LIF pulse neuron layer, a first Dropout layer A, a first linear pulse neuron layer B, and a first Dropout layer B connected in sequence. The construction method thereof is as follows: Figure 3 As shown, a first normalization module is first constructed to perform layer normalization on the input data. The first linear spiking neuron layer A and the first LIF spiking neuron layer are then connected in sequence. For example, LIF-type neuron activation with a step size of 100 is used. The first dropout layer A is then connected, followed by the first linear spiking neuron layer B and the first dropout layer B. The output data of the first dropout layer B is divided by 2, and the residual is calculated with the input data of the first normalization module as the final output of the first half-step feedforward module.

[0071] In this step, layer normalization normalizes all neurons in that layer, making the input data smoother and more stable, facilitating rapid network convergence. The activation of LIF neurons enables the network to fit nonlinear functions. Residual calculations help enhance the expressive power of deep networks. The entire feedforward module enhances the model's capabilities.

[0072] Step 2.1.3: Build a multi-head self-attention module.

[0073] The multi-head self-attention module includes a second normalization module, h self-attention layers and a second Dropout layer connected in sequence. Its construction method is as follows: Figure 4 As shown in the figure, a second normalization module is first constructed to normalize the input data layer by layer, and then h self-attention layers are constructed. For example, h = 3. Finally, a second dropout layer is constructed. The second dropout layer sums the input data of the multi-head self-attention module after passing through each self-attention layer. The residual of the summed result and the input data of the second normalization module is calculated as the final output of the multi-head self-attention module.

[0074] In this step, the attention layer can filter out a small amount of important information from a large amount of input data and focus on this important information. The multi-head attention module improves the spatial resolution of the attention mechanism, increases concurrency and computational efficiency, and mainly plays the role of fitting global features.

[0075] Step 2.1.4: Build the convolutional module.

[0076] The convolution module includes a third normalization module, a point-by-point convolution layer A, a second LIF spike neuron layer A, a one-dimensional depth convolution layer, a batch normalization layer, a second linear spike neuron layer, a second LIF spike neuron layer B, a point-by-point convolution layer B, and a third Dropout layer connected in sequence. Its construction method is as follows: Figure 5 As shown, a third normalization module is first constructed to normalize the input data. A one-dimensional point-by-point convolutional layer A is then connected to reduce network parameters. For example, its convolution kernel size is 1. Linear gating is then used to divide the data into two equal parts along the feature dimension. One part is stimulated by the spiking neurons in the second LIF spiking neuron layer A and then multiplied with the other part. A one-dimensional deep convolutional layer with a convolution kernel of 31 and a batch normalization layer are then connected. After the second linear spiking neuron layer, the one-dimensional deep convolutional layer and batch normalization layer multiply the data stimulated by the spiking neurons in the second LIF spiking neuron layer B with the original data before inputting it into the next layer. Finally, a one-dimensional point-by-point convolutional layer B with a convolution kernel size of 1 and a third dropout layer are connected. The residual of the output data of the third dropout layer and the input data of the third normalization module are calculated as the final output of the convolution module.

[0077] In this step, the convolution layer can effectively extract local features. The superposition of multiple one-dimensional convolution layers can extract more effective features in various dimensions of speech features. The entire convolution module mainly plays the role of local feature fitting.

[0078] Step 2.1.5: Connect a half-step feedforward module constructed in step 2.1.2 after the convolution module, that is, the second half-step feedforward module, and then connect a normalization module for layer normalization.

[0079] The attention mechanism may not be able to fit complex processes well enough. The two feed-forward modules before and after the multi-head self-attention module can serve as supplements to further enhance the model's capabilities.

[0080] Step 2.1.6: Linearly connect m = 16 self-attention convolutional blocks consisting of the modules in steps 2.1.2, 2.1.3, 2.1.4 and 2.1.5 in sequence.

[0081] The self-attention convolution module can better fit the input features from the perspective of both global modeling and local modeling. The superposition of multiple modules enhances the feature fitting ability of the entire encoder.

[0082] In the above steps, the first LIF pulse neuron layer, the second LIF pulse neuron layer A and the second LIF pulse neuron layer B all use multi-step LIF pulse neurons.

[0083] Step 2.2: Construct a linear pulse decoder. Build a layer of linear pulse neurons as a decoder, and use the encoder output data as the input of the linear pulse decoder.

[0084] The data output by the encoder has a fixed-shape encoding state, and the decoder maps it into a sequence of variable length, which is the final recognized text sequence.

[0085] Step 2.3: Input the speech data processed in step 1 into the encoder-decoder network constructed in steps 2.1 and 2.2 for training.

[0086] The training steps are as follows:

[0087] Step 2.3.1: Set network training parameters.

[0088] For example, the batch size of the network training is set to 64, the number of batches is 2048, and the learning rate is 0.01.

[0089] Step 2.3.2: Use the pseudo BP method to perform back propagation of the network. Use a predefined gradient function or sigmoid gradient function to replace the non-differentiable part of the spiking neuron in the BP process: when the membrane potential of the spiking neuron exceeds the threshold, it outputs 1 to the downstream neuron and then resets it to the resting potential. Otherwise, it outputs 0 to the downstream neuron.

[0090] Step 2.3.3: Repeat forward propagation, loss function calculation, backward propagation, and network weight update. Training ends when the loss function on the validation dataset no longer decreases after n consecutive training rounds. For example, n = 4.

[0091] In some embodiments of the present invention, step 3 normalizes the output of the linear pulse decoder and then takes its logarithm (specifically, this can be achieved using the logsoftmax function) to accelerate decoding speed and improve decoding stability. A greedy search strategy is then used to convert the output content into text, with each decoding step selecting the predicted label with the highest current probability (specifically, this can be achieved using the argmax function). The continuous decoding output is the text of the speech recognition.

[0092] The linear spiking neuron layer in the decoder outputs predictions for time durations, representing the output probabilities of all labels in matrix form (for example, the probability of the output label being the word "and" is 0.2). A greedy search strategy is then used for decoding. For each time step, the predictions are normalized and logarithmized using the logsoftmax function to speed up decoding and improve decoding stability. Each decoding step uses the argmax function to obtain the predicted label with the highest probability among the predictions for the current time step. The continuous decoding output is the text generated by speech recognition.

[0093] The method of the present invention can be compiled into a computer program, that is, the steps of the method of the present invention can be implemented using a computer program, thereby obtaining a computer program product.

[0094] The computer program of the present invention can be run on a computing device including a storage unit and a processing unit. The computer program is stored in the storage unit and executed by the processing unit through commands.

[0095] A specific application of the present invention may be an end-to-end speech recognition system based on a pulse neural network, comprising a language acquisition device, a display device, and the computing device described above;

[0096] The speech acquisition device collects the speech data to be recognized, and can be an independent microphone device or a speech input part integrated into other devices, such as a mobile phone.

[0097] The processing unit of the computing device includes a preprocessing module for performing preprocessing, a training module for performing network training, and an operation module for performing conversion. The preprocessing module is mainly used to perform step 1 of the present invention, the training module is mainly used to perform step 2 of the present invention, and the operation module is mainly used to perform step 3 of the present invention. After completing the training of step 2 with sufficient data, the training module is no longer necessary during specific use. At this time, the preprocessing module takes the speech data to be recognized as input, and the operation module takes the preprocessed speech data to be recognized as input, and outputs the obtained text through the trained pulse neural network.

[0098] The display device is used to display the content of the text. The output text can be displayed visually or played in audio form while being displayed.

[0099] To test the effectiveness of the present invention, the original data numbered 84-121123-0028 in the libirspeech dataset was used as the test input. After the method of the present invention, the recognition result obtained was: "i am going sir and i do not hesitate to say that no prayers will be more fervent than mine" (Chinese meaning: I am going, sir. I do not hesitate to say that no prayers will be more fervent than mine).

[0100] In summary, the present invention consists of three steps: preprocessing speech data, training a convolutional spiking neural network with self-attention, and then inputting the preprocessed speech data into the trained spiking neural network for recognition. It can be used for end-to-end speech recognition, particularly with large-scale speech datasets, accurately identifying text content. Compared with other speech recognition schemes, the present invention incorporates more biological characteristics and leverages the energy-saving and efficient characteristics of spiking neurons, offering advantages such as high accuracy, minimal computational effort, and low energy consumption.

[0101] The above description is merely a preferred embodiment of the present invention for a specific problem of image classification. The present invention should not be limited to the problem form, specific implementation examples, or the contents disclosed in the accompanying drawings. Any equivalent or modified versions that do not depart from the spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. An end-to-end speech recognition method based on a spiking neural network, characterized in that: The steps include: Step 1: Speech data preprocessing, including feature extraction and feature enhancement of the original speech data, upsampling, and finally relative position encoding; Step 2: using the data set consisting of the preprocessed speech data to train a convolutional spiking neural network with self-attention; the convolutional spiking neural network with self-attention includes a convolutional spiking encoder with self-attention and a linear spiking decoder; The convolutional pulse encoder with self-attention includes a linear pulse neuron layer, a dropout layer and a self-attention convolution block composed of m pulse neurons connected in sequence, wherein the activation function is replaced by a LIF pulse neuron for excitation in the self-attention convolution block, and residual calculation is performed between the input and output of each module of the self-attention convolution block; The linear pulse decoder is a layer of linear pulse neurons, which takes the output data of the convolutional pulse encoder with self-attention as input; The modules in the self-attention convolution block include a first half-step feedforward module, a multi-head self-attention module, a convolution module, a second half-step feedforward module, and a normalization module connected in sequence; The first half-step feedforward module and the second half-step feedforward module have the same structure, and both include a first normalization module, a first linear pulse neuron layer A, a first LIF pulse neuron layer, a first Dropout layer A, a first linear pulse neuron layer B, and a first Dropout layer B connected in sequence, wherein the output data of the first Dropout layer B is divided by 2 and the residual is calculated with the input data of the first normalization module as the final output of the first half-step feedforward module; The multi-head self-attention module includes a second normalization module, h self-attention layers, and a second Dropout layer connected in sequence. The second Dropout layer sums the input data of the multi-head self-attention module through the results of each self-attention layer, and the summation result is calculated with the input data of the second normalization module as the final output of the multi-head self-attention module; The convolution module includes a third normalization module, a point-by-point convolution layer A, a second LIF pulse neuron layer A, a one-dimensional depth convolution layer, a batch normalization layer, a second linear pulse neuron layer, a second LIF pulse neuron layer B, a point-by-point convolution layer B and a third Dropout layer connected in sequence. The one-dimensional deep convolution layer and the batch normalization layer perform a multiplication operation on the data after the LIF pulse neuron is excited and the data before the excitation, and the output data of the third Dropout layer and the input data of the third normalization module are subjected to residual calculation as the final output of the convolution module; Step 3: After preprocessing the speech data to be recognized, input it into the trained pulse neural network for recognition to achieve speech-to-text conversion.

2. The end-to-end speech recognition method based on a pulse neural network according to claim 1, characterized in that: The process of the step 1, voice data preprocessing is as follows: Step 1.1: Speech feature extraction: First, resample the original speech data according to a uniform sampling rate, then create a spectrogram from the audio signal and convert it into a mel-domain spectrum. Finally, normalize the feature strengths according to their mean and standard deviation to generate a normalized speech feature sequence. Step 1.2: Speech feature enhancement: first perform frequency domain masking on the speech feature sequence processed in step 1.1, and then perform time domain masking; Step 1.3: 2D convolution upsampling: squeeze the speech feature sequence processed in step 1.2 out of the speech channel dimension, then pass it through two layers of 2D convolutional spike neurons with multiple time steps. After that, the channel dimension and feature dimension of the output feature sequence are multiplied to form a new feature dimension. Step 1.4: Relative position encoding: generate a new unique position for each indexed token in the feature sequence processed in step 1.3 through relative position encoding, and then sum the new unique position with the feature sequence before encoding.

3. The end-to-end speech recognition method based on a pulse neural network according to claim 1, characterized in that: The first LIF pulse neuron layer, the second LIF pulse neuron layer A, and the second LIF pulse neuron layer B all use multi-step LIF pulse neurons.

4. The end-to-end speech recognition method based on a pulse neural network according to any one of claims 1 to 3, characterized in that: In step 2, the training process is as follows: Step 2.1: Set network training parameters; Step 2.2: Use the pseudo BP method to perform back propagation of the network. Use the sigmoid gradient function to replace the non-differentiable part of the spiking neuron in the BP process. When the membrane potential of the spiking neuron exceeds the threshold, it outputs 1 to the downstream neuron and then resets it to the resting potential; otherwise, it outputs 0 to the downstream neuron. Step 2.3: Repeat forward propagation, loss function calculation, backward propagation, and network weight update until the loss function on the validation dataset no longer decreases after n consecutive training rounds, and the training ends.

5. The end-to-end speech recognition method based on a pulse neural network according to claim 1 or 3, characterized in that: In step 3, the output of the linear pulse decoder is normalized and then logarithm is taken to speed up the decoding speed and improve the decoding stability; the output content is converted into text using a greedy search strategy, and the predicted label with the highest current probability is selected for each decoding. The continuous decoding output is the text obtained by speech recognition.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processing unit, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computing device comprising a storage unit and a processing unit, wherein the storage unit stores a computer program, wherein: When the processing unit executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. An end-to-end speech recognition system based on a spiking neural network, comprising a speech acquisition device, a display device, and the computing device according to claim 7; The language acquisition device acquires voice data to be recognized; The computing device, wherein the processing unit includes a preprocessing module for performing preprocessing, a training module for performing network training, and an operation module for performing conversion; the preprocessing module takes the speech data to be recognized as input, and the operation module takes the preprocessed speech data to be recognized as input and outputs the obtained text; The display device displays the content of the text.

Images