A speech event stream recognition method based on local recurrent memory pulse neural network

Abstract

This invention discloses a speech event stream recognition method based on a local recurrent memory spiking neural network, comprising: acquiring a speech event stream containing event occurrence time and auditory channel identifiers; binning the stream into speech event vectors according to a preset time interval; constructing a local recurrent memory module composed of input integral neurons, memory neurons, context neurons, and output neurons; generating input modulation signals, memory modulation signals, and output modulation signals based on the current speech event vector and the output pulse of the previous time step; updating the state of each neuron in causal order; forming a cross-time step local recurrent memory pathway through connections from the context pulse of the previous time step to the current memory neuron and from the current memory pulse to the current context neuron; training the network using data with speech category labels and outputting the speech category. This invention can enhance the ability to preserve the features of preceding acoustic events and reduce the overhead of state storage and nonlinear computation.

Description

Technical Field

[0001] This invention relates to the fields of neuromorphic speech processing, spiking neural networks, and brain-like computing technologies, and in particular to a speech event stream recognition method based on a local recurrent memory spiking neural network. Background Technology

[0002] Speech recognition technology is widely used in smart terminals, human-computer interaction, and device control. Traditional speech recognition methods typically require spectral transformation of continuous speech signals and the execution of numerous intensive multiply-accumulate operations using artificial neural networks, resulting in significant computational and storage overhead. With the development of neuromorphic auditory technology, continuous speech signals can be converted into a speech event stream using cochlear implant models or auditory event encoders. This speech event stream consists of multiple discrete pulse events, each including the event occurrence time and the corresponding auditory channel, enabling a sparse, event-driven representation of the time-frequency variations of the speech signal.

[0003] Spiking neural networks transmit and process information through pulse events, exhibiting characteristics such as event-driven processing and computational sparsity, making them suitable for recognizing speech event streams. Among them, leakage integral firing neurons are often used to construct recurrent spiking neural networks due to their simple structure, few state variables, and low computational overhead. Leakage integral firing neurons integrate the input current, generating a pulse when the membrane potential reaches the firing threshold, and causing the membrane potential to gradually leak over time.

[0004] However, in speech event stream recognition, a spoken digit or voice command typically consists of multiple consecutive phonemes and syllables, and the acoustic event features generated in the initial speech segment continuously influence subsequent recognition processes. Because the membrane potential of the leakage integral firing neurons decays over time, effective acoustic features from earlier time steps are easily lost. When silence segments, background noise, or irrelevant impulse events exist in the speech event stream, subsequent inputs may interfere with or cover already formed speech features, making it difficult for the network to accurately recognize speech content with a longer duration.

[0005] To enhance the temporal memory capabilities of spiking neural networks, existing techniques typically employ adaptive firing thresholds, multi-compartment neurons, variable time constants, or complex gating structures. While these methods can extend information retention time, they usually require increasing the internal states of neurons, trainable parameters, and nonlinear operations, leading to increased computational and state storage requirements per time step. This hinders efficient deployment on low-precision, event-driven neuromorphic processors.

[0006] Furthermore, recurrent spiking neural networks typically employ time-based backpropagation for training. As the length of the speech event sequence increases, the gradient needs to continuously pass through membrane potential leakage, pulse firing, and reset processes, which easily leads to attenuation. This makes it difficult to obtain effective training signals from the event information in the early stages of the speech, further limiting the network's ability to learn acoustic features across time.

[0007] In summary, existing speech event stream recognition methods struggle to effectively preserve the acoustic event features of the initial speech segment while maintaining the simplicity and low computational cost of the leakage integral firing neuron structure, and to reduce the interference of silence and noise events on historical information. Therefore, there is an urgent need for a speech event stream recognition method that enhances the preservation of speech features across time steps, has low neuronal state complexity, and is suitable for neuromorphic hardware implementation. Summary of the Invention

[0008] This invention provides a speech event stream recognition method based on a local recurrent memory spiking neural network. While retaining the advantages of simple structure and low computational cost of leakage integral firing neurons, it effectively preserves the acoustic event features of the speech pre-speech segment and reduces the interference of silence and noise events on historical information, thereby improving the recognition accuracy of speech event streams.

[0009] A speech event stream recognition method based on a local recurrent memory spiking neural network includes the following steps: (1) Obtain the speech event stream data for training, and after preprocessing, divide it into speech event vectors corresponding to multiple time steps in chronological order; (2) Construct a spiking neural network for recognizing speech event streams; the spiking neural network includes two connected spiking computation layers, each of which includes at least one local recurrent memory module; the local recurrent memory module includes an input integral neuron, a memory neuron, a context neuron, and an output neuron; The spiking neural network enhances the retention of historical information by constructing a local recurrent memory pathway across time steps between memory neurons and context neurons; the update process of each neuron is as follows: Based on the speech event vector at the current time step and the output neuron pulse at the previous time step, the input modulation current, memory modulation current, and output modulation current are generated respectively. The state of the input integral neuron is updated based on the input modulation current to generate the input pulse at the current time step; The state of the memory neuron is updated based on the context impulse and memory modulation current of the previous time step, and the memory impulse of the current time step is generated. The state of the context neuron is updated based on the input pulse and memory pulse at the current time step, and the context pulse at the current time step is generated. The state of the output neuron is updated based on the context pulse and output modulation current at the current time step, and the output neuron pulse at the current time step is generated. (3) During the training phase, the preprocessed speech event stream data from step (1) is input into the spiking neural network in chronological order, and the output state (membrane potential state and pulse state) of the next pulse calculation layer is input into the speech category output layer to obtain the speech category. The results are then compared with the true values ​​for training. (4) In the application stage, the sequence of speech events to be recognized is input into the trained spiking neural network in chronological order. The state of each local recurrent memory module is updated sequentially in chronological order, and the output state of the next spiking calculation layer is input into the speech category output layer to obtain the speech category.

[0010] The method of this invention improves the long-term information processing of recurrent spiking neural networks: by constructing a local recurrent memory pathway across time steps between memory neurons and context neurons, the ability to retain historical information is enhanced; by setting mutually independent input modulation, memory modulation, and output modulation branches, adaptive adjustment of the information writing, memory updating, and information reading processes is achieved; and by using piecewise linear modulation functions, the implementation overhead of complex nonlinear operations in the inference stage is reduced.

[0011] Further, in step (1), the speech event stream includes multiple speech pulse events, each of which includes at least the event occurrence time and the auditory channel identifier. The speech event stream is time-binned according to a preset time interval, and the speech pulse events corresponding to different auditory channels in each time interval are counted to form speech event vectors corresponding to multiple time steps.

[0012] The specific preprocessing process is as follows: Depending on the type of speech event stream data, the original speech event stream is normalized, standardized, time-aligned, binned, dimensionality-transformed, or sequence-arranged, and each sample is represented as a sequence of length [length missing]. Timing input sequence: in, This indicates the number of time steps contained in the time-series input sequence. Indicates the first The speech vector corresponding to each time step.

[0013] Preferably, for time-series input data composed of asynchronous events, the asynchronous events are binned according to a preset time interval, and the number of events, event polarity, or event position in each time interval is used as the input data for the corresponding time step.

[0014] It is worth noting that this invention does not limit the timing input data to a specific encoding method. Depending on the task being processed, the input data can be input into the spiking neural network using methods such as direct input, frequency encoding, time encoding, or event counting encoding.

[0015] Furthermore, in step (2), the input integral neuron, memory neuron, context neuron and output neuron are all leakage current integral firing neurons, adaptive leakage current integral firing neurons or spiking neurons with variable firing thresholds.

[0016] Furthermore, input integral neurons With context neurons Connections, memory neurons With context neurons Connections, context neurons With output neurons Connections; simultaneously, context neurons The contextual impulses generated in the previous time step are fed back to the memory neurons. This forms the following local loop relationship: .

[0017] Local recurrent relationships enable historical context information to propagate to subsequent time steps through structured interactions between memory neurons and context neurons, thereby reducing information decay caused by relying solely on the membrane potential of a single neuron for the preservation of historical information.

[0018] Further, in step (2), the process of generating the input modulation current, memory modulation current, and output modulation current is as follows: the voice event vector of the current time step is... Compared with the output neuron pulse of the previous time step Feature concatenation is performed to obtain combined data. ,in, This represents a splicing operation along the feature dimension; and generates three types of modulation currents through independent parameter transformations, i.e., combining the data... The input modulation branch, memory modulation branch, and output modulation branch are independent inputs, generating input modulation current, memory modulation current, and output modulation current.

[0019] Specifically, the input modulation current is used to adjust the degree to which current input information enters the local recurrent memory module; the memory modulation current is used to adjust the retention and updating of historical information in memory neurons; and the output modulation current is used to adjust the degree to which context information is read from and output from the local recurrent memory module.

[0020] Preferably, the input modulation branch, memory modulation branch, and output modulation branch each employ independent weight parameters and bias parameters, enabling the three types of modulation currents to be trained for different information processing functions.

[0021] Preferably, the output neuron pulses of the previous time step simultaneously participate in the generation of three types of modulated currents, enabling the local recurrent memory module to adaptively adjust the information writing, memory updating, and information reading processes of the current time step according to the historical output state.

[0022] It is worth noting that the three types of modulation currents are not independent, continuous-value gated states stored in traditional artificial neural networks. Instead, they act as input currents or modulatory quantities of input currents on the corresponding spiking neurons, combining the modulation process with the membrane potential update process of the spiking neurons.

[0023] In step (2), the input relationships of each neuron within the local recurrent memory module are as follows: The input received by the input integrator neuron at the current time step is: ; The input received by the memory neuron at the current time step is: ;; The input received by the context neuron at the current time step is: ; The input received by the output neuron at the current time step is: ; in, , , and These represent the inputs received by the input integral neuron, memory neuron, context neuron, and output neuron at the current time step, respectively. , , and These represent the pulses generated by the input integral neuron, memory neuron, context neuron, and output neuron at the current time step, respectively. , and This represents the scaling parameter for the corresponding modulation current; , , and This represents the connection parameters between different neurons.

[0024] Furthermore, parameters The parameters correspond to the cross-time-step connection from the context neuron to the memory neuron. The connection between the corresponding memory neuron and the context neuron at the current time step together constitutes the local recurrent memory pathway.

[0025] Preferably, the scaling parameters, connection parameters, weight parameters of the modulation branch, and bias parameters are all trainable parameters and are shared between different time steps.

[0026] Preferably, different local cyclic memory modules can be set with independent connection parameters, or some local cyclic memory modules can share connection parameters, so as to adjust the number of model parameters and computational overhead according to actual application requirements.

[0027] In step (2), when the input integral neuron, memory neuron, context neuron, and output neuron are all leakage integral firing neurons, the state update process is as follows: For any spiking neuron among the input integrator neuron, memory neuron, context neuron, and output neuron Its membrane potential is updated according to the following formula: ; Its pulse state is determined according to the following formula: ; in, , This represents the membrane potential of the corresponding neuron at the current time step. Indicates the membrane potential leakage coefficient. Indicates the pulse firing threshold. This represents the input received by the corresponding neuron at the current time step. This represents the step release function.

[0028] When membrane potential Reaching or exceeding the pulse firing threshold When the pulse is generated, the corresponding neuron produces a pulse; after the pulse is generated, the membrane potential of the corresponding neuron is adjusted by soft reset or hard reset.

[0029] Preferably, different neurons employ different membrane potential leakage coefficients or firing thresholds to give different neurons different time response characteristics.

[0030] It is worth noting that the long-term memory capability of this invention is mainly achieved through local recurrent connections between multiple spiking neurons, rather than necessarily through increasing higher-order state variables within a single spiking neuron.

[0031] In this invention, the local recurrent memory module updates the state of the current time step sequentially according to the order of input integral neurons, memory neurons, context neurons, and output neurons.

[0032] It is worth noting that memory neurons receive context impulses from the previous time step, while context neurons receive memory impulses generated in the current time step. Therefore, there are no algebraic loops that need to be solved simultaneously within the same time step, which can guarantee the causality of the network computation process.

[0033] Furthermore, in step (2), each pulse calculation layer is formed by multiple local cyclic memory modules set in parallel; The previous pulse calculation layer receives the speech event vector of the current time step and uses the output pulse generated at the current time step as the input of the next pulse calculation layer at the current time step; each pulse calculation layer saves the corresponding membrane potential state and pulse state in the time dimension and updates them cyclically along the time dimension.

[0034] Furthermore, the spiking neural network also includes a speech category output layer; the speech category output layer includes at least one of a linear classifier, a multilayer perceptron, and a convolutional processing structure; The speech category output layer receives the output state of the next pulse calculation layer and maps it to a speech category.

[0035] The data processing results are obtained through the speech category output layer, as follows: exist Within each time step, the output pulses, membrane potential states, or combinations thereof from multiple time steps of the last pulse calculation layer are collected. The output pulses, membrane potential states, or combinations thereof are then input into the output processing layer, which maps them to the corresponding data processing results through the speech category output layer.

[0036] Preferably, for a temporal classification task, the output neuron states of multiple time steps in the last layer are input into a linear classifier to obtain the prediction results for the corresponding category.

[0037] Preferably, the output pulses or membrane potentials of the output neurons in the last layer at multiple time steps can be accumulated, averaged, selected at the maximum value, or weighted and fused, and the fusion result can be input into the speech category output layer.

[0038] It is worth noting that, in addition to using a fully connected layer, the speech category output layer can also use a multilayer perceptron, a convolutional processing structure, or an attention-based processing structure.

[0039] Furthermore, in step (3), the training process of the spiking neural network includes: The training audio event stream with audio category labels is expanded according to time steps; The speech recognition loss is calculated based on the speech category prediction results and corresponding speech category labels output by the spiking neural network. A time-based backpropagation method is used to calculate the gradient of speech recognition loss with respect to network parameters; During the backpropagation of the pulse firing function, a continuously differentiable alternative gradient function is used to approximate the gradient of the step firing function. The network parameters updated according to the gradient include at least the weight and bias parameters of the modulation branch, the modulation current scaling parameters, and the local loop connection parameters.

[0040] Alternatively, the alternative gradient function may be a rectangular function, a trigonometric function, a fast sigmoid derivative function, or another continuous function that has a non-zero gradient near the firing threshold.

[0041] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention employs input integral neurons, memory neurons, context neurons, and output neurons to construct a local recurrent memory module, and establishes a cross-timestep local recurrent memory pathway between memory neurons and context neurons. Compared to methods that rely solely on the membrane potential of a single neuron to preserve historical information, this local recurrent memory pathway can provide additional propagation paths for historical information, mitigating the decay of effective information over long time spans.

[0042] 2. This invention generates input modulation current, memory modulation current, and output modulation current based on the input data of the current time step and the output state of the previous time step, respectively, to separately regulate the processes of writing current information, maintaining and updating historical information, and reading memory information. When valid information and noise information alternate in the input sequence, it helps to reduce the overwriting of existing memory by irrelevant inputs.

[0043] 3. This invention enhances long-term processing capabilities by employing structured connections between ordinary leakage current integral firing neurons, eliminating the need to set multiple high-order state variables within each neuron. Compared to adaptive threshold neurons, multi-compartment neurons, or complex gated loop structures, this approach helps reduce the state storage requirements and time-step update complexity of individual computational neurons.

[0044] 4. This invention sets up local recurrent connections between memory neurons and context neurons, so that backpropagation gradients can propagate through recurrent connections between neurons in addition to the membrane potential leakage path. This helps to reduce gradient interruptions in continuous time steps and improve the temporal credit allocation capability and network training stability over long time spans.

[0045] 5. In the training process, the present invention gradually converts the smooth nonlinear modulation function into a piecewise linear modulation function, and uses linear transformation and numerical truncation operations to generate the modulation current during the inference stage. This helps to reduce high-precision complex nonlinear calculations and is easy to implement on low-precision, event-driven neuromorphic processors. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram showing the connection relationship of each spiking neuron in the local recurrent memory module of the present invention.

[0048] Figure 2 This is a schematic diagram of the multi-layer spiking neural network structure used for speech event stream recognition according to the present invention.

[0049] Figure 3 This is a schematic diagram of the overall process of the voice event stream recognition method of the present invention.

[0050] Figure 4 This is a schematic diagram of the causal state update process of the local circular memory module of the present invention when processing a single voice event time step. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0053] This embodiment takes the Heidelberg Spiking Digits (SHD) speech recognition task as an example. The speech event stream data processing process includes input data preprocessing, construction of a local recurrent spiking neural network, time-step forward computation, network training, and inference testing. Specifically, each sample in the SHD dataset is a sequence of pulse events obtained from speech-to-digit conversion. Each pulse event includes at least the event occurrence time and the corresponding input channel number, with a total of 700 input channels. First, according to a preset time interval... For each sample's pulse event sequence, time discretization is performed, dividing the sample into discrete time sequences of length T. Within the t-th time step, the statistical time interval is... The pulse events of each input channel are recorded, and a 700-dimensional input pulse vector is constructed based on these events. This converts the original asynchronous pulse events into an input sequence that can be processed step-by-step by the local recurrent spiking neural network.

[0054] Step 1: Construct a local recurrent spiking neural network.

[0055] like Figure 1 , Figure 2 and Figure 3 As shown, specifically, this embodiment constructs a two-layer local recurrent spiking neural network. Each spiking computation layer contains 128 parallel local recurrent memory modules, and each local recurrent memory module includes an input integral neuron. Memory neurons Context neurons and output neurons The first pulse computation layer receives the input vector obtained by linear mapping of the speech event vector at the current time step; the output pulse generated by the first pulse computation layer at the current time step serves as the input of the second pulse computation layer at the current time step; the membrane potentials and pulse behaviors of the output neurons of the second pulse computation layer at all time steps are connected to the twenty-category linear readout layer.

[0056] Preferably, all four types of functional neurons in this embodiment are leakage current integral firing neurons. Each functional neuron is configured with a membrane potential state. , , and and pulse state , , and When the network starts processing a new sample, it initializes all membrane potential states and pulse states to 0; these states are retained and continuously updated between 100 time steps for the same sample.

[0057] Specifically, the three modulation branches correspond to input modulation, memory modulation, and output modulation, respectively. For each layer in the network, the current layer's input is concatenated with the output pulse from the previous time step, and then three independent fully connected transforms are input. The weights and biases of the three modulation branches are shared across all time steps, but different pulse computation layers use their own independent parameters.

[0058] Step 2, the forward computation process for a single time step.

[0059] For any time step t and any pulse computation layer l, according to Figure 4 Perform the following operations in the order shown. For ease of explanation, layer label 1 is omitted below; the specific process is as follows: 2-1, Read the current time step input Output pulse at the previous time step When t=1, the output pulse from the previous time step is set to an all-zero vector; the two are then concatenated along the feature dimension to form a combined feature. .

[0060] 2-2, Input the combined feature z[t] into the modulation branch, the memory modulation branch, and the output modulation branch respectively, and calculate the input modulation current. Memory modulation current and output modulation current .

[0061] Among them, the input modulation current The transmission intensity of the speech event vector at the current time step to the input integrator neuron is adjusted; memory modulation current. Used to regulate the retention or update of memory neurons at the current time step; output modulated current. This is used to adjust the read intensity of the current context to the output neuron. It is noteworthy that the three modulation branches receive the same combined features but use independent parameters, thus enabling the generation of modulation results for different functions.

[0062] 2-3, Calculate the input integral neuron The input current is used to update the membrane potential and pulse state.

[0063] Specifically, the input integrator neuron converts the current input modulation current into a pulse representation. When the input modulation current is large, the current input is more likely to cause the input integrator neuron to reach the firing threshold and generate a pulse; when the input modulation current is small, the influence of the current input on the subsequent context state is weakened.

[0064] 2-4, Calculating memory neurons The input current is used to update the membrane potential and pulse state.

[0065] Specifically, the input to the memory neuron includes both the context pulse from the previous time step and the current memory modulation current. The context pulse from the previous time step carries fused historical information, and the memory modulation current is used to adjust the degree to which this historical information is maintained or updated at the current time step.

[0066] 2-5. Using the input pulses and memory pulses already generated at the current time step, calculate the context neuron. The input current is recorded, and the context state is updated.

[0067] Specifically, context neurons fuse current input information with historical memory information. It's worth noting that memory neurons use context impulses from the previous time step, while context neurons use memory impulses from the current time step; therefore, a transition from the previous time step to the current time step is formed. → → It has a local circular memory path and will not form a circular solution within the current time step.

[0068] 2-6, Calculate the output neuron The input current is calculated, and the output state is updated.

[0069] Specifically, the output neuron generates the output pulse for the current time step based on the current context and the output modulation current. The output pulse is passed to the next pulse calculation layer or output processing layer, and in the next time step, it is used together with the new input to generate three types of modulation currents.

[0070] 2-7, save the updated membrane potentials and pulse states of the four types of functional neurons, and proceed to the next time step. Preferably, after the first pulse calculation layer completes the calculation for the current time step, it will... As the current input to the second pulse calculation layer, the second pulse calculation layer completes the calculation of the current time step according to the same process.

[0071] Step 3: Complete all time step calculations and output the classification results.

[0072] Specifically, all inputs from t=1 to t=100 are processed sequentially according to step 2. Since the state of each functional neuron is continuously retained between different time steps of the same sample, the speech event vector of an earlier time step can continuously influence the subsequent state through the local loop between memory neurons and context neurons; at the same time, the three types of modulation currents change dynamically according to the current speech event vector and historical output, thereby reducing the interference of irrelevant speech event vectors on existing memory, and reading the context information for classification in the later part of the sequence.

[0073] After the 100th time step is completed, preferably, the pulse firing and membrane potential aggregation results of all time steps of the output neuron of the second pulse calculation layer are taken as a sequence representation, and the sequence representation is input into the readout layer to obtain the prediction scores corresponding to the twenty categories.

[0074] Step 4, train the local recurrent spiking neural network: Specifically, the training process is performed on a mini-batch of samples: first, the network state corresponding to each sample is initialized; then, the forward computation of 100 time steps and two spurious computation layers is completed according to step 2; then, the cross-entropy loss is calculated based on the readout layer prediction score and the real digit label.

[0075] ; in, This represents the one-hot encoding of the true label. Since the true gradient of the step pulse firing function is zero at most locations, an alternative gradient is used to approximate the derivative of the step function during time-based backpropagation, allowing a non-zero gradient to be obtained when the membrane potential is near the firing threshold. It is worth noting that backpropagation calculates the gradients of each time step and each network layer in the order from the last time step to the initial time step.

[0076] Preferably, after each mini-batch completes backpropagation, the Adam optimizer is used to update the parameters of the three modulation branches, the connection parameters between the four types of functional neurons, the inter-layer connection parameters, the modulation scaling parameters, and the readout layer parameters; the above process is repeated until the preset number of training rounds is reached or the performance of the validation set no longer improves.

[0077] Step 5, Reasoning Test Phase.

[0078] Specifically, the network parameters obtained during training are loaded during inference. For each speech event stream to be recognized, the speech event vector is expanded into 100 time steps according to the training phase, the network state is initialized, and then three types of modulation current generation, input integral neuron update, memory neuron update, context neuron update, output neuron update, and inter-layer propagation are executed sequentially. After all time steps are completed, the readout layer outputs the classification result.

[0079] Step 6: Implementation results and explanation of working principle.

[0080] Preferably, this embodiment employs a two-layer local recurrent spiking neural network, with each layer containing 128 parallel local recurrent memory modules. Each local recurrent memory module includes an input integrator neuron, a memory neuron, a context neuron, and an output neuron. Training is performed using the SHD training set, achieving a classification accuracy of 94.70% on the test set. Compared to the 71.40% classification accuracy of a recurrent spiking neural network using only a similar number of ordinary leakage integrator neurons without local recurrent neurons, the classification accuracy is improved by 23.30 percentage points. These results demonstrate that, using ordinary leakage integrator neurons, the local recurrent structure can handle long input sequences containing 100 time steps.

[0081] Specifically, in this embodiment, long-term information retention is achieved through two mechanisms: First, the output pulse from the previous time step participates in the generation of the three types of modulation currents in the current time step, enabling the network to adjust the writing, memory updating, and information retrieval of current information according to historical states; Second, cross-time-step local recurrence between context neurons and memory neurons allows historical information to be retained along... The pathway continues to propagate. The two mechanisms mentioned above work together, enabling the network to no longer rely solely on the membrane potential of a single leaky integral firing neuron to preserve long-term information.

[0082] It is worth noting that during backpropagation in training, local loops also provide an additional propagation pathway for gradients, distinct from the membrane potential leakage path of single neurons. When memory neurons and context neurons are in the effective region of the alternative gradient, the gradient can propagate along the connection between them to earlier time steps, thereby reducing the probability of continuous gradient decay in long-sequence training.

[0083] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for speech event stream recognition based on a local recurrent memory spiking neural network, characterized in that, Includes the following steps: (1) Obtain the speech event stream data for training, and after preprocessing, divide it into speech event vectors corresponding to multiple time steps in chronological order; (2) Construct a spiking neural network for recognizing speech event streams; the spiking neural network includes two connected spiking computation layers, each of which includes at least one local recurrent memory module; The local recurrent memory module includes an input integral neuron, a memory neuron, a context neuron, and an output neuron; The spiking neural network enhances the retention of historical information by constructing a local recurrent memory pathway across time steps between memory neurons and context neurons; the update process of each neuron is as follows: Based on the speech event vector at the current time step and the output neuron pulse at the previous time step, the input modulation current, memory modulation current, and output modulation current are generated respectively. The state of the input integral neuron is updated based on the input modulation current to generate the input pulse at the current time step; The state of the memory neuron is updated based on the context impulse and memory modulation current of the previous time step, and the memory impulse of the current time step is generated. The state of the context neuron is updated based on the input pulse and memory pulse at the current time step, and the context pulse at the current time step is generated. The state of the output neuron is updated based on the context pulse and output modulation current at the current time step, and the output neuron pulse at the current time step is generated. (3) During the training phase, the preprocessed speech event stream data from step (1) is input into the spiking neural network in chronological order, and the output state of the next spiking calculation layer is input into the speech category output layer to obtain the speech category. The results are then compared with the true values ​​for training. (4) In the application stage, the sequence of speech events to be recognized is input into the trained spiking neural network in chronological order. The state of each local recurrent memory module is updated sequentially in chronological order, and the output state of the next spiking calculation layer is input into the speech category output layer to obtain the speech category.

2. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 1, characterized in that, In step (1), the voice event stream includes multiple voice pulse events, and each voice pulse event includes at least the event occurrence time and the auditory channel identifier.

3. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 1, characterized in that, In step (2), the input integral neuron, memory neuron, context neuron and output neuron are all leakage current integral firing neurons, adaptive leakage current integral firing neurons or spiking neurons with variable firing thresholds.

4. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 1, characterized in that, In step (2), the process of generating input modulation current, memory modulation current and output modulation current is as follows: the speech event vector of the current time step is concatenated with the output neuron pulse of the previous time step to obtain combined data, and the three types of modulation current are generated by independent parameter transformations.

5. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 1, characterized in that, In step (2), each pulse calculation layer is formed by multiple local cyclic memory modules set up in parallel; The previous pulse calculation layer receives the speech event vector of the current time step and uses the output pulse generated at the current time step as the input of the next pulse calculation layer at the current time step; each pulse calculation layer saves the corresponding membrane potential state and pulse state in the time dimension and updates them cyclically along the time dimension.

6. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 1, characterized in that, The speech category output layer includes at least one of a linear classifier, a multilayer perceptron, and a convolutional processing structure. The speech category output layer receives the output state of the next pulse calculation layer and maps it to a speech category.

7. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 1, characterized in that, In step (3), the training process of the spiking neural network includes: The training audio event stream with audio category labels is expanded according to time steps; The speech recognition loss is calculated based on the speech category prediction results and corresponding speech category labels output by the spiking neural network. A time-based backpropagation method is used to calculate the gradient of the speech recognition loss with respect to the network parameters; During the backpropagation of the pulse firing function, a continuously differentiable alternative gradient function is used to approximate the gradient of the step firing function; The network parameters updated according to the gradient include at least the weight and bias parameters of the modulation branch, the modulation current scaling parameters, and the local loop connection parameters.

8. The speech event stream recognition method based on a local recurrent memory spiking neural network according to claim 7, characterized in that, The alternative gradient function is a rectangular function, a trigonometric function, a fast sigmoid derivative function, or another continuous function that has a non-zero gradient near the firing threshold.

Images