A robot real-time voice interaction method and system based on a spiking neural network

Abstract

This invention discloses a real-time voice interaction method and system for robots based on a spiking neural network. The method involves acquiring a voice dataset and superimposing it with robot operating noise to generate augmented samples. Voice signal features are extracted from these samples and data augmentation is performed. A recurrent spiking neural network model with loop connections is constructed and trained using the augmented data for speech recognition. Real-time voice stream input is acquired from the robot, features are extracted, and input into the trained recurrent spiking neural network model for inference. Based on the pulse decoding results, the robot is controlled to perform interactive actions. This invention reduces recognition power consumption, mitigates the impact of robot operating noise on the accuracy of real-time voice interaction, avoids the problem of no-pulse spiking neurons, and allows the system to adjust the sensitivity of voice interaction commands during operation.

Description

Technical Field

[0001] This invention belongs to the field of neuromorphic computing and human-computer interaction, specifically relating to a real-time voice interaction method and system for robots based on spiking neural networks. Background Technology

[0002] Spiking neural networks (SNNs), as a neural network model simulating the operation of the nervous system, have attracted much attention in the field of robot control in recent years. They mimic the pulse information processing mechanism of the biological brain. SNNs operate in an event-driven manner, meaning they perform computation and communication only when neurons fire pulses. This characteristic allows SNNs to significantly reduce power consumption when processing sparse data, making them particularly suitable for embedded systems and mobile robot applications. Furthermore, the event-driven mechanism of SNNs makes them ideal for processing real-time data, enabling rapid responses to sensor inputs in dynamic environments, thus providing highly efficient real-time processing capabilities.

[0003] However, for training spiking neural networks, the "dead neuron" problem becomes more prominent as network depth increases. This means that some neurons fail to fire during training or inference, leading to a decline in the network's expressive power and performance. Furthermore, suppressing the firing of neurons in the output layer that do not meet the correct category during training can easily lead to overfitting. For voice interaction tasks, spiking neural network models that perform well on training datasets may not achieve the expected performance in real-world robotic applications. This gap stems from the fact that real-time voice data collected from robot microphones contains a significant amount of operational noise, such as cooling fan noise and foot contact noise during robot movement. Summary of the Invention

[0004] To address the shortcomings of existing technologies and improve the accuracy of robot voice interaction, this invention adopts the following technical solution:

[0005] A real-time voice interaction method for robots based on spiking neural networks includes the following steps:

[0006] Step 1: Obtain the voice dataset and collect the robot's operating noise;

[0007] Step 2: Add robot running noise to the samples in the speech dataset to generate augmented samples. Preprocess all samples, extract speech signal features, and perform data augmentation.

[0008] Step 3: Construct a recurrent spiking neural network model with recurrent connections, including an input layer, a hidden layer, and an output layer. The hidden layer and the output layer both adopt a recurrent leaky integral firing layer structure. The recurrent leaky integral firing layer includes a spiking firing layer and a dense feedback layer. The spiking firing layer uses a set of leaky integral firing models, and the dense feedback layer uses a set of dense neuron models. After the input features at one time step are calculated by the spiking firing layer, the output spiking is added element by element to the feature input of the next time step after passing through the dense feedback layer.

[0009] Step 4: Generate target regression labels for the training set and construct a loss function, then backpropagate the error along the time dimension to train the recurrent spiking neural network model;

[0010] Step 5: Obtain real-time voice stream input from the robot's microphone and extract features, input it into the trained recurrent spiking neural network model for inference, and control the robot to perform interactive actions based on the pulse decoding results.

[0011] Furthermore, step two includes the following steps:

[0012] Step 2.1: Traverse the central data of the speech dataset, randomly extract segments of equal length to the current data from the running noise, randomly select the target noise level within the set noise range, scale the noise segments to the target noise level, and add them element-wise with the speech command data to generate an augmented sample carrying noise.

[0013] Step 2.2: Preprocess the original and expanded samples. First, resample the samples at the set sampling rate and normalize the length of each sample. Then, pre-emphasize the high-frequency components using a first-order high-pass filter. Divide the signal into short-time frames and apply a window function to reduce boundary effects. Perform a fast Fourier transform on each windowed frame to calculate the power spectrum. Map the frequency domain information to the Mel frequency scale using a Mel filter bank. Perform a logarithmic transform on the filtered energy value to obtain the logarithmic energy spectrum. Finally, perform a discrete cosine transform on the logarithmic energy spectrum to obtain the Mel frequency cepstral coefficients.

[0014] Step 2.3: Based on the Mel frequency cepstral coefficients, data augmentation is performed through random temporal masking and frequency masking, and the dataset is divided into training and test sets.

[0015] Furthermore, in step 2.2, if the current sample exceeds the threshold of the number of sample points, then both ends are truncated to the same length; otherwise, zero padding is performed at the head to achieve tail alignment of the voice command.

[0016] Furthermore, in step three, the constructed recurrent spiking neural network model with recurrent connections has an input layer for receiving speech features, a hidden layer for learning high-level feature representations of the input data, and an output layer for classifying speech data. One neuron represents one class of speech data. For each input sample, the output layer neuron with the most cumulative pulse firings within the simulation period represents the predicted class.

[0017] Furthermore, the neuron membrane potential formula for the leakage integral firing model in step three is as follows:

[0018] U[t+1]=βU[t]+I in [t+1]+V(S out [t])-U thr

[0019]

[0020] Where U[t+1] represents the neuron's membrane potential at time t+1, β represents the membrane potential decay rate, U[t] represents the neuron's membrane potential at time t, and I in [t+1] represents the input current at time t+1, V(·) represents the dense feedback layer, and S out [t] represents the pulse emitted at time t, U thr This represents the neuronal membrane potential threshold.

[0021] Furthermore, step four includes the following steps:

[0022] Step 4.1: Set the target impulse firing rate for the correct and incorrect categories, and generate target regression labels for the training set, where each element corresponds to the number of target impulses fired by the output layer neuron in one simulation cycle;

[0023] Step 4.2: Construct the mean square error loss function based on the output layer pulse count;

[0024] Step 4.3: Replace the pulse firing step function with a gradient substitution function, and propagate the error backward along the time dimension based on the chain rule, that is, the gradient flows backward from the final output loss of the recurrent spiking neural network to the trainable parameters.

[0025] Step 4.4: Train the recurrent spiking neural network on the training set and save the trained model structure and weights.

[0026] Furthermore, in step 4.1, the generated tags are as follows:

[0027]

[0028] Among them, y i Represents the target regression label, rneg r represents the target pulse firing rate of the incorrect category. pos The target pulse firing rate represents the correct category, and T represents the simulation cycle time step.

[0029] Furthermore, the loss function formula in step 4.2 is as follows:

[0030]

[0031] Where n represents the number of categories, i.e., the number of neurons in the output layer. This represents the prediction result, i.e., the pulse firing result of the output layer neurons within the simulation period.

[0032] Furthermore, step five includes the following steps:

[0033] Step 5.1: The robot acquires real-time speech blocks and adds them to the detection queue. If the data in the sliding window has not reached the target length, it continues reading; if the target length is reached, it extracts the MFCC features of the speech data in the window and moves the window forward.

[0034] Step 5.2: Input the extracted features into the trained recurrent spiking neural network model, perform feedforward inference, record the output layer pulse firing sequence until the set feedforward time step is reached, and input the accumulated pulse firing data into the message queue to be decoded;

[0035] Step 5.3: Read the real-time category sensitivity offset and the pulse sequence at the head of the message queue to be decoded. Add the category sensitivity offset to the pulse sequence to be decoded element by element, and select the neuron category with the most pulse firings as the decoding result.

[0036] Step 5.4: Based on the decoding results, control the quadruped robot to perform preset action interactions.

[0037] A real-time voice interaction system for robots based on spiking neural networks includes a data acquisition module, a data preprocessing and expansion module, a network construction module, a network training and storage module, and a real-time perception and interaction module. According to the aforementioned real-time voice interaction method for robots based on spiking neural networks, the data acquisition module acquires a voice dataset and robot operating noise. After data preprocessing and expansion, a training dataset is obtained. The network construction module constructs a recurrent spiking neural network model, which is then trained by the network training and storage module using the training dataset. The real-time perception and interaction module acquires real-time voice streams from the robot, generates an output pulse sequence using the trained recurrent spiking neural network model, and controls the robot to perform interactive actions based on the decoding result of this sequence.

[0038] The advantages and beneficial effects of this invention are as follows:

[0039] This invention employs regression labeling on the training data during the training process of a spiking neural network. Specifically, it sets separate pulse firing rates for correct and incorrect categories. Compared to the one-hot labeling commonly used in traditional multi-class classification tasks, this regression labeling encourages neurons in the output layer that produce incorrect categories, as well as neurons in the corresponding feedforward paths, to fire a small number of pulses during training, effectively preventing the "dead neuron" problem and network overfitting. Furthermore, quadruped robot noise is introduced into the training data to ensure consistency between model training and real-time voice interaction, mitigating the impact of runtime noise. Finally, this invention reduces interaction power consumption and allows for real-time adjustment of the sensitivity of voice interaction commands during runtime. Attached Figure Description

[0040] Figure 1 This is a flowchart of a real-time voice interaction method for robots based on a spiking neural network, as described in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the structure of the cyclic leakage integral distribution layer in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the structure of the recurrent spiking neural network model in an embodiment of the present invention.

[0043] Figure 4 This is a flowchart illustrating real-time voice interaction performed by a robot based on a trained spiking neural network, as described in an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram of the structure of a real-time voice interaction system for robots based on a spiking neural network, according to an embodiment of the present invention. Detailed Implementation

[0045] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0046] like Figure 1As shown, this invention discloses a real-time voice interaction method for quadruped robots based on spiking neural networks. Compared to traditional artificial neural networks, spiking neural networks provide higher biological realism by simulating the spiking mechanism of biological neurons, especially excelling in processing time-series data. Simultaneously, the event-driven computation, sparse spiking activity, and asynchronous processing of spiking neural networks endow them with high efficiency and low power consumption, making them suitable for resource-constrained platforms such as mobile robots. Therefore, using robot perception algorithms based on spiking neural networks not only addresses the shortcomings of current traditional neural networks but also endows robots with more intelligent and adaptive capabilities, specifically including the following steps:

[0047] Step 1: Obtain an open-source speech dataset and collect noise from the robot's operation, specifically including:

[0048] We obtained a portion of the Google Speech Commands v2 dataset, totaling 8000 samples across 8 categories: UP, DOWN, LEFT, RIGHT, GO, STOP, YES, and NO. Robot operating noise was collected from the quadruped robot's microphone over a 10-minute period, including cooling fan noise and foot contact noise during movement.

[0049] Step 2: Add robot running noise to the samples in the open-source speech dataset to generate augmented samples. Preprocess all samples to extract speech signal features and perform data augmentation, specifically including:

[0050] Step 2.1: Traverse the selected data in the open source speech dataset, randomly extract segments of the same length as the current data from the running noise, and randomly select a target noise level within the set noise range (0.2-0.4). Scale the noise segments to the target noise level and add them element-wise with the speech command data to generate an augmented sample carrying noise.

[0051] Step 2.2: Preprocessing the original and augmented samples. First, resample the samples at a sampling rate of 16000Hz. Normalize the length of each sample. If the current sample exceeds 16000 sample points, truncate both ends to the same length; otherwise, perform zero-padding at the beginning to align the speech command to the end. All normalized data contains 16000 sample points, equivalent to 1 second. Next, pre-emphasize the signal using a first-order high-pass filter to enhance high-frequency components. Divide the signal into short frames of 32 milliseconds each and apply a window function to reduce boundary conditions. The effect is that the interval between the frame headers of adjacent frames is 16 milliseconds, each frame has 0.032*16000=512 sampling points, and a complete sample contains 63 frames; a fast Fourier transform is performed on each windowed frame to calculate the power spectrum; the frequency domain information is mapped to the Mel frequency scale through the Mel filter bank, and the filtered energy value is logarithmically transformed; finally, a discrete cosine transform is performed on the logarithmic energy spectrum to obtain the Mel frequency cepstral coefficients, and the first to 20th coefficients are taken as the features of a single speech sample with a shape of (63,20).

[0052] Step 2.3: Perform random time masking and frequency masking on the obtained Mel frequency cepstral coefficient data to augment the data, and divide it into training set and test set in a 9:1 ratio.

[0053] Step 3: Construct a spiking neural network model with recurrent connections, specifically including:

[0054] Step 3.1: As Figure 2 As shown, the cyclic leaky integral firing layer used to construct the spiking neural network model with recurrent connections consists of a spiking firing layer and a dense feedback layer. The spiking firing layer adopts the leaky integral firing model, and the dense feedback layer adopts the dense neuron model. After the input features of one time step are calculated by the spiking firing layer, the output pulses are added element by element to the input of the next time step after passing through the dense feedback layer.

[0055] like Figure 3As shown, the network structure of the spiking neural network model with recurrent connections includes one input layer, N(2) hidden layers, and one output layer. Both the hidden and output layers adopt a recurrent leak integral firing layer structure, and the layers are fully connected. The number of neurons in the input layer is 1260, which is consistent with the feature dimension of a single preprocessed speech sample; the number of neurons in the first hidden layer is 256; the number of neurons in the second hidden layer is 128; the number of neurons in the output layer is 8; the dense feedback layer in the recurrent leak integral firing layer adopts dense neurons, and the number is consistent with the number of the connected hidden layers. The input layer is used to receive speech features; the hidden layer is used to learn the high-level feature representation of the input data; the output layer is used to classify speech data, and one neuron represents one class of speech data. The simulation period is set to 8 time steps. For each input sample, the output layer neuron with the most cumulative pulse firings in the simulation period represents the predicted class.

[0056] Step 3.2: The leak integral firing model is used in both the hidden and output layers of the model. The neuron membrane potential is calculated using the following formula:

[0057] U[t+1]=βU[t]+I in [t+1]+V(S out [t])-U thr

[0058]

[0059] Where U[t+1] represents the neuron's membrane potential at time t+1, β represents the membrane potential decay rate, U[t] represents the neuron's membrane potential at time t, and I in [t+1] represents the input current at time t+1, V(·) represents the dense feedback layer, and S out [t] represents the pulse emitted at time t, U thr This represents the neuron's membrane potential threshold. In this embodiment, β is 0.9, U thr It is 1.0.

[0060] Step 4: Generate training set target regression labels and construct a loss function. Perform error backpropagation along the time dimension to train the recurrent spiking neural network and save the model structure and weights. Specifically, this includes:

[0061] Step 4.1: Set the target impulse firing rates for correct and incorrect categories, and generate target regression labels for the training set, where each element corresponds to the number of target impulses fired by the output layer neuron in one simulation cycle. The generated labels are as follows:

[0062]

[0063] Among them, y iRepresents the target regression label, r neg r represents the target pulse firing rate of the incorrect category. pos The target pulse firing rate represents the correct category, and T represents the simulation cycle time steps. In this embodiment, r neg For 0.2, r pos The value is 0.8, and T is 8.

[0064] Step 4.2: Construct the mean square error loss function based on the output layer pulse count. The calculation formula is as follows:

[0065]

[0066] Where n represents the number of categories, i.e., the number of neurons in the output layer; in this embodiment, n is 8. This represents the prediction result, i.e., the pulse firing result of the output layer neurons within the simulation period.

[0067] Step 4.3: Replace the pulse firing step function with a gradient substitution function, and propagate the error backward along the time dimension based on the chain rule. That is, the gradient flows backward from the final output loss of the recurrent spiking neural network to the trainable parameters. In this embodiment, the arctangent function is used as the gradient substitution function.

[0068] Step 4.4: Train the recurrent spiking neural network on the training set. In this embodiment, the batch training method is used, with the batch size set to 128, the learning rate set to 0.0005, and the optimizer set to Adam. Each batch is input into the network for 8 time steps of feedforward calculation. The error is calculated according to the loss function, and the error is backpropagated along the time dimension to train the recurrent spiking neural network. After 20 rounds of training, the trained model structure and weights are saved.

[0069] Step 5: As Figure 4 As shown, real-time voice stream input is acquired from the robot's microphone, and features are extracted and input into a recurrent spiking neural network model for inference. Based on the pulse decoding results, the robot is controlled to perform interactive actions, specifically including:

[0070] Step 5.1: Read real-time speech blocks from the microphone and add them to the detection queue. If the data within the sliding window has not reached the target length, continue reading; if the target length has been reached, extract the MFCC features of the speech data within the window and move the window forward. In this embodiment, the speech block length is 16 milliseconds, the target length of the sliding window is 63 speech blocks, and the step size of moving the window forward is 1 speech block.

[0071] Step 5.2: Input the extracted features into the network, perform feedforward inference, and record the output layer pulse emission sequence until the set feedforward time step is reached. Then, input the accumulated pulse emission data into the message queue to be decoded. In this embodiment, the feedforward time step is set to 8.

[0072] Step 5.3: Read the real-time category sensitivity offset and the pulse sequence at the head of the message queue to be decoded. Add the category sensitivity offset to the pulse sequence to be decoded element by element, and select the neuron category with the most pulse firings as the decoding result. In this embodiment, the initial category sensitivity offset is all 0.

[0073] Step 5.4: Based on the decoding results, control the quadruped robot to perform preset action interactions. In this embodiment, the preset actions include standing up, sitting down, walking to the left, walking to the right, walking forward, resting, nodding, and shaking the head, which correspond to UP, DOWN, LEFT, RIGHT, GO, STOP, YES, and NO in the training data, respectively.

[0074] like Figure 5 As shown, the present invention also discloses a real-time voice interaction system for quadruped robots based on spiking neural networks, including a data acquisition module, a data preprocessing and expansion module, a network construction module, a network training and storage module, and a real-time perception and interaction module.

[0075] The data acquisition module is used to acquire open-source speech datasets and collect robot operating noise.

[0076] The data preprocessing and augmentation module is used to overlay robot running noise onto samples from the open-source speech dataset in the data acquisition module to generate augmented samples, preprocess all samples, extract MFCC speech signal features, perform data augmentation, and generate training data.

[0077] Specifically, the samples in the open-source speech dataset are traversed, and segments of the same length as the sample are randomly extracted from the running noise. Within a set noise range, a target noise level is randomly selected, and the noise segments are scaled to the target noise level. The noise segments are then added element-wise with the speech command data to generate an augmented sample carrying noise.

[0078] Preprocessing of the original and augmented samples: First, the samples are resampled at a set sampling rate, and each sample is truncated at both ends or zero-padded at the beginning to normalize its length. Next, a first-order high-pass filter is used for pre-emphasis to enhance high-frequency components. The signal is divided into short-time frames, and a window function is applied to reduce boundary effects. A Fast Fourier Transform is performed on each windowed frame to calculate the power spectrum. The frequency domain information is mapped to the Mel frequency scale using a Mel filter bank, and the filtered energy value undergoes a logarithmic transform. Finally, a Discrete Cosine Transform is performed on the logarithmic energy spectrum to obtain the Mel frequency cepstral coefficients. The obtained Mel frequency cepstral coefficient data is then augmented using random time masking and frequency masking, and divided into training and test sets.

[0079] The network building module is used to construct a recurrent spiking neural network (RON). It contains one input layer, N hidden layers, and one output layer. Fully connected layers connect to each other, and recurrent connections are used within each layer. The input layer receives speech features; the hidden layers learn high-level feature representations of the input data; and the output layer classifies the speech data. Each neuron represents a class of speech data. For each input sample, the output neuron with the highest cumulative number of pulse firings within the simulation time step represents the predicted class. The model uses a leaky integral firing model within both the hidden and output layers. The neuron membrane potential is calculated using the following formula:

[0080] U[t+1]=βU[t]+I in [t+1]+V(S out [t])-U thr

[0081]

[0082] Where U[t+1] is the neuron's membrane potential at time t+1, β is the membrane potential decay rate, and U[t] is the neuron's membrane potential at time t. in [t+1] is the input current at time t+1, V(·) is the dense feedback layer, and S out [t] is the pulse emitted at time t, U thr It is the neuronal membrane potential threshold.

[0083] The network training and storage module generates target regression labels for the training set by setting the target pulse firing rates for the correct and incorrect categories, constructs a mean squared error loss function based on the pulse count of the output layer, replaces the pulse firing step function with a gradient substitution function, and trains the recurrent spiking neural network by backpropagating the error along the time dimension based on the chain rule, and saves the model structure and weights.

[0084] Specifically, the firing rates of target impulses for correct and incorrect categories are set, and target regression labels for the training set are generated, where each element corresponds to the number of target impulses fired by the output layer neuron in one simulation cycle. The generated labels are as follows:

[0085]

[0086] Among them, y i It is the target regression label, r neg The target pulse firing rate, r, is the incorrect category. pos is the target pulse firing rate of the correct category, and T is the simulation cycle time step.

[0087] The mean square error loss function based on the output layer pulse count is constructed, and the calculation formula is as follows:

[0088]

[0089] Where n is the number of categories, i.e., the number of neurons in the output layer. It is the prediction result, that is, the pulse firing result of the output layer neurons within the simulation cycle.

[0090] The gradient substitution function is used instead of the pulse firing step function, and the error is backpropagated along the time dimension based on the chain rule. That is, the gradient flows backward from the final output loss of the recurrent spiking neural network to the trainable parameters.

[0091] Train a recurrent spiking neural network on the training set and save the trained model structure and weights.

[0092] The real-time perception and interaction module is used to acquire real-time voice stream input from the robot's microphone and extract features, which are then input into the recurrent spiking neural network model for inference. Based on the pulse decoding results, the module controls the robot to perform interactive actions.

[0093] Specifically, each time a speech block is read from the microphone, a sliding window is used to maintain the latest speech blocks. The total duration of the data in the sliding window is 1 second. The speech data in the window is pre-emphasized through a first-order high-pass filter to enhance the high-frequency components. It is divided into short-time frames and a window function is applied. A fast Fourier transform is performed on each windowed frame to calculate the power spectrum. The frequency domain information is mapped to the Mel frequency scale through a Mel filter bank. The filtered energy value is logarithmically transformed, and then a discrete cosine transform is performed on the logarithmic energy spectrum to obtain the Mel frequency cepstral coefficients. The obtained Mel frequency cepstral coefficients are normalized in the same way as those on the training set.

[0094] The preprocessed speech features are input into the stored recurrent spiking neural network for inference, the output layer pulse firing sequence is recorded, and the result is input into the message queue to be decoded.

[0095] Read the real-time category sensitivity offset and the pulse sequence at the head of the message queue to be decoded. Add the category sensitivity offset to the pulse sequence to be decoded element by element, and select the neuron category with the most pulse firings as the decoding result.

[0096] Based on the decoding results, control the quadruped robot to perform preset action interactions.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time voice interaction method for robots based on spiking neural networks, characterized in that... Includes the following steps: Step 1: Obtain the voice dataset and collect the robot's operating noise; Step 2: Overlay robot running noise onto the samples in the speech dataset to generate augmented samples, extract speech signal features from the samples, and perform data augmentation. Step 3: Construct a recurrent spiking neural network model with recurrent connections, including an input layer, a hidden layer, and an output layer. The hidden layer and the output layer both adopt a recurrent leaky integral firing layer structure. The recurrent leaky integral firing layer includes a spiking firing layer and a dense feedback layer. The spiking firing layer uses a set of leaky integral firing models, and the dense feedback layer uses a set of dense neuron models. After the input features at one time step are calculated by the spiking firing layer, the output spiking is added element by element to the feature input of the next time step after passing through the dense feedback layer. Step 4: Generate target regression labels for the training set and construct a loss function, then backpropagate the error along the time dimension to train the recurrent spiking neural network model; Step 5: The robot acquires real-time voice stream input and extracts features, inputs them into the trained recurrent spiking neural network model for inference, and controls the robot to perform interactive actions based on the spiking decoding results.

2. The real-time voice interaction method for robots based on spiking neural networks according to claim 1, characterized in that: Step two includes the following steps: Step 2.1: Traverse the central data of the speech dataset, randomly extract segments of equal length to the current data from the running noise, randomly select the target noise level within the set noise range, scale the noise segments to the target noise level, and add them element-wise with the speech command data to generate an augmented sample carrying noise. Step 2.2: Preprocess the original and expanded samples. First, resample the samples at the set sampling rate and normalize the length of each sample. Then, pre-emphasize the signal using a first-order high-pass filter, divide the signal into short-time frames, apply a window function, perform a fast Fourier transform on each windowed frame, calculate the power spectrum, map the frequency domain information onto the Mel frequency scale using a Mel filter bank, perform a logarithmic transform on the filtered energy value to obtain the logarithmic energy spectrum, and finally perform a discrete cosine transform on the logarithmic energy spectrum to obtain the Mel frequency cepstral coefficients. Step 2.3: Based on the Mel frequency cepstral coefficients, data augmentation is performed through random temporal masking and frequency masking, and the dataset is divided into training and test sets.

3. The real-time voice interaction method for robots based on spiking neural networks according to claim 2, characterized in that: In step 2.2, if the current sample exceeds the threshold of the number of sample points, then both ends are truncated to the same length; otherwise, zero padding is performed at the head to achieve tail alignment of the voice command.

4. The real-time voice interaction method for robots based on spiking neural networks according to claim 1, characterized in that: In step three, the constructed recurrent spiking neural network model with recurrent connections has an input layer for receiving speech features, a hidden layer for learning high-level feature representations of the input data, and an output layer for classifying speech data. One neuron represents one class of speech data. For each input sample, the output layer neuron with the most cumulative pulse firings within the simulation period represents the predicted class.

5. The real-time voice interaction method for robots based on spiking neural networks according to claim 1, characterized in that: The neuron membrane potential formula for the leakage integral firing model in step three is as follows: U[t+1]=βU[t]+I in [t+1]+V(S out [t])-U thr Where U[t+1] represents the neuron's membrane potential at time t+1, β represents the membrane potential decay rate, U[t] represents the neuron's membrane potential at time t, and I in [t+1] represents the input current at time t+1, V(·) represents the dense feedback layer, and S out [t] represents the pulse emitted at time t, U thr This represents the neuronal membrane potential threshold.

6. The real-time voice interaction method for robots based on spiking neural networks according to claim 1, characterized in that: Step four includes the following steps: Step 4.1: Set the target impulse firing rate for the correct and incorrect categories, and generate target regression labels for the training set, where each element corresponds to the number of target impulses fired by the output layer neuron in one simulation cycle; Step 4.2: Construct the mean square error loss function based on the output layer pulse count; Step 4.3: Replace the pulse firing step function with a gradient substitution function, and propagate the error backward along the time dimension based on the chain rule, that is, the gradient flows backward from the final output loss of the recurrent spiking neural network to the trainable parameters. Step 4.4: Train the recurrent spiking neural network on the training set and save the trained model structure and weights.

7. A real-time voice interaction method for robots based on a spiking neural network according to claim 6, characterized in that: In step 4.1, the generated tags are as follows: Among them, y i Represents the target regression label, r neg r represents the target pulse firing rate of the incorrect category. pos The target pulse firing rate represents the correct category, and T represents the simulation cycle time step.

8. A real-time voice interaction method for robots based on a spiking neural network according to claim 6, characterized in that: The loss function formula in step 4.2 is as follows: Where n represents the number of categories, i.e., the number of neurons in the output layer. This represents the prediction result, i.e., the pulse firing result of the output layer neurons within the simulation period.

9. A real-time voice interaction method for robots based on a spiking neural network according to claim 1, characterized in that: Step five includes the following steps: Step 5.1: The robot acquires real-time speech blocks and adds them to the detection queue. If the data in the sliding window has not reached the target length, it continues reading; if the target length is reached, it extracts the features of the speech data in the window and moves the window forward. Step 5.2: Input the extracted features into the trained recurrent spiking neural network model, perform feedforward inference, record the output layer pulse firing sequence until the set feedforward time step is reached, and input the accumulated pulse firing data into the message queue to be decoded; Step 5.3: Read the real-time category sensitivity offset and the pulse sequence at the head of the message queue to be decoded. Add the category sensitivity offset to the pulse sequence to be decoded element by element, and select the neuron category with the most pulse firings as the decoding result. Step 5.4: Based on the decoding results, control the robot to perform preset action interactions.

10. A real-time voice interaction system for robots based on spiking neural networks, comprising a data acquisition module, a data preprocessing and expansion module, a network construction module, a network training and storage module, and a real-time perception and interaction module, characterized in that: According to any one of claims 1 to 9, a real-time voice interaction method for robots based on spiking neural networks is described, wherein the data acquisition module acquires a voice dataset and robot operating noise, and the data preprocessing and expansion module obtains a training dataset; the network construction module is used to construct a recurrent spiking neural network model, which is trained by the network training and storage module using the training dataset; the real-time perception and interaction module acquires a real-time voice stream from the robot, generates an output pulse sequence through the trained recurrent spiking neural network model, and controls the robot to perform interactive actions based on the decoding result of the sequence.

Images