Speech recognition system for robots based on deep learning

Abstract

The application relates to the field of speech recognition, and discloses a speech recognition system for robots based on deep learning, which comprises a voice input module for collecting and denoising voice signals; a feature extraction module for converting the voice signals into a mel spectrum graph or MFCC features; a deep learning model module based on a Transformer-XL architecture, which processes input features through a multi-head self-attention mechanism and captures context information in combination with a cross-time step memory unit; an optimal memory management module for dynamically planning and optimizing a memory refresh strategy and dynamically adjusting a memory state; an adaptive attention mechanism module for adaptively adjusting the weight of an attention head based on a Lagrange multiplier method; and an output module for converting model output into text or robot control instructions. The method improves the accuracy and robustness of speech recognition, enhances the processing capacity of the system in long conversations and complex contexts, and is suitable for application scenarios such as robot control and home management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a speech recognition system for robots based on deep learning. BACKGROUND

[0002] With the development of artificial intelligence and robotics, speech recognition technology has become one of the important ways for robots to interact with humans. Existing robot speech recognition systems are usually based on traditional deep learning models, such as recurrent neural networks (RNN) or long short-term memory networks (LSTM). These models perform well in processing short voice commands or simple conversations. However, as voice interaction scenarios become more complex, especially in handling long conversations, continuous voice commands or complex contexts, these models often face the limitation of being unable to effectively capture long-distance dependencies.

[0003] In complex conversation scenarios, traditional speech recognition models have difficulty maintaining accurate memory of context. As the time steps progress, early voice information may gradually be forgotten, causing the model to fail to correctly understand the meaning of subsequent voice input. In addition, existing models use a fixed attention mechanism, which cannot dynamically adjust attention allocation based on input, resulting in some important voice information not being fully focused on, while irrelevant background information is over-processed, affecting the accuracy and efficiency of recognition.

[0004] For long conversations, the memory mechanism of traditional models lacks effective management strategies, causing accumulated redundant information to interfere with the processing of current input, while excessive refreshing of memory state loses key context information. This deficiency in memory management results in suboptimal performance of the system in continuous instructions and complex contexts.

[0005] In addition, although existing technologies can convert voice signals into text or control instructions, in practical applications, how to effectively handle noise in voice, quickly extract efficient features, and achieve real-time speech recognition in complex environments remains a technical problem to be solved. SUMMARY

[0006] To address the deficiencies of the prior art, the present application provides a speech recognition system for robots based on deep learning, which solves the problems of insufficient context capture, memory management and dynamic attention allocation of existing robot speech recognition systems in long conversations and complex contexts, improving the accuracy and processing efficiency of recognition.

[0007] To achieve the above purpose, the present application is implemented by the following technical scheme: a speech recognition system for robots based on deep learning, comprising:

[0008] The speech input module is used to collect external speech signals and pre-process them to generate denoised speech signals;

[0009] A feature extraction module, configured to convert the preprocessed speech signal into a feature vector sequence;

[0010] a deep learning model module, comprising a multi-layer self-attention mechanism and a cross-timestep memory unit based on a Transformer-XL architecture, wherein the deep learning model module generates a speech recognition result based on the feature vector sequence;

[0011] An optimal memory management module, configured to dynamically manage the memory units in the deep learning model module and determine a memory refresh strategy for each time step based on a preset cost function;

[0012] An adaptive attention mechanism module, configured to adaptively adjust the weight of each attention head in the multi-layer self-attention mechanism according to an input feature vector sequence;

[0013] The output module is used to convert the speech recognition results generated by the deep learning model module into text or control instruction output.

[0014] Preferably, the feature extraction module includes:

[0015] Mel-spectrogram generation unit, which is used to convert the input speech signal into the frequency domain through fast Fourier transform to obtain the speech signal spectrogram, and then process the spectrogram through a series of triangular filters to map the frequency components to the Mel scale and generate the corresponding Mel-spectrogram as the feature vector;

[0016] A cepstral feature extraction unit is used to perform discrete cosine transform based on the mel-frequency spectrum, extract cepstral coefficients, and generate mel-frequency cepstral coefficients as input features of the deep learning model;

[0017] an energy normalization unit, configured to perform energy normalization processing on the feature vector to ensure that speech signals of different volumes have a consistent amplitude range in the feature space;

[0018] A feature vector sequence generating unit is used to arrange the Mel-frequency spectrum and Mel-frequency cepstral coefficient features into a feature vector sequence X={x1, x2, ..., x T}, where x t Represents the feature vector at time step t, which is input into the deep learning model module for subsequent processing.

[0019] Preferably, the deep learning model module includes:

[0020] a Transformer-XL modeling unit, configured to model long-time context information of the voice input based on the input sequence of feature vectors X = {x1, x2,..., xT} and construct a deep learning model, wherein the Transformer-XL modeling unit is implemented by a multi-layer stacked self-attention mechanism and a feedforward neural network, and combines a cross-time step memory unit to model long-time context information of the voice input; T} construct a deep learning model, wherein the Transformer-XL modeling unit is implemented by a multi-layer stacked self-attention mechanism and a feedforward neural network, and combines a cross-time step memory unit to model long-time context information of the voice input;

[0021] a multi-head self-attention calculation unit, configured to perform self-attention calculation on the input sequence of feature vectors;

[0022] a cross-time step memory unit, configured to pass and retain a memory state s t of a previous time step t-1 in combination with a current input x t-1 to enable the model to utilize information of previous time steps at a current time step t to implement context understanding of long-time dialog and processing of voice continuity; t

[0023] a residual connection and normalization unit, configured to perform residual connection and normalization processing on outputs of the self-attention calculation unit and the feedforward neural network unit, and specifically include adding the input and the output through residual connection and performing layer normalization processing on a result to obtain a final output of the deep learning model module.

[0024] Preferably, the multi-head self-attention calculation unit specifically includes the following steps:

[0025] obtaining a query matrix Q, a key matrix K and a value matrix V through linear transformation, wherein Q = XW Q , K = XW K and V = XW V , W Q , W K and W V are trainable weight matrices;

[0026] calculating an attention weight of each attention head, wherein d k is a dimension of the key matrix K;

[0027] splicing outputs of all attention heads to form an output MultiHead(Q, K, V) = Concat(head1, head2,..., head h )W O of the multi-head attention, wherein head i is an output of the i-th attention head, and W O is a linear transformation matrix.

[0028] Preferably, the optimal memory management module includes:​

[0029] State monitoring unit, used to monitor the memory state s at each time step t t , where the memory state s t represents the context information retained by the deep learning model at time step t;

[0030] Action decision unit, used to make decisions based on the current memory state s at each time step t t And the preset cost function g(s t , a t ) Determine memory refresh action a t , where a t Indicates whether to perform memory refresh at time step t. The cost function consists of the following two parts:

[0031] Noise cost caused by memory retention t ), indicating the memory state s t Redundant information may be introduced as the time step increases;

[0032] The loss cost of refreshing memory Loss(a t ), indicating key information that may be lost when refreshing the memory;

[0033] The dynamic programming solver is used to solve the memory refresh strategy based on the dynamic programming method and determine the optimal action at each time step by calculating the following Bellman equation

[0034]

[0035] Among them, V(s t ) represents the value function of the current time step, γ is the discount factor, which reflects the impact of future time steps on the current decision;

[0036] A strategy updating unit is used to update the optimal action at each time step t according to the result of the dynamic programming solving unit. Dynamically adjust the memory state and perform memory retention or refresh operations to ensure that the model can adaptively adjust memory management according to the current context and maximize context retention and processing efficiency during speech recognition.

[0037] Preferably, the adaptive attention mechanism module includes:

[0038] Weight initialization unit, used to initialize the weight α of each attention head in the multi-head self-attention mechanism at each time step t i , where i∈{1,2,...,h}, h is the number of attention heads, and the initial weights satisfy the constraints And α i ≥0;

[0039] Attention loss calculation unit, used to calculate each attention head i Processing input feature x at time step t t The loss value f i (x t ), the loss value f i (x t ) is used to measure the performance of the attention head, indicating the error or mismatch degree when the attention head processes the speech features at the current time step;

[0040] The objective function construction unit is used to construct the objective function of adaptive attention optimization based on the Lagrange multiplier method. The specific form is:

[0041]

[0042] Among them, λ is the Lagrange multiplier, which is used to ensure that the sum of the weights of each attention head is 1;

[0043] The weight update unit is used to optimize the objective function by gradient descent and calculate the weight α of each attention head i Gradient:

[0044]

[0045] And gradually optimize the weights through the following update rules:

[0046]

[0047] Among them, η is the learning rate, and through multiple iterations of optimization, the weight of each attention head α i Adaptive adjustment to ensure reasonable allocation of attention resources within the current time step t;

[0048] The weight normalization unit is used to update the weight α after each iteration. i Normalization is performed to ensure that the sum of the weights of all attention heads is still 1 to meet the constraints of the adaptive attention mechanism.

[0049] Preferably, the output module includes:

[0050] A text conversion unit, configured to convert the speech recognition results generated by the deep learning model module into text output;

[0051] The command generation unit is used to convert the generated text information into robot control commands.

[0052] Preferably, the text conversion unit specifically includes the following steps:

[0053] Receive the processed feature vector sequence y={y1,y2,...,y T};

[0054] Use the classification layer to classify each time step y t Decode the feature vector y t Mapping is done to the characters or words corresponding to speech recognition, and these characters or words are concatenated into continuous text.

[0055] Preferably, the command generation unit specifically includes the following steps:

[0056] Analyze the control commands or instructions contained in the text according to the preset instruction recognition model;

[0057] Map the identified instructions to the corresponding robot control operations to generate a control command sequence C = {c1, c2, ..., c T}, used to control the robot to perform corresponding actions;

[0058] If the text message contains multiple instructions, multiple control commands are generated in sequence according to preset priorities or sequence rules.

[0059] The present invention also provides a robot speech recognition method based on deep learning, comprising the following steps:

[0060] Collect external voice signals and perform noise reduction and sampling processing on them;

[0061] Perform feature extraction on the preprocessed speech signal to generate Mel-spectrogram or MFCC features;

[0062] The features are fed into a deep learning model based on the Transformer-XL architecture, which processes the input using a multi-head self-attention mechanism and captures contextual information using memory units across time steps.

[0063] Optimize memory refresh strategies through dynamic programming methods and dynamically adjust the memory state in deep learning models;

[0064] Adaptively adjust the weight of each attention head based on the Lagrange multiplier method;

[0065] Convert the output of the deep learning model into text or robot control instructions to complete the speech recognition process.

[0066] The present invention provides a robot speech recognition system based on deep learning. It has the following beneficial effects:

[0067] 1、The system can capture the context information in long-time speech sequences by introducing a deep learning model based on the Transformer-XL architecture. This architecture allows the model to handle long-range dependencies in speech signals, solving the problem of poor performance of traditional speech recognition systems in long conversations or complex contexts. In addition, the memory unit across time steps preserves important context, making the system's semantic understanding of different conversation segments more accurate and enhancing the overall robustness of the system.

[0068] 2、The system can dynamically adjust the allocation of attention resources according to the input features by using the Lagrange multiplier method to adaptively adjust the weight of each attention head. This means that the model can better focus on important parts of the speech input and ignore irrelevant background information, improving the accuracy of speech recognition. This adaptive mechanism makes the system more flexible and can handle a variety of different speech environments and instructions.

[0069] 3、The system optimizes the memory refresh strategy through dynamic programming, which can dynamically adjust the memory state in the deep learning model to ensure that redundant information is not accumulated or important context is lost when processing long-time speech input. This memory management method can effectively improve the continuity and semantic coherence of speech recognition, especially in scenarios involving multi-step commands or long conversations.

[0070] 4、The system uses the Mel spectrogram and MFCC feature extraction method to efficiently extract the time-frequency information of the speech signal into a feature vector that the model can process. This feature extraction method not only preserves the core information in the speech, but also reduces the interference of noise and redundant information, allowing the subsequent deep learning model to better understand the speech content and improving the processing efficiency of the entire system. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a schematic diagram of the system architecture of the present invention;

[0072] Figure 2 is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present invention will be described in detail below with reference to the drawings in the specification of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present invention.

[0074] Please refer to the drawings in the specification of the present invention Figure 1The embodiment of the application provides a deep learning-based voice recognition system for a robot, which can realize accurate recognition and real-time execution of a voice instruction of the robot by pre-processing, feature extraction, deep learning model processing, optimal memory management, adaptive attention mechanism and final output processing of voice input. The system is based on the Transformer-XL architecture, utilizes the powerful feature extraction capability of the deep learning model, and realizes efficient processing and accurate recognition of complex voice input through dynamic memory management and attention mechanism, and has high robustness and adaptability.

[0075] The system comprises a voice input module, a feature extraction module, a deep learning model module, an optimal memory management module, an adaptive attention mechanism module and an output module. Specifically as follows:

[0076] Voice input module

[0077] The system collects external voice signals through a robot microphone, converts them into denoised signals after preprocessing, and provides high-quality input for subsequent feature extraction and deep learning model processing. The main task of this module is to preliminarily process voice data from a complex environment to ensure that the voice signal can clearly reflect the user's instruction or conversation content, and remove interference factors to provide stable audio input.

[0078] In this embodiment, the main functions of the voice input module include voice signal collection, denoising processing and signal normalization, and the specific implementation process is as follows:

[0079] 1. Voice signal collection:

[0080] Voice signals from the external environment are collected through a microphone or microphone array in the robot. The audio signal is sampled at a specified sampling rate, so as to capture the frequency details of the voice and ensure high audio quality and voice clarity.

[0081] The sampled signal is denoted as x(n), where n represents the time step, and the audio signal is stored as time series data.

[0082] 2. Denoising processing:

[0083] In this embodiment, in order to improve the quality and recognizability of the voice signal, the voice input module first performs denoising processing on the collected voice signal.

[0084] A low-pass filter is used to remove high-frequency noise by limiting the frequency range of the signal (such as below 3.5 kHz) to ensure that the retained frequency mainly contains the effective part of the human voice signal.

[0085] After that, an adaptive filter is used to suppress the environmental noise. This filter can dynamically adjust its filtering characteristics to adapt to changing background noise conditions, further reducing background noise interference and preserving the main frequency components of the speech.

[0086] 3. Signal normalization:

[0087] In this embodiment, in order to avoid the influence of feature extraction process due to the difference of speech input signal intensity, the system normalizes the processed speech signal.

[0088] The normalization process adjusts the amplitude range of the audio signal to a fixed interval. The common method is to scale the maximum absolute value of the signal to 1, so as to ensure that speech signals of different volumes have consistent amplitudes in subsequent processing.

[0089] 4. Signal framing and window function processing:

[0090] In this embodiment, in order to cooperate with the subsequent feature extraction operation, the input speech signal needs to be processed by framing. Framing means cutting the continuous speech signal into multiple shorter fixed length frames, each frame corresponding to the signal in a time window. Each frame will be used independently for subsequent frequency domain analysis.

[0091] Suppose the user inputs an instruction "open the window" when using the robot system. The robot collects the speech signal containing the instruction through its microphone array, along with some background noise such as wind and other sounds in the environment. The speech input module first digitizes the speech signal through the sampler to form a sample sequence.

[0092] Next, the system removes background noise through a low-pass filter and an adaptive filter. After filtering, the speech signal in the "open the window" instruction is preserved, while the wind in the background is greatly weakened. After that, the system normalizes the processed speech signal to adjust the volume to a uniform range, ensuring that the speech signal can stably enter the next feature extraction process.

[0093] Finally, the system frames the processed speech signal into 25ms time windows, and each frame of signal is processed by a window function to enable the subsequent feature extraction module to better capture the frequency information in the speech.

[0094] Through the speech input module in this embodiment, the system can effectively collect and process the speech signal in the external environment, remove noise and ensure consistent signal strength, providing clear and accurate speech input for subsequent deep learning processing.

[0095] Feature extraction module

[0096] When the system analyzes the preprocessed speech signal, it mainly converts the time-domain signal into the frequency-domain signal to extract efficient representations that can reflect the characteristics of the speech. Feature extraction is an important step in the speech recognition system. By extracting features such as the Mel spectrogram and MFCC (Mel Frequency Cepstral Coefficients), the system can convert the speech signal into a sequence of feature vectors suitable for processing by a deep learning model.

[0097] In this embodiment, the feature extraction module converts the preprocessed speech signal into a sequence of feature vectors, which includes the following steps:

[0098] 1. Framing and windowing processing:

[0099] In this embodiment, in order to facilitate subsequent frequency-domain analysis, the preprocessed normalized speech signal x norm (n) is first divided into multiple short-time frames. Each frame usually corresponds to a time window of about 25 ms, and there is a certain overlap (usually 10 ms) between adjacent frames.

[0100] Each frame contains N sampling points where N is usually set to 400 sampling points (assuming a sampling rate of 16 kHz).

[0101] After framing, in order to reduce the spectral distortion caused by the framing boundary, a Hamming window function is applied to each frame of signal for windowing processing. The Hamming window can reduce the discontinuity at the frame boundary and improve the accuracy of spectral estimation.

[0102] 2. Fast Fourier Transform (FFT):

[0103] In this embodiment, the Fast Fourier Transform (FFT) is performed on each frame of windowed signal to convert the time-domain signal into the frequency-domain signal, thereby obtaining the spectral information of each frame of signal.

[0104] Assuming that the windowed signal of each frame is x win (n), its FFT transform result is X(k), where k is the frequency index, representing the amplitude of each frequency component. The mathematical expression of FFT is:

[0105]

[0106] where N is the number of sampling points per frame, k = 0, 1,..., N-1, representing the index of each frequency component.

[0107] 3. Mel spectrogram generation:

[0108] In this embodiment, in order to better reflect the perception characteristics of human ears to different frequencies, the system converts the FFT result into a mel-spectrogram. Specifically, the frequency domain signal is mapped to the mel frequency scale through a set of mel filters, and the mel scale is more in line with the perception characteristics of human ears, especially the sensitivity to low frequency signals.

[0109] The mel filter bank is set, and the triangular filter is used to perform weighted summation on the spectrum |X(k)| 2 to generate a mel-spectrogram M(f) of each frame of signal, and the mathematical expression is:

[0110]

[0111] where H f (k) is the frequency response of the fth mel filter, k min and k max represent the minimum frequency index and the maximum frequency index of the spectrum, respectively.

[0112] 4. Discrete cosine transform (DCT) and MFCC feature extraction:

[0113] In this embodiment, in order to further compress the dimension of the mel-spectrogram and extract key features that can reflect the speech signal, the system performs discrete cosine transform (DCT) on each frame of mel-spectrogram to obtain mel frequency cepstral coefficients (MFCC).

[0114] The mathematical expression of the discrete cosine transform is:

[0115]

[0116] where C(m) is the mth order cepstral coefficient, F is the number of mel filters, and M(f) is the fth mel-spectrogram value.

[0117] Through DCT transformation, the system can effectively reduce the redundancy of the mel-spectrogram and generate more compact MFCC features representing the frequency information of each frame of speech signal.

[0118] 5. Feature vector sequence construction:

[0119] In this embodiment, the MFCC feature vectors generated for each frame are arranged in time sequence to form a feature vector sequence X = {x1, x2,..., x T} of the entire speech segment, where x t represents the MFCC feature vector of the tth frame. The sequence will be input to the deep learning model module for further processing.

[0120] Suppose a robot voice system receives an audio signal containing the user's command: "Turn off the lights." First, the system uses the voice input module to collect and preprocess the audio signal of the command. Then, the feature extraction module begins processing the preprocessed signal. The system first segments the audio signal into 25ms frames and applies a Hamming window to each frame to eliminate discontinuities introduced by frame boundaries.

[0121] Next, the system performs a fast Fourier transform on each frame, converting the time-domain signal into the frequency domain and generating the spectral information for each frame. A Mel filter bank is then used to map the spectral information to the Mel scale, generating a Mel-scale spectrogram for each frame. The system then compresses the Mel-scale spectrogram into MFCC features using a discrete cosine transform, generating a feature vector for each frame.

[0122] Finally, the MFCC feature vectors of all frames are arranged in chronological order to form a feature vector sequence of the speech segment. The system inputs this feature vector sequence into the deep learning model module for speech recognition processing.

[0123] Through the feature extraction module in this embodiment, the system can extract feature vectors with good recognition from the original speech signal, ensuring the speech processing effect of the subsequent deep learning model.

[0124] Deep learning model module

[0125] The system uses a multi-layer self-attention mechanism based on the Transformer-XL architecture and memory units across time steps to deeply process the feature vector sequence of the speech signal. This multi-head self-attention mechanism enables the system to capture long-term dependencies in the input sequence, and the memory units retain information from previous time steps to ensure effective modeling of the context of long-term conversations.

[0126] In this embodiment, the deep learning model module implements deep processing of speech features based on the Transformer-XL architecture, specifically including the following steps:

[0127] 1. Feature vector sequence input:

[0128] In this embodiment, the feature vector sequence X generated by the feature extraction module is {x1, x2, ..., x T} is input to the deep learning model module. Each feature vector x t Corresponding to the MFCC features of the t-th frame in the speech signal, the feature sequence is an arrangement of time steps, reflecting the time-frequency characteristics of the speech signal.

[0129] 2. Multi-head self-attention mechanism:

[0130] In this embodiment, the deep learning model module processes the input feature vector sequence through a multi-head self-attention mechanism. The multi-head self-attention mechanism allows the model to capture information from different perspectives when processing the sequence and weights each time step in the sequence to highlight important information.

[0131] First, the query matrix Q, the key matrix K, and the value matrix V are generated through a linear transformation of the input feature vector x t These matrices are used to calculate attention weights, which are defined as follows:

[0132] Q = XW Q , K = XW K , and V = XW V

[0133] where W Q , W K , and W V are trainable weight matrices representing the parameters of each attention head.

[0134] Next, based on the query matrix and the key matrix, the system calculates the attention weights for each time step. The attention mechanism captures the correlation between different time steps in the sequence through dot-product attention computation:

[0135]

[0136] where d k is the dimension of the key matrix K, QK T represents the dot product operation between the query matrix and the key matrix, which measures the similarity between time steps in the input sequence. The softmax function converts the dot product result into a probability distribution, which is used to weight the content of the value matrix V.

[0137] To improve the expressive power of the model, the system introduces a multi-head attention mechanism. In the multi-head attention mechanism, the input feature vector sequence is processed through multiple parallel attention heads, each with its own independent query, key, and value matrices. The results of multiple attention heads are concatenated and combined through a linear transformation to form the output of the multi-head attention:

[0138] MultiHead(Q, K, V) = Concat(head1, head2,..., head h )W O

[0139] where head i is the output of the i-th attention head, and W O is the linear transformation matrix.

[0140] 3. Cross-time step memory unit:

[0141] In this embodiment, to capture long-time dependencies in the speech sequence, the system introduces a cross-time step memory unit. The memory unit is used to retain the context information of the previous time steps, so that the model can combine the previous history information for processing at the current time step. The design of the memory unit is based on the idea of persistent memory in the Transformer-XL architecture. At each time step t, the model not only considers the current input x t , but also combines the memory state s t-1 from the previous time step t-1.

[0142] s t =MemoryUpdate(s t-1 , MultiHead(Q, K, V))

[0143] The formula indicates that the model combines the multi-head attention output of the current time step with the memory state of the previous time step, updates the memory state s t of the current time step, and passes it to the next time step.

[0144] 4. Feedforward neural network with residual connection:

[0145] In this embodiment, each layer of the deep learning model module is composed of a self-attention mechanism and a feedforward neural network. The output of the multi-head attention is further processed by the feedforward neural network to improve the expression ability of the model. The feedforward neural network contains two linear transformations, and a nonlinear activation function (such as ReLU) is used between each layer.

[0146] To ensure the training stability of the model and enhance the gradient transmission effect, the system uses residual connection and layer normalization after each layer operation. The residual connection directly adds the input and output, thereby alleviating the gradient vanishing problem in deep networks. Layer normalization ensures that the input data has the same statistical properties in each layer.

[0147] The calculation formula of the feedforward neural network is:

[0148] FFN(x) = W2·ReLU(W1·x + b1) + b2

[0149] Where W1 and W2 are weight matrices, b1 and b2 are bias terms, and ReLU is an activation function.

[0150] 5. Output generation:

[0151] In this embodiment, after multi-layer self-attention mechanism, cross-time step memory unit, feedforward neural network and residual connection, the model finally generates a high-dimensional feature vector sequence y = {y1, y2,..., y TThis output feature sequence will be used to generate speech recognition results in the subsequent output module.

[0152] Suppose the user issues a command through the robot: "Play music". After the feature extraction module, the speech signal is converted into a feature vector sequence X = {x1, x2, ..., x T The system's deep learning model receives this feature sequence and first calculates the relevance of each time step using a multi-head self-attention mechanism to determine which time steps contain the most critical information for the current processing. Then, using memory cells across time steps, the system combines historical information from previous time steps with information from the current time step, ensuring that the continuous semantics of "playing music" are understood while maintaining consistency in the speech context.

[0153] Ultimately, the model generates a high-order feature vector sequence that contains a deep understanding of the "play music" instruction and will be further converted into text or control commands in the output module for the robot to execute.

[0154] Through the deep learning model module in this embodiment, the system can effectively capture long-term dependencies in speech sequences and combine contextual information to improve the ability to understand and recognize continuous speech commands.

[0155] Optimal memory management module

[0156] The system dynamically manages the memory cells within the deep learning model, ensuring that critical contextual information is retained during speech recognition while avoiding the accumulation of redundant information. This dynamic management of memory cells allows the system to flexibly refresh or retain memory based on changes in speech input and the current context, ensuring the system's ability to handle extended speech input.

[0157] In this embodiment, the optimal memory management module uses a dynamic programming method to optimize the memory unit state at each time step according to a cost function. The specific implementation steps are as follows:

[0158] 1. Memory status monitoring and action selection:

[0159] In this embodiment, the system generates a memory state s at each time step t t , the memory state contains the context information retained by the model in the previous time step. Every time a new time step is entered, the system needs to decide whether to continue to retain the previous memory state or refresh the memory unit to avoid accumulating noise and redundant information.

[0160] Memory state tThe memory unit ensures the coherence of the speech context by saving data from previous time steps.

[0161] The system makes an action a t at each time step to choose whether to retain or refresh the memory state. The action a t ∈{0,1}, where 0 means to refresh the memory and 1 means to retain the memory.

[0162] 2. Cost function design:

[0163] To optimize the retention and refresh strategy of the memory, a cost function g(s t , a t ) is introduced in this embodiment to measure the loss caused by choosing different actions at each time step. The cost function consists of two parts:

[0164] Noise cost of memory retention Noise(s t ): As the retention time of the memory state s t becomes longer, the historical information may introduce more noise and irrelevant information, which will have a negative impact on the current speech processing. The noise cost describes the impact of the accumulation of redundant information in the memory unit.

[0165] Information loss cost of memory refresh Loss(a t ): When the system chooses to refresh the memory, important context information from previous time steps may be lost, resulting in the system's understanding of the current speech input being not coherent enough. The information loss cost measures the impact of context loss caused by memory refresh.

[0166] Therefore, the cost function can be represented as:

[0167] g(s t , a t ) = α·Noise(s t ) + β·Loss(a t )

[0168] Where α and β are weight parameters of the cost function, used to balance the impact of noise accumulation and information loss.

[0169] 3. Dynamic programming solution:

[0170] In this embodiment, in order to find the optimal memory refresh strategy at each time step, the system uses a dynamic programming method. By calculating the cost of the current memory state s t and action a t , and recursively considering the impact of future time steps, the system can determine the optimal action a

[0171] Dynamic programming solves the problem recursively through the Bellman equation, whose mathematical expression is:

[0172]

[0173] where V(s t ) denotes the value function at the current time step, and V(s t ) represents the minimum value of the total loss in the future given the state s t ; γ is the discount factor, reflecting the influence of future time steps on the current decision.

[0174] By solving the Bellman equation, the system can find the optimal action at each time step, i.e., the decision that minimizes the total loss in the future.

[0175] 4. Memory state update:

[0176] In this embodiment, at each time step t, the system selects the optimal action according to the result of dynamic programming solution. If the system chooses to retain the memory (i.e., ), it continues to use the memory state s t from the previous time step at the current time step; if the system chooses to refresh the memory (i.e., ), it resets the current memory state to re-capture new contextual information.

[0177] In this way, the system can reasonably adjust the memory state at each time step according to the current speech input, thereby ensuring effective processing of long-time speech input.

[0178] Suppose the user issues a series of commands during the speech dialogue, such as: "Turn on the light, dim the brightness, and turn off the light". The optimal memory management module of the system must reasonably manage the memory unit between each command. For commands that are related in a short period of time, such as "dim the brightness and turn off the light", the system should retain the memory state of the previous command to ensure the continuity of the context. However, when the user's topic changes significantly, the system should refresh the memory unit to avoid accumulating irrelevant information.

[0179] For example, in the process of handling "turn on the light" and "dim the brightness", the system may choose to retain the memory because these commands are related in a short period of time, and retaining the context can help the system better understand the logical relationship between "dim the brightness" and "turn on the light". However, when executing "turn off the light" later, the system may judge that the information of the previous command is no longer critical, and therefore chooses to refresh the memory unit to avoid redundant information affecting the processing of the new command.

[0180] Through the optimal memory management module in this embodiment, the system can maintain effective management of important contextual information during long-term voice input, ensure the processing quality of continuous dialogues, and avoid the impact of accumulated irrelevant information on system performance.

[0181] Adaptive attention mechanism module

[0182] The system flexibly allocates attention resources during speech recognition by adaptively adjusting the weights of each attention head in the multi-head self-attention mechanism. Features at different time steps and inputs have varying importance during recognition. The adaptive attention mechanism allows the system to optimize the weights of multiple attention heads based on the dynamic changes in input, ensuring that important features at each time step receive greater attention.

[0183] In this embodiment, the adaptive attention mechanism module dynamically adjusts the weights of the multi-head self-attention mechanism according to the changes in input features and time steps. The specific implementation steps are as follows:

[0184] 1. Initialization of attention head weights:

[0185] In this embodiment, the system processes the input feature vector sequence X = {x1, x2, ..., x T Initially, the weights α of multiple attention heads are i Set to equal values ​​to satisfy the constraints:

[0186]

[0187] Among them, α i represents the weight of the i-th attention head, and h is the number of attention heads.

[0188] 2. Calculation of query, key and value matrices:

[0189] In this embodiment, in each attention head, the input feature vector x t The query matrix Q, key matrix K, and value matrix V are generated through linear transformation. These matrices are used to calculate the attention score and feature weight respectively. The calculation formulas of the query matrix, key matrix, and value matrix are as follows:

[0190]

[0191] in, is the trainable weight matrix of the i-th attention head, and i represents the i-th attention head.

[0192] 3. Objective function construction of adaptive weights:

[0193] In this embodiment, the system dynamically adjusts the weight of each attention head according to the input feature changes and the processing effect of each attention head on the input i . To this end, the system introduces the Lagrange multiplier method to optimize the weight of the attention head. The objective function L(α) is constructed as follows:

[0194]

[0195] where f i (x t ) represents the loss function of the i-th attention head when processing the input feature x t , which measures the performance of the attention head at the current time step; λ is the Lagrange multiplier, which is used to ensure that the sum of the weights of all attention heads is 1.

[0196] 4. Gradient descent method for weight update:

[0197] In this embodiment, the system optimizes the objective function L(α) by gradient descent method, calculates the weight gradient of each attention head, and gradually adjusts the weight according to the gradient update rule. For each attention head α i , its gradient is:

[0198]

[0199] and gradually optimizes the weight by the following update rule:

[0200]

[0201] where η is the learning rate, which is optimized by multiple iterations, so that the weight of each attention head α i is adaptively adjusted to ensure reasonable allocation of attention resources at the current time step t.

[0202] 5. Weight normalization processing:

[0203] In this embodiment, after each weight update, to ensure that the weight of the attention head satisfies the constraint condition , the system normalizes the updated weight α i , and the specific normalization formula is:

[0204]

[0205] Through normalization processing, the system ensures that the sum of the weights of each attention head is 1, so that the attention distribution can be reasonably allocated at each time step.

[0206] 6. Output generation of multi-head self-attention:

[0207] In this embodiment, the multi-head self-attention mechanism adjusted by weight self-adaptation aggregates the relevant information of the input features to generate the attention output at each time step t. The outputs of multiple attention heads are spliced and linearly transformed to generate the final output.

[0208] Suppose the user issues a voice instruction: "Turn off the light, turn on the TV." The feature extraction module converts this voice signal into a feature vector sequence X = {x1, x2,..., x T} and the adaptive attention mechanism module receives this input sequence. At each time step, the system first calculates the query matrix, key matrix, and value matrix through each attention head, and on this basis, dynamically adjusts the weight of each attention head according to the current input feature and the loss value of each attention head.

[0209] For example, during the processing of the voice instruction "Turn off the light", some attention heads may pay more attention to the short keyword "off", while other attention heads may need to process background noise or the context of "light". Therefore, the system adjusts the weights of the attention heads through gradient descent, so that the attention heads related to "turn off the light" get more weight, while other irrelevant information is assigned a smaller weight.

[0210] Subsequently, during the processing of the "turn on the TV" instruction, the system may redistribute the weights of the attention heads to focus more resources on the semantic understanding of "turn on" and "TV" to ensure correct recognition.

[0211] Through the adaptive attention mechanism module in this embodiment, the system can flexibly adjust the weights of the attention heads according to the changes in the input voice signal, ensuring sufficient attention to important features at different time steps, thereby improving the accuracy of voice recognition and the adaptability to complex instructions.

[0212] Output module

[0213] The main function of the system is to convert the voice features processed by the deep learning model into voice recognition results, and further convert the results into control instructions suitable for the robot to execute or display as text information. The output module is the last step of the voice recognition system, which ensures that the high-dimensional feature vector processed by the deep learning model can be correctly parsed and converted into a format that the system can understand, and then perform operations corresponding to the user's intention.

[0214] In this embodiment, the output module mainly includes three functional units: a text conversion unit, a control command generation unit, and a feedback processing unit, and the specific implementation steps are as follows:

[0215] 1. Text conversion unit:

[0216] In this embodiment, the system first converts the high-dimensional feature vector y = {y1, y2,..., y T} generated by the deep learning model into a text result. Each feature vector y t corresponds to the model output at time step t for the speech input, which is mapped to the corresponding character or word by the classification layer.

[0217] Character mapping: The system maps the feature vector y t to a specific speech unit (such as a phoneme, letter, or word) according to the trained classification model, generating a sequence of character outputs at each time step.

[0218] Splicing and correction: The system splices the character results at each time step into complete text while applying a language model for grammar and semantic correction, ensuring the coherence and correctness of the recognition results.

[0219] For example, assume the feature vector y t is mapped to the character C t by the classification layer. The system splices all C t to obtain the speech recognition result:

[0220] Text = {C1, C2,..., C T}

[0221] If there are obvious grammatical errors or semantic inconsistencies in the recognized text, the system will automatically correct them through the pre-set language model.

[0222] 2. Control command generation unit:

[0223] In this embodiment, the output module further parses the text result generated by the text conversion unit into control commands executable by the robot, especially important when it comes to home control or robot operation.

[0224] Command recognition: The system will identify the instruction keywords in the text according to the pre-set voice instruction rules and convert them into corresponding control commands. For example, for the instruction "turn on the TV", the system will identify the corresponding operation instruction "turn on" and "TV" as the execution object.

[0225] Command mapping: The recognized operation instruction and execution object will be mapped to the specific operation of the robot control module. Assuming that the control command CtC_tCt is the combination of instruction keywords and execution objects, the system will convert it into an executable command sequence:

[0226] C = {c1, c2,..., c T}

[0227] Sequential execution and priority processing: For complex instructions such as "turn on the TV and tune to channel 5", the system will identify multiple instructions and execute them in order or according to priority. First, identify "turn on the TV" as the first instruction, and then identify "tune to channel 5" as the second instruction.

[0228] 3. Feedback processing unit:

[0229] In this embodiment, the output module also includes a processing function for executing feedback on the robot, so that the system or user can timely understand the result status of the execution.

[0230] State feedback: After the robot executes the control command, the system will receive feedback information from the robot execution layer to determine whether the command is successfully executed or an exception occurs. For example, if the robot successfully turns on the TV, the system will generate feedback information indicating successful execution; if the command fails (e.g., the robot cannot find the TV device), the system will generate an error report or prompt information.

[0231] User prompt: The generated feedback information can be returned to the user in the form of screen display or voice playback, allowing the user to know the current status of the robot's task execution at any time.

[0232] Suppose the user inputs the instruction "turn off the light and play music" through voice input. The system first processes it through the deep learning model to generate a high-dimensional feature vector sequence C = {c1, c2, …, c T}. After receiving the feature sequence, the output module maps the feature vector to the text result "turn off the light and play music" through the text conversion unit.

[0233] Subsequently, the system identifies two operation commands "turn off the light" and "play music" in the control command generation unit and maps them to two control instructions that the robot can understand. The system executes "turn off the light" and "play music" operations in order and ensures that the two instructions are executed in order through the sequential execution unit.

[0234] After execution, the robot feeds back the current state "the light is off and the music is playing" to the system. The system returns this state to the user through voice or text through the feedback processing unit.

[0235] Through the output module in this embodiment, the system can convert the voice recognition result into executable control commands and combine real-time feedback to ensure accurate execution of the instructions.

[0236] The speech recognition system for robots based on deep learning of the present application can realize efficient processing of complex speech input through the cooperative work of preprocessing, feature extraction, deep learning model processing, optimal memory management, adaptive attention mechanism and output module, solves the recognition problem of traditional speech recognition systems under long-time speech input and complex context, and significantly improves the recognition accuracy and robustness of the system.

[0237] The present application can realize accurate recognition of complex dialogues and continuous speech input through the multi-level self-attention mechanism and memory management strategy of the deep learning model, and is especially suitable for robot control systems in embedded environments, and has strong practical application value and innovation.

[0238] Please refer to the accompanying Figure 2 The present application also provides a speech recognition method for robots based on deep learning, comprising the following steps:

[0239] S1: Collect external speech signals and perform noise reduction and sampling processing on the signals to ensure the clarity and stability of the speech data.

[0240] S2: Feature extraction is performed on the preprocessed speech signals to generate a mel spectrum graph or MFCC feature for subsequent deep learning processing.

[0241] S3: The extracted features are input into a deep learning model based on the Transformer-XL architecture, and the input is processed using a multi-head self-attention mechanism, and the context information in the speech is captured by combining the memory units across time steps.

[0242] S4: The memory refresh strategy is optimized by a dynamic programming method, and the memory state in the deep learning model is dynamically adjusted to improve the ability to recognize continuous speech.

[0243] S5: The weight of each attention head is adaptively adjusted based on the Lagrange multiplier method to ensure that the model reasonably allocates attention resources under different input features.

[0244] S6: The results output by the deep learning model are converted into text or control instructions to complete speech recognition and realize robot control.

[0245] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations of these embodiments can be made without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A robot speech recognition system based on deep learning, characterized in that: include: The speech input module is used to collect external speech signals and pre-process them to generate denoised speech signals; A feature extraction module, configured to convert the preprocessed speech signal into a feature vector sequence; a deep learning model module, comprising a multi-layer self-attention mechanism and a cross-timestep memory unit based on a Transformer-XL architecture, wherein the deep learning model module generates a speech recognition result based on the feature vector sequence; An optimal memory management module, configured to dynamically manage the memory units in the deep learning model module and determine a memory refresh strategy for each time step based on a preset cost function; An adaptive attention mechanism module, configured to adaptively adjust the weight of each attention head in the multi-layer self-attention mechanism according to an input feature vector sequence; An output module, used to convert the speech recognition results generated by the deep learning model module into text or control instruction output; The adaptive attention mechanism module includes: Weight initialization unit, used at each time step Initialize the weights of each attention head in the multi-head self-attention mechanism ,in , is the number of attention heads, and the initial weights satisfy the constraints and ; Attention loss calculation unit, used to calculate each attention head At time step Processing input features The loss value , the loss value Used to measure the performance of the attention head, indicating the degree of error or mismatch when the attention head processes speech features at the current time step; The objective function construction unit is used to construct the objective function of adaptive attention optimization based on the Lagrange multiplier method. The specific form is: in, is the Lagrange multiplier, which is used to ensure that the sum of the weights of each attention head is 1; The weight update unit is used to optimize the objective function by gradient descent and calculate the weight of each attention head. Gradient: And gradually optimize the weights through the following update rules: in, is the learning rate, and through multiple iterations of optimization, the weight of each attention head is Adaptive adjustment to ensure that at the current time step Rationally allocate attention resources; Weight normalization unit, used to update the weights after each iteration Normalization is performed to ensure that the sum of the weights of all attention heads is still 1 to meet the constraints of the adaptive attention mechanism.

2. The deep learning-based robot speech recognition system according to claim 1, characterized in that: The feature extraction module includes: Mel-spectrogram generation unit, which is used to convert the input speech signal into the frequency domain through fast Fourier transform to obtain the speech signal spectrogram, and then process the spectrogram through a series of triangular filters to map the frequency components to the Mel scale and generate the corresponding Mel-spectrogram as the feature vector; A cepstral feature extraction unit is used to perform discrete cosine transform based on the mel-frequency spectrum, extract cepstral coefficients, and generate mel-frequency cepstral coefficients as input features of the deep learning model; an energy normalization unit, configured to perform energy normalization processing on the feature vector to ensure that speech signals of different volumes have a consistent amplitude range in the feature space; A feature vector sequence generating unit is used to arrange the Mel frequency spectrum and Mel frequency cepstral coefficient features into a feature vector sequence according to the time axis. },in Indicates that at time step The feature vector is input into the deep learning model module for subsequent processing.

3. The deep learning-based robot speech recognition system according to claim 1, characterized in that: The deep learning model module includes: Transformer-XL modeling unit for sequences of feature vectors based on the input Build a deep learning model, where the Transformer-XL modeling unit uses a multi-layer stacked self-attention mechanism and feedforward neural network, combined with memory units across time steps, to model the long-term contextual information of speech input; Multi-head self-attention calculation unit, used to perform self-attention calculation on the input feature vector sequence; Cross-time step memory unit, used to transfer and retain memory state between time steps , the previous time step Memory With the current input Combined, the model is Ability to leverage information from previous time steps to achieve contextual understanding and speech continuity processing for long conversations; The residual connection and normalization unit is used to perform residual connection and normalization processing on the outputs of the self-attention calculation unit and the feedforward neural network unit, specifically including adding the input and output through residual connection and performing layer normalization processing on the result to obtain the final output of the deep learning model module.

4. The deep learning-based robot speech recognition system according to claim 3, characterized in that: The multi-head self-attention calculation unit specifically includes the following steps: Obtain the query matrix through linear transformation , key matrix Sum Matrix ,in 、 、 , 、 and is the trainable weight matrix; Calculate the attention weight of each attention head ,in is the bond matrix Dimensions; Concatenate the outputs of all attention heads to form the output of multi-head attention ,in For the The output of an attention head, is the linear transformation matrix.

5. The deep learning-based robot speech recognition system according to claim 1, characterized in that: The optimal memory management module includes: The state monitoring unit is used to monitor the Memory status , where the memory state Represents the deep learning model at time step Context information retained when Action decision unit, used at each time step Based on the current memory state and the preset cost function Confirm memory refresh action ,in Indicates that at time step Whether to perform memory refresh when the cost function is composed of the following two parts: The noise cost of memory retention , indicating the memory state Redundant information may be introduced as the time step increases; The cost of refreshing memory , indicating key information that may be lost when refreshing the memory; The dynamic programming solver is used to solve the memory refresh strategy based on the dynamic programming method, and to determine the optimal action at each time step by calculating the following Bellman equation : in, represents the value function of the current time step, is the discount factor, reflecting the impact of future time steps on the current decision; A strategy updating unit is used to update the strategy at each time step according to the result of the dynamic programming solving unit. Through optimal action Dynamically adjust the memory state and perform memory retention or refresh operations to ensure that the model can adaptively adjust memory management according to the current context and maximize context retention and processing efficiency during speech recognition.

6. The deep learning-based robot speech recognition system according to claim 1, characterized in that: The output module includes: A text conversion unit, configured to convert the speech recognition results generated by the deep learning model module into text output; The command generation unit is used to convert the generated text information into robot control commands.

7. The deep learning-based robot speech recognition system according to claim 6, characterized in that: The text conversion unit specifically includes the following steps: Receives the processed feature vector sequence from the deep learning model module }; Use the classification layer to classify each time step Decode the feature vector Mapping is done to the characters or words corresponding to speech recognition, and these characters or words are concatenated into continuous text.

8. The deep learning-based robot speech recognition system according to claim 6, characterized in that: The command generation unit specifically includes the following steps: Analyze the control commands or instructions contained in the text according to the preset instruction recognition model; Map the identified instructions to the corresponding robot control operations to generate a control command sequence , used to control the robot to perform corresponding actions; If the text message contains multiple instructions, multiple control commands are generated in sequence according to preset priorities or sequence rules.

Images