Speech recognition system for robot based on deep learning

Abstract

The invention relates to the field of voice recognition, and discloses a voice recognition system for a robot based on deep learning, and the system comprises a voice input module which is used for collecting and reducing the noise of a voice signal; the feature extraction module converts the voice signal into a Mel spectrogram or MFCC features; the deep learning model module processes input features through a multi-head self-attention mechanism based on a Transform-XL architecture, and captures context information in combination with a memory unit crossing time steps; the optimal memory management module optimizes a memory refreshing strategy through dynamic programming and dynamically adjusts a memory state; the adaptive attention mechanism module adaptively adjusts the weight of an attention head based on a Lagrange multiplier method; the output module converts the model output into a text or a robot control instruction. The method improves the accuracy and robustness of speech recognition, enhances the processing capability of the system in long-time dialogues and complex contexts, and is suitable for application scenes such as robot control and home management.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition system for robots based on deep learning. Background Art

[0002] With the development of artificial intelligence and robotics, speech recognition technology has become one of the important ways for robots to interact with humans. Existing robot speech recognition systems are usually based on traditional deep learning models, such as recurrent neural networks (RNN) or long short-term memory networks (LSTM), which perform well when processing short voice commands or simple conversations. However, as voice interaction scenarios become more complex, especially when dealing with long conversations, continuous voice commands or complex contexts, these models often face the limitation of being unable to effectively capture long-distance dependencies.

[0003] In complex conversation scenarios, traditional speech recognition models have difficulty maintaining accurate memory of context. As time goes by, early speech information may be gradually forgotten, causing the model to be unable to correctly understand the meaning of subsequent speech input. In addition, the existing models are relatively fixed in the use of attention mechanisms and cannot dynamically adjust attention allocation according to input, resulting in some important speech information not receiving sufficient attention, while irrelevant background information is over-processed, affecting the accuracy and efficiency of recognition.

[0004] For long conversations, the memory mechanism of traditional models lacks an effective management strategy, which causes the accumulated redundant information to interfere with the processing of current input, while excessive refreshing of memory status will lose key contextual information. This lack of memory management leads to suboptimal performance of the system under continuous instructions and complex contexts.

[0005] In addition, although existing technologies can convert voice signals into text or control instructions, in practical applications, how to effectively deal with noise in voice, quickly extract efficient features, and achieve real-time voice recognition in complex environments are still technical problems that need to be solved. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a robot speech recognition system based on deep learning, which solves the problems of insufficient context capture, memory management and dynamic attention allocation of the existing robot speech recognition system in long-term conversations and complex contexts, and improves the recognition accuracy and processing efficiency.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A robot speech recognition system based on deep learning, comprising:

[0008] The speech input module is used to collect external speech signals and pre-process them to generate denoised speech signals;

[0009] A feature extraction module, used for converting the preprocessed speech signal into a feature vector sequence;

[0010] A deep learning model module, including a multi-layer self-attention mechanism and a cross-time step memory unit based on a Transformer-XL architecture, wherein the deep learning model module generates a speech recognition result according to the feature vector sequence;

[0011] An optimal memory management module, used to dynamically manage the memory units in the deep learning model module and determine the memory refresh strategy for each time step based on a preset cost function;

[0012] An adaptive attention mechanism module, used for adaptively adjusting the weight of each attention head in the multi-layer self-attention mechanism according to an input feature vector sequence;

[0013] The output module is used to convert the speech recognition results generated by the deep learning model module into text or control instruction output.

[0014] Preferably, the feature extraction module includes:

[0015] A Mel-spectrogram generation unit is used to convert the input speech signal into the frequency domain through fast Fourier transform to obtain the spectrum of the speech signal, and process the spectrum through a series of triangular filters to map the frequency components to the Mel scale, and generate the corresponding Mel-spectrogram as a feature vector;

[0016] A cepstral feature extraction unit, used for performing discrete cosine transform based on the Mel-frequency spectrum, extracting cepstral coefficients, and generating Mel-frequency cepstral coefficients as input features of a deep learning model;

[0017] An energy normalization unit, used for performing energy normalization processing on the feature vector to ensure that speech signals of different volumes have a consistent amplitude range in the feature space;

[0018] A feature vector sequence generating unit is used to arrange the Mel-frequency spectrum and Mel-frequency cepstral coefficient features into a feature vector sequence X={x1, x2, ..., x T}, where x t Represents the feature vector at time step t, which is used to input into the deep learning model module for subsequent processing.

[0019] Preferably, the deep learning model module includes:

[0020] Transformer-XL modeling unit is used to construct a feature vector sequence X = {x1, x2, ..., x T}Build a deep learning model, wherein the Transformer-XL modeling unit uses a multi-layer stacked self-attention mechanism and a feedforward neural network combined with a memory unit across time steps to achieve modeling of long-term context information of speech input;

[0021] A multi-head self-attention calculation unit, used to perform self-attention calculation on the input feature vector sequence;

[0022] Cross-time step memory unit, used to transfer and retain the memory state s between each time step t , the memory s of the previous time step t-1 t-1 With the current input x t Combined, the model can use the information of previous time steps at the current time step t to achieve contextual understanding of long-term conversations and processing of speech continuity;

[0023] The residual connection and normalization unit is used to perform residual connection and normalization processing on the outputs of the self-attention calculation unit and the feedforward neural network unit, specifically including adding the input and output through the residual connection and performing layer normalization processing on the result to obtain the final output of the deep learning model module.

[0024] Preferably, the multi-head self-attention calculation unit specifically includes the following steps:

[0025] Through linear transformation, we get the query matrix Q, key matrix K and value matrix V, where Q = XW Q , K = XW K 、V=XW V , W Q , W K and W V is the trainable weight matrix;

[0026] Calculate the attention weight of each attention head where d k is the dimension of the key matrix K;

[0027] The outputs of all attention heads are concatenated to form the output of multi-head attention MultiHead(Q, K, V) = Concat(head1, head2, ..., head h )W O , where head i is the output of the i-th attention head, W O is the linear transformation matrix.

[0028] Preferably, the optimal memory management module includes:

[0029] The state monitoring unit is used to monitor the memory state s at each time step t t , where the memory state s t represents the context information retained by the deep learning model at time step t;

[0030] The action decision unit is used to make an action decision based on the current memory state s at each time step t. t And the preset cost function g(s t , a t ) Determine memory refresh action a t , where a t Indicates whether to perform memory refresh at time step t. The cost function includes the following two parts:

[0031] Noise cost caused by memory retention t ), indicating the memory state s t Redundant information that may be introduced as the time step increases;

[0032] The loss cost of refreshing memory is Loss(a t ), indicating critical information that may be lost when refreshing the memory;

[0033] The dynamic programming solver is used to solve the memory refresh strategy based on the dynamic programming method, and determine the optimal action at each time step by calculating the following Bellman equation

[0034]

[0035] Among them, V(s t ) represents the value function of the current time step, γ is the discount factor, which reflects the impact of future time steps on the current decision;

[0036] A strategy updating unit is used to update the optimal action at each time step t according to the result of the dynamic programming solving unit. Dynamically adjust the memory state and perform memory retention or refresh operations to ensure that the model can adaptively adjust memory management according to the current context and maximize context retention and processing efficiency during speech recognition.

[0037] Preferably, the adaptive attention mechanism module includes:

[0038] Weight initialization unit, used to initialize the weight α of each attention head in the multi-head self-attention mechanism at each time step t i , where i∈{1,2,...,h}, h is the number of attention heads, and the initial weights satisfy the constraints And α i ≥0;

[0039] Attention loss calculation unit, used to calculate each attention head i Processing input feature x at time step t t The loss value f i (x t ), the loss value f i (x t ) is used to measure the performance of the attention head, indicating the error or mismatch degree of the attention head when processing speech features at the current time step;

[0040] The objective function construction unit is used to construct the objective function of adaptive attention optimization based on the Lagrange multiplier method. The specific form is:

[0041]

[0042] Among them, λ is the Lagrange multiplier, which is used to ensure that the sum of the weights of each attention head is 1;

[0043] The weight updating unit is used to optimize the objective function by gradient descent and calculate the weight α of each attention head. i The gradient is:

[0044]

[0045] And gradually optimize the weights through the following update rules:

[0046]

[0047] Among them, η is the learning rate. Through multiple iterations of optimization, the weight of each attention head α i Adaptive adjustment to ensure reasonable allocation of attention resources within the current time step t;

[0048] The weight normalization unit is used to update the weight α after each iteration. i Normalization is performed to ensure that the sum of the weights of all attention heads is still 1 to meet the constraints of the adaptive attention mechanism.

[0049] Preferably, the output module includes:

[0050] A text conversion unit, used to convert the speech recognition result generated by the deep learning model module into text output;

[0051] The command generation unit is used to convert the generated text information into robot control commands.

[0052] Preferably, the text conversion unit specifically includes the following steps:

[0053] Receive the processed feature vector sequence y={y1, y2, ..., y T};

[0054] Use the classification layer to classify each time step y t Decode the feature vector y t Mapping is done to the characters or words corresponding to speech recognition, and these characters or words are concatenated into continuous text.

[0055] Preferably, the command generation unit specifically includes the following steps:

[0056] Analyze the control commands or instructions contained in the text according to the preset instruction recognition model;

[0057] Map the identified instructions to the corresponding robot control operations to generate a control command sequence C = {c1, c2, ..., c T}, used to control the robot to perform corresponding actions;

[0058] If the text information contains multiple instructions, multiple control commands are generated in sequence according to preset priorities or sequence rules.

[0059] The present invention also provides a robot speech recognition method based on deep learning, comprising the following steps:

[0060] Collect external voice signals and perform noise reduction and sampling processing on them;

[0061] Extract features from the preprocessed speech signal to generate Mel-spectrogram or MFCC features;

[0062] Input the features into a deep learning model based on the Transformer-XL architecture, process the input using a multi-head self-attention mechanism, and capture contextual information based on memory units across time steps;

[0063] Optimize memory refresh strategy through dynamic programming method and dynamically adjust memory status in deep learning models;

[0064] Adaptively adjust the weight of each attention head based on the Lagrange multiplier method;

[0065] Convert the output of the deep learning model into text or robot control instructions to complete the speech recognition process.

[0066] The present invention provides a robot speech recognition system based on deep learning, which has the following beneficial effects:

[0067] 1. By introducing a deep learning model based on the Transformer-XL architecture, the system can capture contextual information in long speech sequences. This architecture allows the model to process long-distance dependencies in speech signals, solving the problem that traditional speech recognition systems perform poorly in long conversations or complex contexts. In addition, memory units across time steps retain important context, making the system's semantic understanding of different conversation segments more accurate and enhancing the overall robustness of the system.

[0068] 2. The present invention uses the Lagrange multiplier method to adaptively adjust the weight of each attention head, and the system can dynamically adjust the allocation of attention resources according to the characteristics of the input. This means that the model can better focus on the important parts of the speech input, ignore irrelevant background information, and improve the accuracy of speech recognition. This adaptive mechanism makes the system more flexible and can cope with a variety of different speech environments and instructions.

[0069] 3. The system of the present invention optimizes the memory refresh strategy through dynamic programming methods, and can dynamically adjust the memory state in the deep learning model to ensure that redundant information is not accumulated or important context is not lost when processing long-term voice input. This memory management method can effectively improve the continuity and semantic coherence of speech recognition, especially in scenarios involving multi-step commands or long-term conversations.

[0070] 4. The present invention uses the Mel-spectrogram and MFCC feature extraction method, and the system can efficiently extract the time-frequency information of the speech signal into a feature vector that can be processed by the model. This feature extraction method not only retains the core information in the speech, but also reduces the interference of noise and redundant information, so that the subsequent deep learning model can better understand the speech content and improve the processing efficiency of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the system architecture of the present invention;

[0072] Figure 2 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Please refer to the attached Figure 1, the embodiment of the present invention provides a robot speech recognition system based on deep learning, which can realize accurate recognition and real-time execution of robot voice commands by preprocessing the voice input, feature extraction, deep learning model processing, optimal memory management, adaptive attention mechanism and final output processing. The system is based on the Transformer-XL architecture, and utilizes the powerful feature extraction capability of the deep learning model, as well as the dynamic memory management and attention mechanism, to realize efficient processing and accurate recognition of complex voice input, with high robustness and adaptability.

[0075] The system includes a speech input module, a feature extraction module, a deep learning model module, an optimal memory management module, an adaptive attention mechanism module and an output module. The details are as follows:

[0076] Voice input module

[0077] The system collects external voice signals through the robot microphone, converts them into denoised signals after preprocessing, and provides high-quality input for subsequent feature extraction and deep learning model processing. The main task of this module is to preliminarily process voice data from complex environments, ensure that the voice signal can clearly reflect the user's instructions or conversation content, remove interference factors, and provide stable audio input.

[0078] In this embodiment, the main functions of the voice input module include voice signal acquisition, noise removal and signal normalization. The specific implementation process is as follows:

[0079] 1. Voice signal collection:

[0080] The robot collects voice signals from the external environment through microphones or microphone arrays. The audio signal is sampled at a specified sampling rate to capture the frequency details of the voice, ensuring high audio quality and voice clarity.

[0081] The sampled signal is recorded as x(n), where n represents the time step, and the audio signal is stored as time series data.

[0082] 2. Denoising:

[0083] In this embodiment, in order to improve the quality and recognizability of the voice signal, the voice input module first performs denoising processing on the collected voice signal.

[0084] A low-pass filter is used to remove high-frequency noise by limiting the frequency range of the signal (eg, below 3.5kHz) to ensure that the retained frequencies mainly contain the valid part of the human voice signal.

[0085] After that, the environmental noise is suppressed using an adaptive filter, which can dynamically adjust its filtering characteristics to adapt to the changing background noise conditions, thereby further reducing the background noise interference and retaining the main frequency components of the speech.

[0086] 3. Signal normalization:

[0087] In this embodiment, in order to avoid the influence of the feature extraction process on the strength difference of the speech input signal, the system performs normalization processing on the processed speech signal.

[0088] The normalization process adjusts the amplitude range of the audio signal to a fixed interval. A common method is to scale the maximum absolute value of the signal to 1, thereby ensuring that speech signals of different volumes have consistent amplitudes in subsequent processing.

[0089] 4. Signal framing and window function processing:

[0090] In this embodiment, in order to cooperate with the subsequent feature extraction operation, the input speech signal needs to be framed. Framing refers to cutting the continuous speech signal into multiple shorter frames of fixed length, each frame corresponding to a signal in a time window. Each frame will be used independently for subsequent frequency domain analysis.

[0091] Suppose a user inputs a command when using the robot system: "Open the window". The robot collects the voice signal containing the command through its microphone array, along with some background noise, such as wind and other sounds in the environment. The voice input module first digitizes the voice signal through the sampler to form a sampling sequence.

[0092] Next, the system uses a low-pass filter and an adaptive filter to remove background noise. After filtering, the voice signal in the "open the window" command is retained, while the wind sound in the background is greatly weakened. After that, the system normalizes the processed voice signal and adjusts the volume to a uniform range to ensure that the voice signal can stably enter the next feature extraction process.

[0093] Finally, the system divides the processed speech signal into frames according to 25ms time windows, and the signal of each frame is processed by a window function so that the subsequent feature extraction module can better capture the frequency information in the speech.

[0094] Through the voice input module in this embodiment, the system can effectively collect and process voice signals in the external environment, remove noise and ensure consistent signal strength, and provide clear and accurate voice input for subsequent deep learning processing.

[0095] Feature extraction module

[0096] When analyzing the preprocessed speech signal, this system mainly converts the time domain signal into the frequency domain signal to extract an efficient representation that can reflect the speech features. Feature extraction is an important step in the speech recognition system. By extracting features such as Mel-frequency spectrogram and MFCC (Mel-frequency cepstral coefficient), the system can convert the speech signal into a feature vector sequence suitable for deep learning model processing.

[0097] In this embodiment, the feature extraction module converts the preprocessed speech signal into a feature vector sequence, which specifically includes the following steps:

[0098] 1. Framing and windowing processing:

[0099] In this embodiment, in order to facilitate the subsequent frequency domain analysis, the preprocessed normalized speech signal x is first norm (n) It is divided into multiple short time frames. Each frame usually corresponds to a time window of about 25ms, and there is a certain overlap between adjacent frames (usually 10ms).

[0100] Each frame contains N sampling points Where N is usually set to 400 samples (assuming a sampling rate of 16kHz).

[0101] After framing, in order to reduce the spectrum distortion caused by the frame boundary, the Hamming Window function is applied to each frame signal for windowing. The Hamming Window can reduce the discontinuity at the frame boundary and improve the accuracy of spectrum estimation.

[0102] 2. Fast Fourier Transform (FFT):

[0103] In this embodiment, a fast Fourier transform (FFT) is performed on each frame of the signal after the windowing process to convert the time domain signal into a frequency domain signal, thereby obtaining the spectrum information of each frame of the signal.

[0104] Assume that the windowed signal of each frame is x win (n), its FFT transformation result is X(k), where k is the frequency index, indicating the amplitude of each frequency component. The mathematical expression of FFT is:

[0105]

[0106] Wherein, N is the number of sampling points per frame, and k=0, 1, ..., N-1 represents the index of each frequency component.

[0107] 3. Mel spectrum generation:

[0108] In this embodiment, in order to better reflect the human ear's perception characteristics of different frequencies, the system converts the FFT result into a Mel spectrum. Specifically, a set of Mel filters is used to map the frequency domain signal to the Mel frequency scale, which is more in line with the human ear's perception characteristics, especially the higher sensitivity to low-frequency signals.

[0109] Set the Mel filter bank to use a triangular filter on the spectrum |X(k)| 2 Perform weighted summation to generate the Mel spectrum M(f) of each frame signal, and its mathematical expression is:

[0110]

[0111] Among them, H f (k) is the frequency response of the f-th Mel filter, k min and k max Represent the minimum frequency index and maximum frequency index of the spectrum respectively.

[0112] 4. Discrete Cosine Transform (DCT) and MFCC feature extraction:

[0113] In this embodiment, in order to further compress the dimension of the mel-spectrogram and extract key features that can reflect the speech signal, the system performs discrete cosine transform (DCT) on the mel-spectrogram of each frame to obtain mel-frequency cepstral coefficients (MFCC).

[0114] The mathematical expression of discrete cosine transform is:

[0115]

[0116] Among them, C(m) is the m-th order cepstral coefficient, F is the number of Mel filters, and M(f) is the f-th Mel spectrum value.

[0117] Through DCT transformation, the system can effectively reduce the redundancy of the Mel-spectrogram and generate more compact MFCC features to represent the frequency information of each frame of speech signal.

[0118] 5. Feature vector sequence construction:

[0119] In this embodiment, the MFCC feature vectors generated for each frame are arranged in chronological order to form a feature vector sequence X={x1, x2, ..., x T}, where x t Represents the MFCC feature vector of the tth frame. This sequence will be passed as input to the deep learning model module for further processing.

[0120] Suppose the robot voice system receives an audio clip containing the user's command: "Turn off the lights". First, the system collects and preprocesses the voice signal of the command through the voice input module. Then, the feature extraction module starts to process the preprocessed signal. The system first divides the audio signal into multiple 25ms frames and applies the Hamming window function to each frame to eliminate the discontinuity caused by the frame boundary.

[0121] Next, the system performs a fast Fourier transform on each frame of the signal, converting the time domain signal into a frequency domain signal to generate the spectrum information of each frame. Then, a Mel filter bank is used to map the spectrum information to the Mel scale to generate a Mel spectrum map for each frame. Next, the system compresses the Mel spectrum map into MFCC features through discrete cosine transform to obtain the feature vector of each frame.

[0122] Finally, the MFCC feature vectors of all frames are arranged in chronological order to form a feature vector sequence of the speech segment. The system inputs the feature vector sequence into the deep learning model module for speech recognition processing.

[0123] Through the feature extraction module in this embodiment, the system can extract feature vectors with good recognition from the original speech signal, ensuring the speech processing effect of the subsequent deep learning model.

[0124] Deep Learning Model Module

[0125] The system uses a multi-layer self-attention mechanism based on the Transformer-XL architecture and memory units across time steps to deeply process the feature vector sequence of the speech signal. Through the multi-head self-attention mechanism, the system can capture long-term dependencies in the input sequence and retain information from previous time steps through memory units to ensure effective modeling of the context of long-term conversations.

[0126] In this embodiment, the deep learning model module implements deep processing of speech features based on the Transformer-XL architecture, which specifically includes the following steps:

[0127] 1. Feature vector sequence input:

[0128] In this embodiment, the feature vector sequence X generated by the feature extraction module is T} is input to the deep learning model module. Each feature vector x t Corresponding to the MFCC features of the tth frame in the speech signal, the feature sequence is an arrangement of time steps, reflecting the time-frequency characteristics of the speech signal.

[0129] 2. Multi-head self-attention mechanism:

[0130] In this embodiment, the deep learning model module processes the input feature vector sequence through a multi-head self-attention mechanism. The multi-head self-attention mechanism allows the model to capture information from different angles when processing the sequence and weight each time step in the sequence to highlight important information.

[0131] First, by inputting the feature vector x t Linear transformation of , generates query matrix Q, key matrix K and value matrix V. These matrices are used to calculate attention weights, which are defined as follows:

[0132] Q=XW Q , K = XW K 、V=XW V

[0133] Among them, W Q , W K and W V is a trainable weight matrix representing the parameters of each attention head.

[0134] Next, based on the query matrix and the key matrix, the system calculates the attention weight for each time step. The attention mechanism captures the correlation between different time steps in the sequence through dot product attention calculation:

[0135]

[0136] Among them, d k is the dimension of the key matrix K, QK T Represents the dot product operation between the query matrix and the key matrix, which is used to measure the similarity between each time step in the input sequence. The softmax function converts the dot product result into a probability distribution, which is used to weight the contents of the value matrix V.

[0137] In order to improve the expressiveness of the model, the system introduces a multi-head attention mechanism. In the multi-head attention mechanism, the input feature vector sequence is processed by multiple parallel attention heads, each of which has its own independent query, key, and value matrices. The results of multiple attention heads are concatenated and combined through linear transformation to form the output of the multi-head attention:

[0138] MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O

[0139] Among them, head i is the output of the i-th attention head, W O is the linear transformation matrix.

[0140] 3. Memory units across time steps:

[0141] In this embodiment, in order to capture the long-term dependencies in the speech sequence, the system introduces a cross-time step memory unit. The memory unit is used to retain the context information of the previous time step, so that the model can combine the previous historical information for processing at the current time step. The design of the memory unit is based on the idea of ​​persistent memory in the Transformer-XL architecture. At each time step t, the model not only considers the current input x t , and also combines the memory state s from the previous time step t-1 t-1 :

[0142] s t =MemoryUpdate(s t-1 , MultiHead(Q, K, V))

[0143] This formula indicates that the model combines the multi-head attention output of the current time step with the memory state of the previous time step to update the memory state s of the current time step t , and pass it to the next time step.

[0144] 4. Feedforward Neural Network with Residual Connection:

[0145] In this embodiment, each layer of the deep learning model module consists of a self-attention mechanism and a feedforward neural network. The output of the multi-head attention is further processed by the feedforward neural network to improve the expressiveness of the model. The feedforward neural network contains two layers of linear transformation and uses a nonlinear activation function (such as ReLU) between each layer.

[0146] To ensure the training stability of the model and enhance the gradient transfer effect, the system uses residual connections and layer normalization after each layer operation. Residual connections directly add input and output, thereby alleviating the gradient vanishing problem in deep networks. Layer normalization ensures that the input data has the same statistical characteristics in each layer.

[0147] The calculation formula of the feedforward neural network is:

[0148] FFN(x)=W2·ReLU(W1·x+b1)+b2

[0149] Among them, W1 and W2 are weight matrices, b1 and b2 are bias terms, and ReLU is the activation function.

[0150] 5. Output generation:

[0151] In this embodiment, after multi-layer self-attention mechanism, cross-time step memory unit, feedforward neural network and residual connection, the model finally generates a high-dimensional feature vector sequence y = {y1, y2, ..., y T}. This output feature sequence will be used to generate speech recognition results in the subsequent output module.

[0152] Suppose the user issues a command through the robot: "Play music". After the feature extraction module, the speech signal is converted into a feature vector sequence X = {x1, x2, ..., x T The system's deep learning model module receives the feature sequence and first calculates the correlation of each time step through a multi-head self-attention mechanism to determine which time step's information is key to the current processing. Then, through the memory unit across time steps, the system combines the historical information of the previous time step with the information of the current time step to ensure that the consistency of the speech context is maintained while understanding the continuous semantics of "playing music".

[0153] Ultimately, the model generates a high-order feature vector sequence that contains a deep understanding of the "play music" instruction and will be further converted into text or control commands in the output module for the robot to execute.

[0154] Through the deep learning model module in this embodiment, the system can effectively capture the long-term dependencies in the speech sequence and combine the context information to improve the understanding and recognition capabilities of continuous speech commands.

[0155] Optimal memory management module

[0156] The system dynamically manages the memory units in the deep learning model to ensure that during the speech recognition process, the model can reasonably retain key contextual information while avoiding the cumulative impact of redundant information. Through dynamic management of memory units, the system can flexibly refresh or retain memory according to changes in speech input and the current context, thereby ensuring the system's ability to handle long-term speech input.

[0157] In this embodiment, the optimal memory management module adopts a dynamic programming method to optimize the memory unit state at each time step according to the cost function. The specific implementation steps are as follows:

[0158] 1. Memory status monitoring and action selection:

[0159] In this embodiment, the system generates a memory state s at each time step t t , which contains the context information retained by the model in the previous time step. Whenever entering a new time step, the system needs to decide whether to continue to retain the previous memory state or refresh the memory unit to avoid accumulating noise and redundant information.

[0160] Memory Status tRepresents the historical information retained by the system at time step t. The memory unit ensures the coherence of the speech context by saving data from previous time steps.

[0161] The system will make an action a at each time step t , used to choose whether to keep or refresh the memory state. Action a t ∈{0, 1}, where 0 means refreshing the memory and 1 means retaining the memory.

[0162] 2. Cost function design:

[0163] In order to optimize the memory retention and refresh strategy, this embodiment introduces the cost function g(s t , a t ), which is used to measure the loss caused by choosing different actions at each time step. The cost function consists of two parts:

[0164] Noise cost of memory retention t ): With the memory state s t The longer the retention time is, the more noise and irrelevant information the historical information may introduce, which will have a negative impact on the current speech processing. The noise cost describes the impact of the accumulation of redundant information in the memory unit.

[0165] The information loss cost of memory refresh Loss(a t ): When the system chooses to refresh the memory, it may lose important contextual information from previous time steps, resulting in an incoherent understanding of the current speech input. The information loss cost measures the impact of context loss caused by memory refreshing.

[0166] Therefore, the cost function can be expressed as:

[0167] g(s t , a t )=α·Noise(s t )+β·Loss(a t )

[0168] Among them, α and β are the weight parameters of the cost function, which are used to balance the impact between noise accumulation and information loss.

[0169] 3. Dynamic programming solution:

[0170] In this embodiment, in order to find the optimal memory refresh strategy at each time step, the system uses a dynamic programming method. By calculating the current memory state s t and action a t and recursively consider the impact of future time steps, the system is able to determine the optimal action at each time step

[0171] Dynamic programming is solved recursively through the Bellman equation, and the mathematical expression of the Bellman equation is:

[0172]

[0173] Among them, V(s t ) represents the value function of the current time step, indicating that in state s t γ is the discount factor, which reflects the impact of future time steps on the current decision.

[0174] By solving the Bellman equation, the system can find the optimal action at each time step That is, the decision to minimize total future losses.

[0175] 4. Memory status update:

[0176] In this embodiment, at each time step t, the system selects the optimal action based on the result of dynamic programming solution. If you choose to keep the memory (i.e. ), the system continues to use the memory state s of the previous time step in the current time step t ; If you choose to refresh the memory (ie ), the system resets the current memory state to recapture the new context information.

[0177] In this way, the system can reasonably adjust the memory state according to the current speech input at each time step, thereby ensuring effective processing of long-term speech input.

[0178] Suppose a user issues a series of commands during a voice conversation, such as "Turn on the lights, dim the brightness, and then turn off the lights." The system's optimal memory management module must properly manage memory cells between each command. For commands that are consecutively related in a short period of time, such as "Dim the brightness, then turn off the lights," the system should retain the memory state of the previous command to ensure context continuity. However, when the user's topic transition is large, the system should refresh the memory cells to avoid accumulating irrelevant information.

[0179] For example, when processing "turn on the light" and "dim the brightness", the system may choose to retain the memory because these commands are related in a short period of time, and retaining the context can help the system better understand the logical relationship between "dim the brightness" and "turn on the light". However, when executing "turn off the light again" later, the system may determine that the information of the previous command is no longer critical, so it chooses to refresh the memory unit to avoid redundant information affecting the processing of the new command.

[0180] Through the optimal memory management module in this embodiment, the system can maintain effective management of important contextual information during long-term voice input, ensure the processing quality of continuous dialogues, and avoid the impact of accumulated irrelevant information on system performance.

[0181] Adaptive Attention Mechanism Module

[0182] The system flexibly allocates attention resources during speech recognition by adaptively adjusting the weights of each attention head in the multi-head self-attention mechanism. Features at different time steps and different inputs have different importance in the recognition process. The adaptive attention mechanism allows the system to optimize the weights of multiple attention heads according to the dynamic changes of the input, so that important features in each time step can receive greater attention.

[0183] In this embodiment, the adaptive attention mechanism module dynamically adjusts the weights of the multi-head self-attention mechanism according to the changes in input features and time steps. The specific implementation steps are as follows:

[0184] 1. Initialization of attention head weights:

[0185] In this embodiment, the system processes the input feature vector sequence X = {x1, x2, ..., x T}. Initially, the weights α of multiple attention heads i Set to equal values ​​to satisfy the constraints:

[0186]

[0187] Among them, α i represents the weight of the i-th attention head, and h is the number of attention heads.

[0188] 2. Query, key and value matrix calculation:

[0189] In this embodiment, in each attention head, the input feature vector x t The query matrix Q, key matrix K, and value matrix V are generated through linear transformation. These matrices are used to calculate the attention score and feature weight respectively. The calculation formulas of the query matrix, key matrix, and value matrix are as follows:

[0190]

[0191] in, is the trainable weight matrix of the i-th attention head, and i represents the i-th attention head.

[0192] 3. Objective function construction of adaptive weights:

[0193] In this embodiment, the system dynamically adjusts the weight α of each attention head according to the input feature changes and the processing effect of each attention head on the input. i To this end, the system introduces the Lagrange multiplier method to optimize the weight of the attention head. The objective function L(α) is constructed as follows:

[0194]

[0195] Among them, f i (x t ) indicates that the i-th attention head is processing the input feature x t The loss function at the time t is used to measure the performance of the attention head at the current time step; λ is the Lagrange multiplier, which is used to ensure that the sum of the weights of all attention heads is 1.

[0196] 4. Gradient descent method for weight update:

[0197] In this embodiment, the system optimizes the objective function L(α) by the gradient descent method, calculates the weight gradient of each attention head, and gradually adjusts the weight according to the gradient update rule. i , whose gradient is:

[0198]

[0199] And gradually optimize the weights through the following update rules:

[0200]

[0201] Among them, η is the learning rate. Through multiple iterations of optimization, the weight of each attention head α i Adaptive adjustment ensures that attention resources are properly allocated within the current time step t.

[0202] 5. Weight normalization processing:

[0203] In this embodiment, after each weight update, to ensure that the weight of the attention head meets the constraint condition The system updates the weight α i Normalization is performed, and the specific normalization formula is:

[0204]

[0205] Through normalization, the system ensures that the sum of the weights of each attention head is 1, so that the attention distribution can be reasonably allocated at each time step.

[0206] 6. Output generation of multi-head self-attention:

[0207] In this embodiment, the multi-head self-attention mechanism after weight adaptive adjustment aggregates the relevant information of the input features to generate the attention output of each time step t. The outputs of multiple attention heads are concatenated and linearly transformed to generate the final output.

[0208] Suppose the user issues a voice command: "Turn off the lights, turn on the TV". The feature extraction module converts the voice signal into a feature vector sequence X = {x1, x2, ..., x T}, the adaptive attention mechanism module receives the input sequence. At each time step, the system first calculates the query matrix, key matrix and value matrix through each attention head, and then dynamically adjusts the weight of each attention head based on the current input features and the loss value of each attention head.

[0209] For example, in the process of processing the voice command "turn off the lights", some attention heads may pay more attention to the short-term keyword "off", while other attention heads may need to process background noise or the context of "lights". Therefore, the system adjusts the weights of the attention heads through the gradient descent method, so that the attention heads related to "turn off the lights" get a larger weight, while other irrelevant information is assigned a smaller weight.

[0210] Later, when processing the "turn on the TV" command, the system may reallocate the weights of the attention heads, focusing more resources on the semantic understanding of "turn on" and "TV" to ensure correct recognition.

[0211] Through the adaptive attention mechanism module in this embodiment, the system can flexibly adjust the weight of the attention head according to the changes in the input voice signal, ensuring sufficient attention to important features at different time steps, thereby improving the accuracy of voice recognition and the ability to adapt to complex instructions.

[0212] Output Module

[0213] The main function of the system is to convert the speech features processed by the deep learning model into speech recognition results, and further convert the results into control instructions suitable for robot execution or display as text information. The output module is the last step of the speech recognition system. It ensures that the high-dimensional feature vector processed by the deep learning model can be correctly parsed and converted into an instruction format that the system can understand, and then perform operations corresponding to the user's intentions.

[0214] In this embodiment, the output module mainly includes three functional units: a text conversion unit, a control command generation unit, and a feedback processing unit. The specific implementation steps are as follows:

[0215] 1. Text conversion unit:

[0216] In this embodiment, the system first converts the high-dimensional feature vector y={y1, y2, ..., y T} is converted to text results. Each feature vector y t The model output corresponding to the speech input at time step t is mapped to the corresponding character or word through the classification layer.

[0217] Character mapping: The system converts the feature vector y into t Mapped to specific speech units (such as phonemes, letters, or words), the resulting output is a sequence of characters at each time step.

[0218] Splicing and correction: The system splices the character results of each time step into a complete text, and applies the language model for grammatical and semantic correction to ensure the coherence and correctness of the recognition results.

[0219] For example, suppose the feature vector y t Get character C through classification mapping t , the system will set all C t Splicing to get the speech recognition result:

[0220] Text = {C1, C2, ..., C T}

[0221] If there are obvious grammatical errors or semantic incoherence in the recognized text, the system will automatically correct it using the preset language model.

[0222] 2. Control command generation unit:

[0223] In this embodiment, the output module further parses the text results generated by the text conversion unit into control commands executable by the robot, which is particularly important when it comes to home control or robot operation.

[0224] Command recognition: The system will recognize the command keywords in the text according to the preset voice command rules and convert them into corresponding control commands. For example, for the command "turn on the TV", the system will recognize the operation command corresponding to "turn on" and take "TV" as the execution object.

[0225] Command mapping: The identified operation instructions and execution objects will be mapped to the specific operations of the robot control module. Assuming that the control command CtC_tCt is a combination of instruction keywords and execution objects, the system converts it into an executable command sequence:

[0226] C = {c1, c2, ..., c T}

[0227] Sequential execution and priority processing: For complex instructions, such as "Turn on the TV first, then tune to channel 5", the system will recognize multiple instructions and execute them in sequence or priority. First, "Turn on the TV" is recognized as the first instruction, and then "Turn to channel 5" is recognized as the second instruction.

[0228] 3. Feedback processing unit:

[0229] In this embodiment, the output module also includes a processing function for the robot's execution feedback, so that the system or the user can understand the result status of the execution in a timely manner.

[0230] Status feedback: After the robot executes the control command, the system will receive feedback information from the robot execution layer to determine whether the command is successfully executed or an exception occurs. For example, if the robot successfully turns on the TV, the system will generate feedback information of successful execution; if the command execution fails (such as the robot cannot find the TV device), the system will generate an error report or prompt information.

[0231] User Tips: The generated feedback information can be returned to the user through screen display or voice playback, so that the user can always understand the status of the robot's current task execution.

[0232] Suppose the user issues a command through voice input: "Turn off the lights and play music". The system first processes it through the deep learning model to generate a high-dimensional feature vector sequence C = {c1, c2, ..., c T After receiving the feature sequence, the output module maps the feature vector into a text result: "Turn off the lights and play music" through the text conversion unit.

[0233] Subsequently, the system identified two operation commands in the control command generation unit: "turn off the light" and "play music", and mapped them into two control instructions that the robot can understand. The system executes the "turn off the light" and "play music" operations in sequence, and ensures that the two instructions are executed in sequence through the sequential execution unit.

[0234] After the execution is completed, the robot will feedback the current status to the system: "The lights are off and the music is playing." The system will return this status to the user through voice or text through the feedback processing unit.

[0235] Through the output module in this embodiment, the system can convert the speech recognition results into executable control commands, and combine them with real-time feedback to ensure the accurate execution of the instructions.

[0236] The deep learning-based robot speech recognition system of the present invention can realize efficient processing of complex speech input through the collaborative work of preprocessing, feature extraction, deep learning model processing, optimal memory management, adaptive attention mechanism and output module, solves the recognition problem of traditional speech recognition system under long-term speech input and complex context, and significantly improves the recognition accuracy and system robustness.

[0237] The present invention can realize accurate recognition of complex conversations and continuous speech inputs through the multi-level self-attention mechanism and memory management strategy of the deep learning model. It is particularly suitable for robot control systems in embedded environments and has strong practical application value and innovation.

[0238] Please refer to the attached Figure 2 The present invention also provides a robot speech recognition method based on deep learning, comprising the following steps:

[0239] S1: Collect external voice signals, perform noise reduction and sampling processing on the signals to ensure the clarity and stability of the voice data.

[0240] S2: Extract features from the preprocessed speech signal to generate Mel-spectrogram or MFCC features for subsequent deep learning processing.

[0241] S3: The extracted features are input into a deep learning model based on the Transformer-XL architecture, and the multi-head self-attention mechanism is used to process the input, combined with memory units across time steps to capture the contextual information in the speech.

[0242] S4: Optimize the memory refresh strategy through dynamic programming method and dynamically adjust the memory state in the deep learning model to improve the ability to recognize continuous speech.

[0243] S5: Adaptively adjust the weight of each attention head based on the Lagrange multiplier method to ensure that the model reasonably allocates attention resources under different input features.

[0244] S6: Convert the output of the deep learning model into text or control instructions to complete speech recognition and realize robot control.

[0245] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A robot speech recognition system based on deep learning, characterized in that: include: The speech input module is used to collect external speech signals and pre-process them to generate denoised speech signals; A feature extraction module, used for converting the preprocessed speech signal into a feature vector sequence; A deep learning model module, including a multi-layer self-attention mechanism and a cross-time step memory unit based on a Transformer-XL architecture, wherein the deep learning model module generates a speech recognition result according to the feature vector sequence; An optimal memory management module, used to dynamically manage the memory units in the deep learning model module and determine the memory refresh strategy for each time step based on a preset cost function; An adaptive attention mechanism module, used for adaptively adjusting the weight of each attention head in the multi-layer self-attention mechanism according to an input feature vector sequence; The output module is used to convert the speech recognition results generated by the deep learning model module into text or control instruction output.

2. The robot speech recognition system based on deep learning according to claim 1, characterized in that: The feature extraction module comprises: A Mel-spectrogram generation unit is used to convert the input speech signal into the frequency domain through fast Fourier transform to obtain the spectrum of the speech signal, and process the spectrum through a series of triangular filters to map the frequency components to the Mel scale, and generate the corresponding Mel-spectrogram as a feature vector; A cepstral feature extraction unit, used for performing discrete cosine transform based on the Mel-frequency spectrum, extracting cepstral coefficients, and generating Mel-frequency cepstral coefficients as input features of a deep learning model; An energy normalization unit, used for performing energy normalization processing on the feature vector to ensure that speech signals of different volumes have a consistent amplitude range in the feature space; A feature vector sequence generating unit is used to arrange the Mel-frequency spectrum and Mel-frequency cepstral coefficient features into a feature vector sequence X={x1, x2,…, x T }, where x t Represents the feature vector at time step t, which is used to input into the deep learning model module for subsequent processing.

3. The robot speech recognition system based on deep learning according to claim 1, characterized in that: The deep learning model module includes: Transformer-XL modeling unit is used to construct a feature vector sequence X = {x1, x2, …, x T }Build a deep learning model, wherein the Transformer-XL modeling unit uses a multi-layer stacked self-attention mechanism and a feedforward neural network combined with a memory unit across time steps to achieve modeling of long-term context information of speech input; A multi-head self-attention calculation unit, used to perform self-attention calculation on the input feature vector sequence; Cross-time step memory unit, used to transfer and retain the memory state s between each time step t , the memory s of the previous time step t-1 t-1 With the current input x t Combined, the model can use the information of previous time steps at the current time step t to achieve contextual understanding of long-term conversations and processing of speech continuity; The residual connection and normalization unit is used to perform residual connection and normalization processing on the outputs of the self-attention calculation unit and the feedforward neural network unit, specifically including adding the input and output through the residual connection and performing layer normalization processing on the result to obtain the final output of the deep learning model module.

4. The robot speech recognition system based on deep learning according to claim 3, characterized in that: The multi-head self-attention calculation unit specifically includes the following steps: Through linear transformation, we get the query matrix Q, key matrix K and value matrix V, where Q = XW Q , K = XW K 、V=XW V , W Q , W K and W v is the trainable weight matrix; Calculate the attention weight of each attention head where d k is the dimension of the key matrix K; The outputs of all attention heads are concatenated to form the output of multi-head attention MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O , where head i is the output of the i-th attention head, W O is the linear transformation matrix.

5. The robot speech recognition system based on deep learning according to claim 1, characterized in that: The optimal memory management module includes: The state monitoring unit is used to monitor the memory state s at each time step t t , where the memory state s t represents the context information retained by the deep learning model at time step t; The action decision unit is used to make an action decision based on the current memory state s at each time step t. t And the preset cost function g(s t ,a t ) Determine memory refresh action a t , where a t Indicates whether to perform memory refresh at time step t. The cost function includes the following two parts: Noise cost caused by memory retention t ), indicating the memory state s t Redundant information that may be introduced as the time step increases; The loss cost of refreshing memory is Loss(a t ), indicating critical information that may be lost when refreshing the memory; The dynamic programming solver is used to solve the memory refresh strategy based on the dynamic programming method, and determine the optimal action at each time step by calculating the following Bellman equation Among them, V(s t ) represents the value function of the current time step, γ is the discount factor, which reflects the impact of future time steps on the current decision; A strategy updating unit is used to update the optimal action at each time step t according to the result of the dynamic programming solving unit. Dynamically adjust the memory state and perform memory retention or refresh operations to ensure that the model can adaptively adjust memory management according to the current context and maximize context retention and processing efficiency during speech recognition.

6. The robot speech recognition system based on deep learning according to claim 1, characterized in that: The adaptive attention mechanism module includes: Weight initialization unit, used to initialize the weight α of each attention head in the multi-head self-attention mechanism at each time step t i , where i∈{1,2,...,h}, h is the number of attention heads, and the initial weights satisfy the constraints And α i ≥0; Attention loss calculation unit, used to calculate each attention head i Processing input feature x at time step t t The loss value f i (x t ), the loss value f i (x t ) is used to measure the performance of the attention head, indicating the error or mismatch degree of the attention head when processing speech features at the current time step; The objective function construction unit is used to construct the objective function of adaptive attention optimization based on the Lagrange multiplier method. The specific form is: Among them, λ is the Lagrange multiplier, which is used to ensure that the sum of the weights of each attention head is 1; The weight updating unit is used to optimize the objective function by gradient descent and calculate the weight α of each attention head. i The gradient is: And gradually optimize the weights through the following update rules: Among them, η is the learning rate. Through multiple iterations of optimization, the weight of each attention head α i Adaptive adjustment to ensure reasonable allocation of attention resources within the current time step t; The weight normalization unit is used to normalize the updated weight α after each iteration to ensure that the sum of the weights of all attention heads is still 1 to meet the constraints of the adaptive attention mechanism.

7. The robot speech recognition system based on deep learning according to claim 1, characterized in that: The output module comprises: A text conversion unit, used to convert the speech recognition result generated by the deep learning model module into text output; The command generation unit is used to convert the generated text information into robot control commands.

8. The robot speech recognition system based on deep learning according to claim 7, characterized in that: The text conversion unit specifically comprises the following steps: Receive the processed feature vector sequence y={y1,y2,…,y T }; Use the classification layer to classify each time step y t Decode the feature vector y t Mapping is done to the characters or words corresponding to speech recognition, and these characters or words are concatenated into continuous text.

9. The robot speech recognition system based on deep learning according to claim 7, characterized in that: The command generation unit specifically includes the following steps: Analyze the control commands or instructions contained in the text according to the preset instruction recognition model; Map the identified instructions to the corresponding robot control operations to generate a control command sequence C = {c1, c2, ..., c T }, used to control the robot to perform corresponding actions; If the text information contains multiple instructions, multiple control commands are generated in sequence according to preset priorities or sequence rules.

10. A robot speech recognition method based on deep learning, applied to the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: Collect external voice signals and perform noise reduction and sampling processing on them; Extract features from the preprocessed speech signal to generate Mel-spectrogram or MFCC features; Input the features into a deep learning model based on the Transformer-XL architecture, process the input using a multi-head self-attention mechanism, and capture contextual information based on memory units across time steps; Optimize memory refresh strategy through dynamic programming method and dynamically adjust memory status in deep learning models; Adaptively adjust the weight of each attention head based on the Lagrange multiplier method; Convert the output of the deep learning model into text or robot control instructions to complete the speech recognition process.

Images