Training methods for speech recognition models, speech recognition methods and related devices

Abstract

This application provides a training method for a speech recognition model, a speech recognition method, and related apparatus, belonging to the field of artificial intelligence technology. The method selects target words from the words in the training speech data based on the target position and size of a window, obtains keys based on the target words, obtains attention features based on the query, key, and value, classifies the attention features to obtain predicted text data, adjusts the position and size range of the window based on the predicted text data to obtain a trained speech recognition model, and randomly selects the target position of the window from the window's position range and the target size of the window from the window's size range, increasing the model's non-linearity, reducing computational overhead and storage consumption, and facilitating the model's handling of long sequence inputs; it can adjust the window size and interaction method according to data characteristics and task requirements, improving the model's adaptability and generalization ability on different datasets and tasks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to training methods for speech recognition models, speech recognition methods, and related devices. Background Technology

[0002] In speech recognition tasks, speech recognition models utilize attention mechanisms to focus on key features of the input speech data, improving the efficiency and accuracy of the speech recognition model. The attention mechanism requires weighted summarization of the input feature sequence and calculation of relevance scores between each pair of words at different positions. For example, for an input speech sequence of length n, the space and time complexity of the attention mechanism are both O(n^2) because it needs to store a matrix of relevance scores between all positions and calculate the relevances involved in all positions. For example, for an input sequence of length 100, a 100×100 relevance score matrix with 10,000 elements needs to be stored, and 10,000 relevance score calculations need to be performed. For longer input sequences, this space and time complexity increases quadratically, increasing memory consumption and reducing computational efficiency. The increase in space and time complexity limits the ability of attention-based speech recognition models to process long sequences of speech data. When processing long sequences of speech data, memory requirements and computational overhead become very high, thus reducing the scalability and computational efficiency of speech recognition models. Summary of the Invention

[0003] The main objective of this application is to propose a training method for a speech recognition model, a speech recognition method, and related devices, which reduces computational overhead and storage consumption, and is beneficial for the model to handle long sequence inputs.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for training a speech recognition model, the method comprising:

[0005] Acquire training speech data and input the training speech data into the initial speech recognition model;

[0006] The training speech data is segmented to obtain multiple word units;

[0007] The query and value of the attention mechanism are obtained based on the lexical units;

[0008] Target words are selected from the words based on the target position and target size of the window, and the key of the attention mechanism is obtained based on the target words. The number of target words is less than the number of words. The target position of the window is randomly selected from the window's position range, and the target size of the window is randomly selected from the window's size range.

[0009] Attention features are derived from the query, the key, and the value;

[0010] Based on the attention features, the predicted text data is obtained through classification.

[0011] The position and size of the window are adjusted based on the predicted text data to obtain a trained speech recognition model.

[0012] In some embodiments, obtaining the query and value of the attention mechanism based on the lexical units includes:

[0013] The query using the attention mechanism is obtained by multiplying the terminology and the first weight parameter corresponding to the query.

[0014] The value of the attention mechanism is obtained by multiplying the word and the value by the second weight parameter.

[0015] In some embodiments, obtaining the key of the attention mechanism based on the target lexical includes:

[0016] The key of the attention mechanism is obtained by multiplying the target lexical and the third weight parameter corresponding to the key.

[0017] In some embodiments, obtaining attention features based on the query, the key, and the value includes:

[0018] The attention score is obtained by adding the position code to the product of the query and the key;

[0019] Attention features are obtained by multiplying the attention score by the value.

[0020] In some embodiments, the positional encoding corresponding to the i-th attention score is represented as: in, PE i Let be the positional encoding corresponding to the i-th attention score, k be a positive integer greater than or equal to 1, and d be the dimension of the training speech data.

[0021] In some embodiments, classifying based on the attention features to obtain predicted text data includes:

[0022] Based on the attention features, the probability values ​​of each word in the vocabulary corresponding to the attention features are obtained;

[0023] The target word is determined from the words in the lexicon based on the probability value.

[0024] The predicted text data is composed of the target words.

[0025] To achieve the above objectives, a second aspect of this application provides a speech recognition method, which includes:

[0026] Acquire the speech data to be recognized;

[0027] The speech data to be recognized is input into the trained speech recognition model for speech recognition to obtain the speech recognition result;

[0028] The trained speech recognition model is obtained by training according to the speech recognition model training method described in the first aspect of the embodiments of this application.

[0029] To achieve the above objectives, a third aspect of this application provides a training apparatus for a speech recognition model, the training apparatus comprising:

[0030] The input module is used to acquire training speech data and input the training speech data into the initial speech recognition model.

[0031] The word segmentation module is used to segment the training speech data to obtain multiple word units;

[0032] The attention module is used to obtain the query and value of the attention mechanism based on the word, select the target word from the word through a window, obtain the key of the attention mechanism based on the target word, and obtain the attention feature based on the query, the key and the value. The number of target words is less than the number of words.

[0033] The classification module is used to classify the text based on the attention features to obtain the predicted text data;

[0034] The parameter tuning module is used to adjust the position and size of the window based on the predicted text data to obtain a trained speech recognition model.

[0035] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the training method for the speech recognition model as described in the first aspect of the present application and the speech recognition method as described in the second aspect of the present application.

[0036] To achieve the above objectives, a fifth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for the speech recognition model as described in the first aspect of the present application and the speech recognition method as described in the second aspect of the present application.

[0037] The speech recognition model training method, speech recognition method, and related apparatus proposed in this application involve segmenting training speech data to obtain multiple word units; obtaining the query and value of the attention mechanism based on the word units; randomly selecting the target position of the window from the window's position range and the target size of the window from the window's size range; selecting the target word unit from the word units of the training speech data based on the target position and target size of the window; obtaining the key based on the target word unit; obtaining attention features based on the query, key, and value; classifying the attention features to obtain predicted text data; and adjusting the position and size range of the window based on the predicted text data to obtain the trained speech recognition model. By randomly generating the attention window range and randomizing the interaction, the nonlinearity of the model is increased, enabling the model to better represent complex input correlations; computational overhead and storage consumption are reduced, which is beneficial for the model to handle long sequence inputs; the randomization of the attention mechanism can increase the model's robustness to uncertainty and noise; and the window size and interaction mode can be flexibly adjusted according to data characteristics and task requirements in different scenarios, improving the model's adaptability and generalization ability on different datasets and tasks. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the steps of the training method for the speech recognition model provided in an embodiment of this application;

[0039] Figure 2 This is a diagram of the sub-steps of step S300 provided in the embodiments of this application;

[0040] Figure 3 This is a diagram of the sub-steps for obtaining the key of the attention mechanism based on the target lexical provided in an embodiment of this application;

[0041] Figure 4 This is a sub-step diagram of step S500 provided in the embodiments of this application;

[0042] Figure 5 This is a flowchart illustrating the steps of the speech recognition method provided in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the attention mechanism provided in an embodiment of this application;

[0044] Figure 7 This is a structural diagram of the training device for the speech recognition model provided in the embodiments of this application;

[0045] Figure 8 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] In speech recognition tasks, speech recognition models utilize attention mechanisms to focus on key features of the input speech data, improving the efficiency and accuracy of the speech recognition model. The attention mechanism requires weighted summarization of the input feature sequence and calculation of relevance scores between each pair of words at different positions. For example, for an input speech sequence of length n, the space and time complexity of the attention mechanism are both O(n^2) because it needs to store a matrix of relevance scores between all positions and calculate the relevances involved in all positions. For example, for an input sequence of length 100, a 100×100 relevance score matrix with 10,000 elements needs to be stored, and 10,000 relevance score calculations need to be performed. For longer input sequences, this space and time complexity increases quadratically, increasing memory consumption and reducing computational efficiency. The increase in space and time complexity limits the ability of attention-based speech recognition models to process long sequences of speech data. When processing long sequences of speech data, memory requirements and computational overhead become very high, thus reducing the scalability and computational efficiency of speech recognition models.

[0050] For input speech features T is the length of the speech feature, D is the embedding dimension of the word, and the query and key are obtained based on the input speech features, according to formula A. i =q i ×K T +positional_encoding obtains the i-th row element of the attention score matrix from the query and key, and the matrix consists of multiple rows of elements. Each row of the attention score matrix consists of elements from each query vector. Bond matrix The query vector and the relationships between different word positions are obtained through multiplication. Each q-vector is multiplied by the K matrix, requiring T×2D floating-point computations. The calculation of the attention matrix A involves T q-vectors, and the size of K is... Therefore, both the time complexity and computational complexity are O(T). 2 ).

[0051] To address the above issues, embodiments of this application provide a training method for a speech recognition model, a speech recognition method, and related apparatus, thereby improving the model's adaptability and generalization ability across different datasets and tasks.

[0052] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0053] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0054] The speech recognition model training method, speech recognition method, and related apparatus provided in this application relate to the field of artificial intelligence technology. The speech recognition model training method and speech recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech recognition model training method and speech recognition method, but is not limited to the above forms.

[0055] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0056] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant regulations. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the normal operation of embodiments of this application obtained.

[0057] This application provides a method for training a speech recognition model, a speech recognition method, and related apparatus. The specific details are illustrated in the following embodiments, starting with a description of the training method for the speech recognition model in this application.

[0058] This application provides a method for training a speech recognition model. (Refer to...) Figure 1 The training method for a speech recognition model includes the following steps:

[0059] Step S100: Obtain training speech data and input the training speech data into the initial speech recognition model;

[0060] Step S200: Perform word segmentation on the training speech data to obtain multiple word units;

[0061] Step S300: Obtain the query and value of the attention mechanism based on the word elements;

[0062] Step S400: Select target words from word groups according to the target position and target size of the window, and obtain the key of the attention mechanism based on the target words;

[0063] Step S500: Obtain attention features based on the query, key, and value;

[0064] Step S600: Classify according to attention features to obtain predicted text data;

[0065] Step S700: Adjust the position and size of the window based on the predicted text data to obtain the trained speech recognition model.

[0066] In this embodiment, instead of using all the words obtained from segmenting the training speech data as keys to calculate attention features with the query, the learnable window selects only a subset of words as keys to calculate attention features with the query. This increases the nonlinearity of the speech recognition model, enhances its robustness to uncertainty and noise, and enables it to better represent complex input correlations. Furthermore, by flexibly adjusting the window's position and size according to different scenarios, the model can adapt to the requirements of specific tasks and the characteristics of the dataset. This allows the model to have different focuses and emphases for different inputs, thereby improving its adaptability and generalization ability across different datasets and tasks, and ultimately enhancing its flexibility.

[0067] The following provides a detailed explanation of steps S100 to S700 in the training method of the speech recognition model.

[0068] In step S100 of some embodiments, training speech data is acquired and input into an initial speech recognition model. The training speech data can come from a large speech database or from speech data collected for matching downstream tasks. Training the speech recognition model with a large amount of speech data from a large speech database enables the speech recognition model to have better speech recognition capabilities. Training the speech recognition model with speech data collected for matching downstream tasks enables the speech recognition model to have better speech recognition capabilities for downstream tasks.

[0069] The training speech data is labeled with the real values. For example, if the speech information of the training speech data is "You are so beautiful", then the training speech data is labeled with the text information "You are so beautiful".

[0070] In step S200 of some embodiments, the training speech data is segmented to obtain multiple tokens. Tokenization is the process of converting data to be processed into a numerical form that a computer can understand. The process of tokenization is the process of breaking down the data to be processed into smaller units, which are usually called tokens.

[0071] First, the training speech data is preprocessed. For example, noise is removed through denoising; sampling rate conversion ensures a uniform sampling rate across different training speech data sets; pre-emphasis enhances the high-frequency components of the speech and flattens the signal; and windowing divides the training speech data into small frames, such as a Hamming window, with each frame typically lasting 20-40 milliseconds. Feature extraction is performed on each frame, using methods such as Mel-Frequency Cepstral Coefficients (MFCC) and Perceptual Linear Prediction (PLP). Voice Activity Detection (VAD) is used to detect the start and end points of speech, and to identify speech segments and silence segments to exclude silence and non-speech parts. Lexical boundary detection methods are used to detect lexical boundaries, and word segmentation is performed based on these boundaries to obtain multiple lexical units. For example, energy-based methods can be used to detect abrupt changes in speech energy to locate lexical boundaries, or phoneme-based methods can be used to predict lexical boundaries using phoneme models.

[0072] In some embodiments, an energy-based method is used to detect abrupt changes in speech energy to locate lexical boundaries, and word segmentation is performed based on these boundaries to obtain multiple lexical units. For each frame of speech signal, its energy is calculated. The energy can be calculated using the following formula: Where E[n] represents the energy of the nth frame of speech signal, x[i,n] represents the i-th sample of the nth frame of speech signal, and N represents the frame size. To reduce the influence of different speakers, environments, and devices, the energy is normalized. An energy threshold is determined empirically or experimentally to identify energy abrupt changes. Energy abrupt changes are detected by comparing the energy differences between consecutive frames. If the energy difference between two consecutive frames exceeds a preset threshold, a word boundary is considered to exist. To reduce the impact of noise and short-term fluctuations, the detected boundaries can be smoothed; for example, median filtering or dynamic programming algorithms can be used to optimize the boundary positions. After determining the word boundaries, word segmentation is performed based on the word boundaries to obtain multiple word units.

[0073] In some embodiments, using a phoneme-based approach to predict word boundaries typically involves using a phoneme model to identify specific phoneme transition points in the speech signal, which generally correspond to word boundaries. The main steps in using a phoneme-based approach to predict word boundaries include: phoneme model training, speech signal preprocessing, phoneme identification, word boundary detection, and boundary optimization. In the phoneme model training step, a large amount of annotated speech data is collected, containing phoneme-level annotations; features such as Mel-frequency cepstral coefficients and perceptual linear prediction are extracted from the speech signal. The extracted features and annotated data are used to train the phoneme model, which is typically based on a Hidden Markov Model or a deep learning model (such as a Recurrent Neural Network or Long Short-Term Memory Network). In the speech signal preprocessing step, sampling rate conversion ensures a consistent sampling rate for the speech signal; pre-emphasis enhances the high-frequency components of the speech signal; framing divides the speech signal into a series of short frames; and feature extraction extracts acoustic features from each frame. In the phoneme recognition step, phoneme transition points are detected in the phoneme sequence. These transition points may be the locations of word boundaries. An empty list is first set up to store the detected word boundaries. Then, the phoneme sequence is traversed, checking for changes in consecutive phonemes. When a phoneme change is detected, the position of the current frame is recorded as a potential word boundary. In the boundary optimization step, smoothing techniques are used to reduce false detections and smooth the boundaries, such as through dynamic programming or Gaussian smoothing. Contextual information is considered to confirm the boundaries, for example, using a language model to evaluate the reasonableness of the boundaries, further accurately determining the boundaries. After determining the word boundaries, word segmentation is performed based on the word boundaries to obtain multiple word units.

[0074] Reference Figure 6 Attention mechanisms are techniques used in neural networks to enhance a model's ability to focus on important information. The core idea is to allow the model to dynamically select and focus on different parts of the input data based on the needs of the current task. Attention mechanisms mimic the human visual attention mechanism, focusing on important parts of information while ignoring less important parts; that is, assigning different levels of attention to different parts of the input information. Attention mechanisms calculate the importance weight of each element in the input sequence and then perform a weighted summation of these weights to obtain a new representation. This method allows models to more flexibly select and focus on important information when processing long sequences of data.

[0075] In step S300 of some embodiments, the query and value of the attention mechanism are obtained based on the lexical units. The lexical units can be represented as: T is the length of the acoustic feature, and D is the embedding dimension of each lexical unit.

[0076] Reference Figure 2 The attention mechanism obtains queries and values ​​based on lexical units, including the following steps:

[0077] Step S310: Obtain the query of the attention mechanism based on the product of the lexical and the first weight parameter corresponding to the query;

[0078] Step S320: Obtain the value of the attention mechanism based on the product of the second weight parameter corresponding to the word and the value.

[0079] The query in an attention mechanism is typically a variable or a set of variables that represents the query posed by the data point currently being processed. For example, when processing a speech signal, the query is the word in the speech signal that needs attention or processing. The purpose of this query is to find the most relevant part from other data points. The query is used to calculate the similarity or relevance with other keys, and these similarity scores determine how much weight should be assigned to each value when synthesizing the final output representation of the attention mechanism.

[0080] The query for the attention mechanism can be represented by the following formula: q = x n *w q Where q represents the query of the attention mechanism; x n w represents the nth sample in the word group. q This indicates that the query is for the corresponding first weight parameter.

[0081] have

[0082] The values ​​in an attention mechanism are data points associated with a key. In an attention mechanism, values ​​are typically feature vectors extracted from the original input data, representing different aspects or attributes of the input data. The role of the values ​​in the attention mechanism is to provide detailed information related to the key, which is used to generate the final output representation of the attention mechanism. During the computation of the attention mechanism, the values ​​are weighted and summed with attention scores calculated from the query and key to obtain a weighted value. This weighted value is the output of the attention mechanism, containing the most relevant parts of the original input data.

[0083] The value of the attention mechanism can be expressed by the following formula: v = x n *w v Where v represents the value of the attention mechanism; x n w represents the nth sample in the word group. v This represents the second weight parameter corresponding to the value. The values ​​of the attention mechanism are in matrix form, represented as follows:

[0084] In step S400 of some embodiments, target tokens are selected from the tokens based on the target position and target size of the window, and keys for the attention mechanism are obtained according to the target tokens.

[0085] The keys of the attention mechanism are data points used for comparison with queries. In the attention mechanism, keys are usually feature vectors extracted from the original input data, and these feature vectors represent different aspects or attributes of the input data. The role of the keys of the attention mechanism in the attention mechanism is to help determine the values most relevant to the queries. By comparing the similarity or matching degree between the queries and each key, attention scores can be calculated. These attention scores determine the weights that should be assigned to each value when synthesizing the final output representation of the attention mechanism.

[0086] The target position of the window is randomly selected from within the position range of the window, and the target size of the window is randomly selected from within the size range of the window. For example, the position range of the window is to start selecting from the 2nd sample of the tokens to starting from the 5th sample of the tokens, and the size range of the window is to select 3 samples to selecting 6 samples. A random number 3 is generated by the random number generator, and the target position of the window is to start selecting from the 3rd sample of the tokens; a random number 4 is generated by the random number generator, and the target size of the window is to select 4 samples. By using the random number generator to generate the position and size of each window, it is ensured that the position and size of each window are within the predefined window position range and size range.

[0087] Target tokens are selected from the tokens x=(x1,…,x T ) through the window. The size of the window is less than T to ensure that the number of target tokens is less than the number of tokens. The size of the window is win, where 0 < win < T, and win target tokens are selected from the tokens. win is a learnable parameter, and as the network continuously learns, it will find appropriate window values for different tasks or datasets. Furthermore, the matrix composed of the keys of the attention mechanism is represented as Specifically, when the size of the window is 3 and the position of the window is to start sampling from the second sample, the target tokens selected from the tokens through this window are x2, x3, and x4. In particular, for any row vector in the K matrix, it is any token randomly sampled from the input features.

[0088] Refer to Figure 3 , specifically, obtaining the keys of the attention mechanism according to the target tokens includes:

[0089] Step S410, obtaining the keys of the attention mechanism according to the product of the target tokens and the third weight parameter corresponding to the keys.

[0090] The keys of the attention mechanism can be represented by the following formula: k = x n * wk Where k represents the key of the attention mechanism; x n w represents the nth sample in the word group. v This represents the third weight parameter corresponding to the key. Compared to the traditional method of calculating the key based on all samples in a word, the method of selecting the target word from the word through a window and obtaining the key of the attention mechanism based on the target word can improve efficiency and reduce time and computational complexity.

[0091] Reference Figure 4 In step S500 of some embodiments, attention features are obtained based on the query, key, and value, including the following steps:

[0092] Step S510: Add the position code to the product of the query and the key to obtain the attention score;

[0093] Step S520: Obtain the attention features based on the product of the attention score and the value.

[0094] Attention score can be expressed as: A i =q i ×K T +positional_encoding; where q represents the i-th attention score. i Let K represent the i-th query vector of the attention mechanism, K represent the matrix consisting of the keys of the attention mechanism, and positional_encoding represent the positional encoding.

[0095] For each Vectors, without needing to interact with the entire context, only need to interact with the target key. Specifically, for input features... Arbitrary query vector Extracted through a window (where x′1,…,x′) win The dot product operation of the matrix (∈x) yields win values. Each of these win values ​​has row i as its row value and column i as the label of each vector in each of the K matrices; values ​​other than the label are defaulted to 0. This is a more efficient method compared to the traditional method of multiplying each query vector by... Matrix multiplication, multiplying each query vector by... Matrix multiplication can improve efficiency and reduce time and computational complexity.

[0096] To prevent positional shifts, a positional code is added to each element (the product of the query and the key). The positional code corresponding to the i-th attention score is represented as: in, PE i Let be the positional encoding corresponding to the i-th attention score, k be a positive integer greater than or equal to 1, and d be the dimension of the training speech data.

[0097] The matrix of attention scores is sparse.

[0098] Perform a dot product operation between the attention score and the value. The attention feature is obtained from this product. The attention feature Y can then be expressed as: Y = A * V.

[0099] In step S600 of some embodiments, the predicted text data is obtained by classifying based on attention features.

[0100] Specifically, attention features are passed through activation layers, pooling layers, fully connected layers, and softmax layers.

[0101] By applying non-linear transformations to attention features through activation layers, the network can better learn and process complex data, enhancing the model's expressive power. Non-linear activation functions allow the network to combine and transform features at multiple levels, thereby capturing complex structures and patterns in the input data. Non-linear activation functions also help the network better propagate gradients during training, which is crucial for learning via backpropagation. For example, activation layers can use functions such as the Sigmoid, Tanh, and ReLU.

[0102] Pooling layers downsample attention features, reducing data dimensionality and computational cost while preserving important feature information. Pooling layers reduce the width and height of attention features, thereby lowering data dimensionality. This helps reduce the number of parameters in subsequent layers, reducing computational complexity and preventing overfitting. Pooling operations increase the tolerance of features to small transformations, making the network somewhat invariant to small translations, rotations, or deformations of the input image. Commonly used pooling functions include max pooling and average pooling. Max pooling uses the maximum value of a certain number of pixels in the convolutional layer's output as the pooling layer's output. Average pooling uses the average value of a certain number of pixels in the convolutional layer's output as the pooling layer's output. This downsampling operation of pooling layers reduces data dimensionality and computational cost while preserving important features, improving the network's generalization ability.

[0103] Fully connected layers integrate the features extracted from previous layers and map them onto the classification results. The computation process of a fully connected layer involves element-wise multiplication of the outputs of each preceding layer with the weights of the fully connected layer, summing the results, and adding a bias term. Through its internal weights and biases, the fully connected layer can learn the complex non-linear relationships between the input features. For example, a fully connected layer maps features to classification results for the texts "you," "you all," and "we."

[0104] The softmax layer transforms the scores of the fully connected layer for each category into a probability distribution, where the probability of the i-th category is represented as: Where zi represents the score of the fully connected layer for the i-th category, zj represents the score of the fully connected layer for the j-th category, and e zi It is the exponent of zi, e zj It is the index of zj, y i This represents the probability of the i-th category. For example, the probability of the text corresponding to "you" obtained through the softmax layer is 0.5, the probability of the text corresponding to "you all" is 0.4, and the probability of the text corresponding to "we" is 0.1.

[0105] The probabilities of each category output by the softmax layer are sorted, and the category with the highest probability is selected as the predicted text data. For example, the probability of the text corresponding to "you" is 0.5, the probability of the text corresponding to "you all" is 0.4, and the probability of the text corresponding to "we" is 0.1. The category with the highest probability is the text corresponding to "you", so the text corresponding to "you" is selected as the predicted text data.

[0106] In step S700 of some embodiments, the position and size range of the window are adjusted according to the predicted text data to obtain a trained speech recognition model.

[0107] Specifically, a loss function is calculated based on the true values ​​corresponding to the predicted text data and the training speech data. The parameters of the speech recognition model are then adjusted by reverse calculation based on the loss function. The parameters of the speech recognition model include the position range and size range of the window, resulting in a trained speech recognition model.

[0108] For example, the loss function can be one or more of the classification cross-entropy function, attention loss function, and mean squared error function.

[0109] Traditional attention mechanisms typically use fixed window sizes and interaction methods, limiting the model's expressive power. However, methods that randomly generate attention window sizes and sizes to achieve random interactions introduce more non-linear capabilities, enabling the model to better represent complex input correlations. By randomly generating attention window sizes and random interactions, the model can flexibly adjust the window size and interaction methods in different scenarios. This flexibility allows the model to adapt to the requirements of specific tasks and the characteristics of the dataset, improving the model's sensitivity and performance.

[0110] The interaction between each token and other tokens is represented by a generated attention score matrix, where each row contains only 5 non-zero attention scores, with scores of zero at other positions, and the positions of the non-zero scores in each row are randomly selected. This method of randomly generating the attention window range and position can increase the model's non-linear capabilities. By introducing randomness in attention, the model's performance can be improved by better capturing the correlations between the input speech data. Randomization of the attention mechanism can increase the model's robustness to uncertainty and noise. Randomly generating the attention window range and random interactions allows the model to have different focuses and emphases for different inputs, thereby improving the model's adaptability and generalization ability on different datasets and tasks. For each token within the window, other tokens within the window or globally can be randomly selected for interaction. This can be achieved by randomly selecting the token's index and setting a non-zero value at the corresponding position in the attention score matrix.

[0111] This application provides a speech recognition method. (Refer to...) Figure 5 The speech recognition method includes the following steps:

[0112] Step S800: Obtain the speech data to be recognized.

[0113] Step S900: Input the speech data to be recognized into the trained speech recognition model to perform speech recognition and obtain the speech recognition result.

[0114] The trained speech recognition model is obtained by training according to the speech recognition model training method described above.

[0115] In step S800 of some embodiments, voice data to be identified is collected by a recording device, and the voice data is not labeled.

[0116] In step S900 of some embodiments, the speech data to be recognized is input into a trained speech recognition model for speech recognition.

[0117] In speech recognition models, the speech data to be recognized undergoes preprocessing. For example, noise is removed through denoising; sampling rate conversion ensures a uniform sampling rate across different speech data sets; pre-emphasis enhances the high-frequency components and flattens the signal; and windowing divides the speech data into small frames, such as a Hamming window, with each frame typically lasting 20-40 milliseconds. Feature extraction is performed on each frame, using methods such as Mel-frequency cepstral coefficients and perceptual linear prediction. Speech activity detection identifies the start and end points of speech and distinguishes between speech segments and silences to exclude quiet and non-speech parts. Lexical boundary detection methods detect word boundaries, and word segmentation is performed based on these boundaries to obtain multiple words. For example, energy-based methods detect abrupt changes in speech energy to locate word boundaries, or phoneme-based methods utilize phoneme models to predict word boundaries.

[0118] Reference Figure 6 The query obtained through the attention mechanism is derived by multiplying the term and the first weight parameter corresponding to the query. The query through the attention mechanism can be represented by the following formula: q = x n *w q Where q represents the query of the attention mechanism; x n w represents the nth sample in the word group. q This indicates that the query is for the corresponding first weight parameter.

[0119] have

[0120] The value of the attention mechanism is obtained by multiplying the second weight parameter corresponding to the word and its value. The value of the attention mechanism can be expressed by the following formula: v = x n *w v Where v represents the value of the attention mechanism; x n w represents the nth sample in the word group. v This represents the second weight parameter corresponding to the value. The values ​​of the attention mechanism are in matrix form, represented as follows:

[0121] Randomly select the target position of the window from within the position range of the window, and randomly select the target size of the window from within the size range of the window. The position range of the window and the size range of the window are obtained during the training process of the speech recognition model. For example, the position range of the window is to start selecting from the 2nd sample of the token to starting from the 5th sample of the token, and the size range of the window is to select 3 samples to selecting 6 samples. A random number 3 is generated by the random number generator, and the target position of the window is to start selecting from the 3rd sample of the token; a random number 4 is generated by the random number generator, and the target size of the window is to select 4 samples. By using the random number generator to generate the position and size of each window, it is ensured that the position and size of each window are within the predefined window position range and size range.

[0122] Select the target tokens from the token x=(x1,…,x T ) through the window. The size of the window is less than T to ensure that the number of target tokens is less than the number of tokens. The size of the window is win, where 0 < win < T, and win target tokens are selected from the tokens.

[0123] The key of the attention mechanism can be expressed by the following formula: k = x n *w<000​​​​​​​​​​​​​​​​​​​​​​​​​​ Arbitrary query vector Extracted through a window (where x′1,…,x′) win The dot product operation of the matrix (∈x) yields win values. The row value of each of the win values ​​is i, and the column value is the label of each vector in each of the K matrices; values ​​other than the label are defaulted to 0.

[0125] The attention feature is obtained by multiplying the attention score and the value. Then, the attention feature Y can be expressed by the following formula: Y = A * V.

[0126] Attention features are processed through activation layers, pooling layers, fully connected layers, and softmax layers for classification to convert speech sequences into text sequences. The activation layer performs a non-linear transformation on the attention features, the pooling layer downsamples the attention features, the fully connected layer integrates the features extracted from previous layers and maps them onto the classification results, and the softmax layer converts the scores of each category from the fully connected layer into a probability distribution. The probabilities of each category are then ranked according to the output of the softmax layer, and the category with the highest probability is selected as the speech recognition result.

[0127] For example, the word "thank you" contained in the speech data to be recognized is mapped to the classification results of "crab" text, "thank you" text, and "write more" text through a fully connected layer; the softmax layer yields a probability of 0.1 for the corresponding "crab" text, 0.6 for the corresponding "thank you" text, and 0.3 for the corresponding "write more" text; the category with the highest probability is "thank you" text, and "thank you" text is selected as the speech recognition result.

[0128] In this embodiment, by randomly generating the attention window range and randomizing the interaction, the nonlinearity of the model is increased, enabling the model to better represent complex input correlations; computational overhead and storage consumption are reduced, which is beneficial for the model to handle long sequence inputs; randomization of the attention mechanism can increase the model's robustness to uncertainty and noise; and the window size and interaction mode can be flexibly adjusted according to data characteristics and task requirements in different scenarios, improving the model's adaptability and generalization ability on different datasets and tasks.

[0129] Embodiments of this application propose a training apparatus for a speech recognition model. (Refer to...) Figure 7 The training device for the speech recognition model includes: an input module 801, a word segmentation module 802, an attention module 803, a classification module 804, and a parameter tuning module 805.

[0130] The input module 801 is used to acquire training speech data and input the training speech data into the initial speech recognition model; the word segmentation module 802 is used to segment the training speech data into multiple word units; the attention module 803 is used to obtain the query and value of the attention mechanism based on the word units, select the target word unit from the word units through the window, obtain the key of the attention mechanism based on the target word unit, and obtain the attention features based on the query, key and value, where the number of target word units is less than the number of word units; the classification module 804 is used to classify based on the attention features to obtain the predicted text data; the parameter tuning module 805 is used to adjust the position and size of the window based on the predicted text data to obtain the trained speech recognition model.

[0131] It is understood that the speech recognition model training apparatus of the embodiments of this application applies the speech recognition model training method described above. The modules of the speech recognition model training apparatus of the embodiments of this application correspond to the steps of the speech recognition model training method. The speech recognition model training apparatus and the speech recognition model training method adopt the same technical means, solve the same technical problems, and bring about the same technical effects, which will not be repeated here.

[0132] This application also provides an electronic device. (Refer to...) Figure 8 The electronic device includes a memory 902 and a processor 901. The memory 902 stores a computer program, and the processor 901 executes the computer program to implement the training method and speech recognition method of the aforementioned speech recognition model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0133] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented using read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the training method and speech recognition method of the speech recognition model in the embodiments of this application.

[0134] The input / output interface 903 is used to realize information input and output; the communication interface 904 is used to realize communication interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.); the bus 905 transmits information between various components of the device (such as processor 901, memory 902, input / output interface 903 and communication interface 904); wherein the processor 901, memory 902, input / output interface 903 and communication interface 904 realize communication connection between each other within the device through the bus 905.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described training method for the speech recognition model and the speech recognition method.

[0136] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0138] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0141] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0142] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0144] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training a speech recognition model, characterized in that, include: Acquire training speech data and input the training speech data into the initial speech recognition model; The training speech data is segmented to obtain multiple word units; The query and value of the attention mechanism are obtained based on the lexical units; Target words are selected from the words based on the target position and target size of the window, and the key of the attention mechanism is obtained based on the target words. The number of target words is less than the number of words. The target position of the window is randomly selected from the window's position range, and the target size of the window is randomly selected from the window's size range. Attention features are derived from the query, the key, and the value; Based on the attention features, the predicted text data is obtained through classification. The position and size of the window are adjusted based on the predicted text data to obtain a trained speech recognition model.

2. The training method for the speech recognition model according to claim 1, characterized in that, The step of obtaining the query and value of the attention mechanism based on the lexical units includes: The query using the attention mechanism is obtained by multiplying the terminology and the first weight parameter corresponding to the query. The value of the attention mechanism is obtained by multiplying the word and the value by the second weight parameter.

3. The training method for the speech recognition model according to claim 1, characterized in that, The key for obtaining the attention mechanism based on the target lexical includes: The key of the attention mechanism is obtained by multiplying the target lexical and the third weight parameter corresponding to the key.

4. The training method for the speech recognition model according to claim 1, characterized in that, The process of obtaining attention features based on the query, the key, and the value includes: The attention score is obtained by adding the position code to the product of the query and the key; Attention features are obtained by multiplying the attention score by the value.

5. The training method for the speech recognition model according to claim 4, characterized in that, The positional encoding corresponding to the i-th attention score is represented as: in, PE i Let be the positional encoding corresponding to the i-th attention score, k be a positive integer greater than or equal to 1, and d be the dimension of the training speech data.

6. The training method for the speech recognition model according to claim 1, characterized in that, The step of classifying based on the attention features to obtain predicted text data includes: Based on the attention features, the probability values ​​of each word in the vocabulary corresponding to the attention features are obtained; The target word is determined from the words in the lexicon based on the probability value. The predicted text data is composed of the target words.

7. A speech recognition method, characterized in that, include: Acquire the speech data to be recognized; The speech data to be recognized is input into the trained speech recognition model for speech recognition, and the speech recognition result is obtained. The trained speech recognition model is obtained by training the speech recognition model according to any one of claims 1 to 6.

8. A training device for a speech recognition model, characterized in that, include: An input module is used to acquire training speech data and input the training speech data into an initial speech recognition model; The word segmentation module is used to segment the training speech data to obtain multiple word units; The attention module is used to obtain the query and value of the attention mechanism based on the word, select the target word from the word through a window, obtain the key of the attention mechanism based on the target word, and obtain the attention feature based on the query, the key and the value. The number of target words is less than the number of words. The classification module is used to classify the text based on the attention features to obtain the predicted text data; The parameter tuning module is used to adjust the position and size of the window based on the predicted text data to obtain a trained speech recognition model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the training method of the speech recognition model according to any one of claims 1 to 6 and the speech recognition method according to claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the speech recognition model according to any one of claims 1 to 6 and the speech recognition method according to claim 7.

Images