Automatic voice recording method for hotword online customization update in a telephone scenario

Abstract

The application discloses an automatic voice recording method for hot word online customization and update in a telephone scenario, and comprises the following steps: firstly, an automatic speech recognition model based on a deep neural network is trained; then, the automatic speech recognition model is retrained through a telephone audio file to generate a pre-training model; a differentiated language model self-update model and a hot word table are customized, and the weight of a decoding stage is adjusted in real time to form a self-adaptive speech recognition hot word system. The application proposes a speech recognition system for online customization of hot words in a telephone customer service based on different scenarios, generates a hot word table in different scenarios, and improves the accuracy of the efficiency of voice transcription into text; a hot word weight real-time update algorithm is proposed, hot word frequency is counted online, model parameters are updated in real time in a decoding stage of an acoustic model according to the frequency, and 8K telephone audio is relearned in a manner of transfer learning, so that the accuracy of audio speech recognition in a telephone scenario is improved.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology, specifically relating to an automatic voice recording method for online customization and updating of hot words in telephone scenarios. Background technology:

[0002] In the telecommunications industry, such as telephone customer service and telephone inquiries, most communication is conducted via voice calls. Specifically, for industries like telephone customer service, call recordings are essential for backup and future review. While this method of backup is simple and easy, it consumes more storage space than text-based backups; furthermore, it hinders quick location of relevant information when reviewing the recordings.

[0003] Thanks to the emergence of deep learning technology, automatic speech recognition systems have developed rapidly, significantly improving the accuracy and speed of automatically converting speech signals into text. Therefore, transcribing recorded calls into text using automatic speech recognition systems can effectively reduce storage space and facilitate easy retrieval.

[0004] However, current automatic speech recognition models are trained on general datasets and often lack differentiation and customization for different domains and scenarios. How to customize speech recognition models according to different scenarios to improve accuracy is an urgent problem to be solved. Furthermore, current hot word customization technology in speech recognition only allows customization of hot words or nouns within specific domains, requiring manual addition, uploading, and updating of hot words. Manually collecting and customizing hot word lists for various industries is impractical, especially since some popular and personalized words are time-sensitive and specific. This means that even if the domain and scenario are consistent, an updated hot word list will render the old list unsuitable, leading to inaccurate speech recognition in certain scenarios. In addition, telephone audio files are mostly 8K sampling rate audio, while mainstream speech recognition models use 16K audio files; directly using these files will result in a decrease in accuracy. Summary of the Invention:

[0005] The technical problem to be solved by this invention is to provide a customizable voice recognition system for telephone scenarios in different industries, thereby enabling the customization of differentiated voice recognition systems and improving the accuracy of voice recognition.

[0006] To address the aforementioned issues, this invention provides an automatic voice recording method for online customization and updating of hot words in telephone scenarios. This method customizes hot words in real time for different scenarios and updates the weights in the speech recognition and decoding stage in real time using a hot word frequency algorithm, thereby resolving the above deficiencies.

[0007] A method for automatically recording and updating trending words online in telephone scenarios, comprising the following steps:

[0008] S1: Train an automatic speech recognition model based on a deep neural network using a publicly available general speech dataset:

[0009] The training of the automatic speech recognition model includes determining the basic parameters of the automatic speech recognition model; initializing the weights of each layer of the automatic speech recognition model; and determining the optimization method.

[0010] S2: Retrain the automatic speech recognition model using 8k telephone audio files collected in a telephone scenario to generate a pre-trained model of the automatic speech recognition model based on the 8K telephone scenario.

[0011] S3: Deploy and run the pre-trained model. During operation, the pre-trained model is used to recognize the voice in the call, and the obtained text data is stored for online hot word customization.

[0012] S4: Customize a differentiated language model and update the hot word list, and adjust the weights in the decoding stage in real time to form an adaptive speech recognition hot word system.

[0013] Preferably, the automatic speech recognition model adopts the Transformer model, which is based on the Seq2seq model of Attention and consists of an encoder and a decoder, and is trained through a multi-head attention module.

[0014] The specific training process for the Transformer model is as follows:

[0015] S1-1: First, the audio data in the publicly available general speech dataset is preprocessed and transformed into Fbank audio features. The width of the data is expanded by Embedding word embedding and then transformed into a three-dimensional tensor.

[0016] S1-2: The Transformer encoder does not process word position information. A position encoder is added before the tensor is imported into the Encoder layer to add information that different word positions may produce different semantics into the embedded 3D tensor to make up for the lack of position information.

[0017]

[0018] Among them, PE (pos,2i) PE (pos,2i+1) These are the 2i and 2i+1th components of the encoding vector at position pos, respectively, d model This represents the dimension of the word vector.

[0019] S1-3: The Transformer model inputs the preprocessed 3D tensor (after being processed by the position encoder) into the Encoder layer. The Attention layer is the main part of the Encoder layer, training the tensor W using three trainable array parameters Q, K, and V. To enable the Transformer model to focus on different aspects of information, it divides the tensor into h head vectors, forming multiple subspaces. It then projects these subspaces onto Q, K, and V through h linear transformations, finally concatenating the different Attention results. The formula is as follows:

[0020]

[0021] Head t =Attention(QW t Q KW t K VW t V )

[0022] MultiHead(Q,K,V)=Concat(Head1,...,Head t )

[0023] Where dk represents the dimension of the model, and Head t MultiHead represents the output of multi-head attention for the t-th head vector, and Concat represents the concatenation operation on the model.

[0024] S1-4: The Feed Forward layer transforms the feature vectors extracted by Attention, increasing the model's expressive power. The Feed Forward layer has two layers: the first layer uses ReLU activation, and the second layer uses a linear activation function, as shown in the following formula:

[0025] FFN = Max(0, xW1+b1)W2+b2

[0026] Where x represents the output of the previous layer, and W1, W2, b1, b2 are trainable parameters.

[0027] After training the Encoder layer, the resulting 3D tensor is used as the input to the K and V layers of the Decoder layer.

[0028] S1-5: Embedding word embeddings are performed on the text files corresponding to the audio data in the publicly available general speech dataset. The data is then expanded into a three-dimensional tensor, and positional encoding is added before being imported into the Decoder layer. The Decoder layer feeds the three-dimensional tensor into the Attention layer and the Feed Forward layer for training. The Q of the Decoder layer comes from the input of the Decoder layer, and K and V come from the output of the Encoder layer; the Q, K, and V of the Encoder all come from the input of the Encoder layer.

[0029] S1-6: After the Decoder layer, the 3D tensor output by the Decoder layer is passed into the linear layer and the Softmax layer to obtain text data. This data is then compared with the text files corresponding to the audio data in the general speech dataset for training. The loss function is calculated, and the difference between the forward calculation result and the true value of each iteration of the model is calculated to guide the next step of training in the correct direction until the model converges.

[0030] The specific method for S2 is as follows:

[0031] The automatic speech recognition model (referred to as the teacher model) trained on publicly available standard audio data with a sampling frequency of 16kHz using 8kHz telephone audio files collected in a telephone scenario is retrained and fine-tuned. The teacher model is then used to guide the training of the student model. During the training process, the student model is trained to approximate the posterior probability distribution of the teacher model as closely as possible. This generates a pre-trained model for the automatic speech recognition model based on the 8kHz telephone scenario.

[0032] The specific method for S4 is as follows:

[0033] S4-1: Segment the text data, remove stop words, and extract word frequencies.

[0034] S4-2: Normalize the word frequencies based on probability differences, using the word frequencies as follows:

[0035]

[0036] Wf represents word frequency, Wf * σ represents the normalized word frequency, μ represents the mean word frequency, and σ represents the standard deviation of word frequency.

[0037] S4-3: Extract the general hot word list and the differential hot word list for different scenarios, and determine the corresponding activation threshold and differential threshold based on the median word frequency. If a hot word's frequency in the current scenario exceeds the activation threshold but is less than the differential threshold across different scenarios, it is considered a differential hot word. If a word is considered a differential hot word, it means it only appears in one or a few specific scenarios. Weight adjustments are only made during the model's decoding stage in scenarios where the frequency exceeds the differential threshold. If a hot word's frequency in the current scenario exceeds both the activation threshold and the differential threshold across different scenarios, it is considered a general hot word. If a word is considered a general hot word, it means it appears in many scenarios. Weights are increased during the model's decoding stage in all scenarios. The determination formula is as follows:

[0038]

[0039] "word" refers to the word used for discrimination; "Hotword" is the word used for discrimination. common Hotword (general term) particulary These are differentiated hot words. Wf represents word frequency, AcTh represents activation threshold, and DiffTh represents difference threshold.

[0040] S4-4: When the system decodes audio to generate text data, it adjusts and optimizes the decoding results according to the hot word weight table. When the pinyin of the text output by the Transformer model's decoder is the same as a hot word in the hot word table, a hot word discrimination mode is performed, a new path containing the hot word is added, the offset score is added to the decoding score of this path, and the scoring is re-rating. The formula is as follows:

[0041] δ k =log(1+Wf)

[0042]

[0043] Where score(y1, ..., ys) is the final score on the path from the first character y1 to the last character ys in Beam Search decoding, P lm (y k |y1,...,y k-1 δ(x) represents the probability that the k-th character appears in the output. k This represents the offset fraction of the parameter.

[0044] The newly obtained path is compared with other paths, and the path with the highest score is selected as the optimal path. The text result is then saved. The saved text data is used to supplement the new corpus, and the activation threshold and differentiation threshold are updated online.

[0045] S4-5: Design the update frequency of the adaptive speech recognition hot word system, which can be adjusted by the user through the interval of the update weight or set to update automatically during idle time.

[0046] S4-6: This invention also provides an interactive hot word update method. Users can generate JSON files of their frequently used hot words in the format of "hot word-part-of-speech-weight level" and upload them to the system. The offset scores are adjusted according to the weight levels to customize hot words to meet individual needs. These customized hot words have initial weights, and the weights, activation thresholds, and difference thresholds can be dynamically adjusted during system use to improve efficiency in speech recognition and better recognize user speech.

[0047] The beneficial effects of this invention are as follows:

[0048] 1. In response to the problem that existing technologies can only customize hot words or nouns in specific fields, this invention proposes a voice recognition system for telephone customer service that can customize hot words online based on different scenarios, generating hot word lists for different scenarios, thereby improving the efficiency and accuracy of speech-to-text conversion.

[0049] 2. To address the issue that current methods cannot update hot word weights in real time due to the time-sensitivity and specificity of some popular and personalized words, this invention proposes a real-time hot word weight update algorithm that counts hot word frequencies online and updates model parameters in real time during the decoding stage of the acoustic model based on the word frequencies.

[0050] 3. To address the issue of the mismatch between the sampling rate of data such as telephone communication and that of mainstream speech recognition models, this invention employs transfer learning to relearn 8K telephone audio, thereby improving the accuracy of audio speech recognition in telephone scenarios. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0052] Figure 2 This is a schematic diagram of the speech recognition model process of the present invention. Detailed Implementation

[0053] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. The technical methods provided by various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] Reference Figure 1 The flowchart of the present invention is shown, which specifically includes the following steps:

[0055] S1: For the speech dataset, it is packaged and organized into text samples as corpus, and then imported into the speech recognition model. The speech dataset used in this patent is trained using the aishell1 dataset with a recording duration of 178 hours.

[0056] The speech dataset is preprocessed as follows;

[0057] S1.1: Obtain the address of audio data in the publicly available general speech dataset and generate an index file of "filename address";

[0058] S1.2: Obtain the text files corresponding to the voice files in the voice dataset and generate an index file of "filename voice text";

[0059] S1.3: Obtain the Chinese character index table in the text file corresponding to the speech file in the speech data, and introduce "BEG", "END", and "UKW" at the beginning, which refer to the beginning, end, and unknown word of the speech data, respectively.

[0060] S2: The Transformer model is used as the automatic speech recognition model. The Transformer model is based on the Seq2seq model with Attention and consists of an encoder and a decoder. The parameters and weights are initialized through a multi-head Attention module. The model consists of 6 layers of encoder and 6 layers of decoder. The number of iterations is set to 60 rounds. The Adam optimizer is used in the model.

[0061] Transformers are widely used in speech recognition models. Compared to CNN and RNN models, they can directly calculate the correlation between each word, offering advantages such as fast training speed and high recognition accuracy. The Transformer model mainly consists of Attention layers and Feed Forward layers. Figure 2 The structure of the Transformer model is shown. The specific training process for the Transformer model is as follows:

[0062] S2.1: The audio file is pre-emphasized, windowed, and framed to eliminate the impact of aliasing, high-order harmonic distortion, and high-frequency distortion caused by the human vocal organs themselves and the equipment used to acquire the speech signal. Fbank features with a length equal to the number of frames and a width of 80 are extracted. Word embedding processing is performed on the Fbank features through an embedding layer, keeping the Fbank feature length unchanged, and expanding it into a matrix with a width of 280, transforming it into a three-dimensional tensor.

[0063] S2.2: Positional encoding is applied to the feature vectors. This involves adding a positional encoder before the audio file is imported into the Encoder layer. Information about the different semantics arising from different word positions is incorporated into the embedded 3D tensor to compensate for missing positional information. The formula is as follows:

[0064]

[0065] Among them, PE (pos，2i) PE (pos，2i+1) These are the 2i and 2i+1th components of the encoding vector at position pos, respectively, d model This represents the dimension of the word vector.

[0066] S2.3: The system imports the preprocessed tensor into the Encoder layer. The Attention layer is the main part of the Encoder layer, using three trainable array parameters Q, K, and V to train the tensor W using Attention. To allow the model to focus on different aspects of information, the model divides the tensor into 8 head vectors, projects them onto Q, K, and V through 8 linear transformations, and finally concatenates the different Attention results in the actual training of the model. The formula is as follows:

[0067]

[0068] Head t =Attention(QW t Q KW t K VW t V )

[0069] MultiHead(Q,K,V)=Concat(Head1,...,Head t )

[0070] Where dk represents the dimension of the model, and Head t This indicates the t-th head vector, MultiHead represents the output of multi-head attention, and Concat represents the concatenation operation on the model.

[0071] After training the Encoder layer, the results are used as the inputs to the K and V of the Decoder layer.

[0072] S2.4: The Feed Forward layer transforms the feature vectors extracted by Attention, increasing the model's expressive power. The activation function of the first layer of the Feed Forward layer is ReLU, and the second layer is a linear activation function, as shown in the following formula:

[0073] FFN = max(0, xW1+b1)W2+b2

[0074] Where x represents the output of the previous layer, and W1, W2, b1, b2 are trainable parameters.

[0075] S2.5: After embedding words into the text file, the data is expanded in dimension and positional encoding is added before being imported into the Decoder layer. The Decoder layer feeds the 3D tensor into the Attention layer and FeedForward layer for training. However, the Q of the Decoder module comes from the input of the Decoder, while K and V come from the output of the Encoder layer; the Q, K, and V of the Encoder all come from the input of the Encoder layer.

[0076] S2.6: After the Decoder layer, the output is compared with the text in the dataset for training. The automatic speech recognition model is trained until it converges; offline word error rate, real-time rate, and other metrics are verified.

[0077] S3: Due to the significant noise in telephone recordings, noise reduction is performed on the telephone audio files. The 8K audio from the telephone scenario is converted into a spectrogram. This spectrogram is then trained using transfer learning. The resulting automatic speech recognition model (referred to as the teacher model) is trained on publicly available standard audio data with a sampling frequency of 16KHz. This teacher model is then retrained to obtain the student model. The student model is fine-tuned and used to guide its training. During training, the student model is optimized to approximate the posterior probability distribution of the teacher model, generating a pre-trained model for automatic speech recognition based on the 8K telephone scenario. This pre-trained model can effectively recognize telephone customer service dialogues in general scenarios, but its accuracy is poor for different dialogue scenarios and it lacks adaptive functionality.

[0078] S4: Deploy the pre-trained model to the server and run it. Cache the conversations between telephone customer service representatives and users in different scenarios on the server and utilize the model for speech recognition. Simultaneously, identify different dialogue scenarios for telephone customer service: such as dialogue scenarios for the telecommunications industry, dialogue scenarios for the e-commerce industry, and dialogue scenarios between the education industry and parents; and store the obtained text data for online hot keyword customization.

[0079] S5: Generate corresponding hot word lists according to the characteristics of different scenarios, import them into the pre-trained model for language model training, customize differentiated language model self-updating models and hot word lists for different scenarios, and adjust the weights.

[0080] For the differential customization of the language model described in S4, an embodiment of the present invention also discloses a method for modifying the weight based on interactive hot words, which specifically includes the following steps:

[0081] S5.1: Import the dialogue voice data into the pre-trained model, convert it into text data and cache it, and classify the data in different scenarios at the same time;

[0082] S5.2: Preprocess the text and process the word frequency. Since the initial text data does not conform to the specification for extracting hot words, it needs to be preprocessed:

[0083] Delete the identifiers of "BEG", "END", and "UKW". For example, if the document is "ENG Customer, hello. The package you purchased has been successfully processed END", after removing the flags, it becomes "Customer, hello. The package you purchased has been successfully processed";

[0084] Segment the text data using a word segmentation plug-in. For example, after segmentation, the above example sentence becomes "Customer / Hello / You / Purchased / Package / Has / Been / Successfully / Processed";

[0085] Remove stop words, that is, certain words or terms will be automatically filtered out before or after processing natural language data (or text). Introduce a common stop word list to delete the stop words after word segmentation. After removing the stop words from the above example sentence, it becomes "Customer / Purchased / Package / Has / Been / Successfully / Processed".

[0086] S5.3: After preprocessing the text data, extract the word frequency, convert the text represented by discrete symbols into digital representation for subsequent use by the model;

[0087] S5.4: Normalize the word frequency;

[0088]

[0089] Wf * Represents the normalized word frequency, Wf represents the word frequency, μ represents the mean word frequency, and σ represents the standard deviation.

[0090] According to the probability difference, extract the general hot word list and the differential hot word list in different scenarios. The format of the hot word list is shown in Table 1 below, where it is assumed that the hot words are customer, package, and success.

[0091]

[0092] S5.5: Determine the corresponding activation threshold and difference threshold according to the larger quantile value of the word frequency. The activation threshold refers to the frequency of a certain hot word in all scenarios reaching a threshold; the difference threshold refers to the frequency of a certain hot word in a certain scenario reaching a threshold.

[0093] If a hot word is used more frequently in the current scenario than the activation threshold but less than the difference threshold in different scenarios, or more frequently than the difference threshold in all scenarios, it will be considered a general hot word.

[0094] If a hot word is used more frequently in the current scenario than the activation threshold and more frequently than the difference threshold in a certain scenario, it will be regarded as a differentiated hot word.

[0095] If a hot word's usage frequency in the current scenario does not exceed the activation threshold but exceeds the difference threshold in different scenarios, it will be deleted from the hot word list.

[0096] For example, in Table 1, if the general threshold is 0.04, the differentiation threshold for scenario 1 is 0.05, the differentiation threshold for scenario 1 is 0.04, and the differentiation threshold for scenario 1 is 0.03, then the word "customer" is a differentiated hot word for scenarios 2 and 3, "package" is a differentiated hot word for scenario 1, and "success" is a general hot word.

[0097] S5.6-1: When it is a common hot word, increase its weight in the decoding stage of the speech recognition model in all scenarios.

[0098] S5.6-2: When it is a differentiated hot word, the weight adjustment is only performed in the decoding stage of the speech recognition model in scenarios where the difference threshold is exceeded.

[0099] Preferably, users can upload their frequently used hot words to the server in a certain format to customize the hot words to meet their needs. The customized hot words will initially have an initial weight, which will be dynamically adjusted during the user's use to increase efficiency in speech recognition and better recognize the user's speech.

[0100] S6: During online prediction, preload the model, set differentiated weights for each model for different scenarios, update the model, and then cache the predicted speech data. Compare the obtained new path with other paths, and select the path with the highest score as the optimal path. Save the text result. Supplement the data with the real-time saved text data as new corpus, and update the activation threshold and differentiation threshold online. Repeat the above steps to update the model parameters.

[0101] Specific embodiments of the invention have been described above. It should be understood that those skilled in the art can make various modifications, additions, or similar substitutions to the described embodiments without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. An automatic voice recording method for hotword online customization update in a telephone scenario, characterized in that, The steps are as follows: S1: Train an automatic speech recognition model based on a deep neural network using a publicly available general speech dataset: The training of the automatic speech recognition model includes determining the basic parameters of the automatic speech recognition model; initializing the weights of each layer of the automatic speech recognition model; and determining the optimization method. S2: Retrain the automatic speech recognition model using 8k telephone audio files collected in telephone scenarios to generate a pre-trained model of the automatic speech recognition model based on 8K telephone scenarios. The 8K audio from the telephone scenario is converted into a spectrogram. The spectrogram is then trained using transfer learning. The resulting automatic speech recognition model, called the teacher model, is trained on 16K public standard audio data. This model is then retrained to obtain the student model. The model is then fine-tuned and used to guide the student model during training. During the training process, the student model is made to approximate the posterior probability distribution of the teacher model as closely as possible, thus generating a pre-trained model for the automatic speech recognition model based on the 8K telephone scenario. S3: Deploy and run the pre-trained model; during operation, use the pre-trained model to recognize the voice of the call and store the obtained text data for online hot word customization; S4: Customize a differentiated language model self-updating model and hot word list, and adjust the weights in the decoding stage in real time to form an adaptive speech recognition hot word system; S4-1: Segment the text data, remove stop words, and extract word frequencies; S4-2: Normalize the word frequencies based on probability differences, using the word frequencies as follows: Wf represents the word frequency, Wf * is the normalized word frequency, μ represents the mean of the word frequency, and σ represents the standard deviation of the word frequency. S4-3: Extract the general hot word list and the differential hot word list for different scenarios, and determine the corresponding activation threshold and differential threshold based on the median value of word frequency. If a hot word is used more frequently than the activation threshold but less than the differential threshold in different scenarios in the current scenario, it is regarded as a differential hot word. Weight adjustments are made only during the decoding phase of the model in scenarios where the difference threshold is exceeded. If a hot word's frequency of use in the current scenario exceeds both the activation threshold and the difference threshold across different scenarios, it is treated as a general hot word. Weights are increased during the decoding phase of the model in all scenarios. The determination formula is as follows: word is the word to be discriminated, Hotword common is a general hotword, Hotword particulary is a differentiated hotword; Wf represents word frequency, AcTh represents activation threshold, and DiffTh represents differentiation threshold S4-4: When the system decodes audio to generate text data, it adjusts and optimizes the decoding results according to the hot word weight table. When the pinyin of the text output by the Transformer model's decoder is the same as a hot word in the hot word table, a hot word discrimination mode is performed, a new path containing the hot word is added, the offset score is added to the decoding score of this path, and the score is re-evaluated. The formula is as follows: delta k = log(1 + Wf) Where, score(y1,...,y s ) decode the first character y1 to the last character y1 using Beam Search. s The final score on the path, P lm (y k |y1,...,y k-1 δ(x) represents the probability that the k-th character appears in the output. k Indicates the offset fraction of the parameter; The new path is compared with other paths, and the path with the highest score is selected as the optimal path based on the score. The text result is saved. The saved text data in real time is used to supplement the new corpus, and the activation threshold and differentiation threshold are updated online. S4-5: Design the update frequency of the adaptive speech recognition hot word system, which can be adjusted by the user through the interval of the update weight or set to update automatically during idle time; S4-6: It also provides an interactive hot word update method. At the same time, users can generate a JSON file of their frequently used hot words in the format of "hot word-part of speech-weight level" and upload it to the system. The offset score is adjusted according to the weight level to obtain user-defined hot words to meet their own needs. The aforementioned custom hot words have initial weights, and the weights, activation thresholds, and difference thresholds can be dynamically adjusted during system use to improve efficiency in speech recognition and better recognize user speech.

2. The automatic voice recording method of hotword online customization update in a telephone scenario according to claim 1, characterized in that, The automatic speech recognition model described above adopts the Transformer model, which is based on the Seq2seq model with Attention and consists of an encoder and a decoder. It is trained through a multi-head attention module. The specific training process for the Transformer model is as follows: S1-1: First, the audio data in the publicly available general speech dataset is preprocessed and transformed into Fbank audio features. The width of the data is expanded by Embedding word embedding and then transformed into a three-dimensional tensor. S1-2: The Transformer encoder does not process word position information. A position encoder is added before the tensor is imported into the Encoder layer to add information that different word positions may produce different semantics into the embedded 3D tensor to make up for the lack of position information. where PE (pos,2i) , PE (pos,2i+1) are the 2i, 2i+1 components of the encoding vector of position pos, respectively, d model denotes the dimension of the word vector. S1-3: The Transformer model imports the pre-processed 3D tensor (after being processed by the position encoder) into the Encoder layer. The Attention layer is the main part of the Encoder layer, using three trainable array parameters Q, K, and V to train the tensor W. To enable the Transformer model to focus on different aspects of information, it divides the tensor into h head vectors, forming multiple subspaces. It then projects these subspaces onto Q, K, and V through h linear transformations, finally concatenating the different Attention results. The formula is as follows: Head t = Attention(QW t Q ,KW t K ,VW t V ) MultiHead(Q, K, V) = Concat(Head1,..., Head t ) where dk represents the dimension of the model, Head t denotes the output result of the tth head vector MultiHead denotes multi-head attention, and Concat denotes a concatenation operation on the model. S1-4: The Feed Forward layer transforms the feature vectors extracted by Attention, increasing the model's expressive power. The Feed Forward layer has two layers: the first layer uses ReLU as the activation function, and the second layer uses a linear activation function, as shown in the following formula: FFN = Max(0, xW1+b1)W2+b2 Where x represents the output of the previous layer, and W1, W2, b1, b2 are trainable parameters; After training the Encoder layer, the resulting 3D tensor is used as the input to the K and V layers of the Decoder layer. S1-5: Embedding word embeddings are performed on the text files corresponding to the audio data in the publicly available general speech dataset. The data is then expanded into a three-dimensional tensor, and positional encoding is added before it is imported into the Decoder layer. The Decoder layer feeds the three-dimensional tensor into the Attention layer and the FeedForward layer for training. The Q of the Decoder layer comes from the input of the Decoder layer, and K and V come from the output of the Encoder layer. The Q, K, and V of the Encoder all come from the input of the Encoder layer. S1-6: After the Decoder layer, the 3D tensor output by the Decoder layer is passed into the linear layer and the Softmax layer to obtain text data. This data is then compared with the text files corresponding to the audio data in the general speech dataset for training. The loss function is calculated, and the difference between the forward calculation result and the true value of each iteration of the model is calculated to guide the next step of training in the correct direction until the model converges.

Images