A speech recognition method and device based on a self-attention mechanism enhancement

Abstract

The application discloses a speech recognition method and device based on a self-attention mechanism enhancement, relates to the technical field of speech recognition, and comprises the following steps: obtaining speech data to be recognized; extracting spectral features from the speech data, performing speech decoding on the spectral features to obtain decoded text; dividing the decoded text into a plurality of subwords, and performing label prediction on each subword to divide the subword into a term start, a term middle or a non-term; taking the non-term as a separator and the term start as a starting point, sequentially splicing the starting point and continuous term middles adjacent to the starting point to form a professional term; projecting the professional term into a multi-head self-attention vector, and replacing corresponding subwords in the decoded text to generate a mixed feature sequence; and performing linear decoding on the mixed feature sequence to obtain speech recognition text. Through implementation of the application, the problem that existing technologies rely on globally fixed noise suppression and are difficult to adaptively extract effective speech features can be solved, and the speech recognition accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method and apparatus based on a self-attention mechanism. Background Technology

[0002] With the rapid development of speech recognition technology, especially in the application of digital humans and voice assistants, traditional speech recognition methods can no longer meet the increasingly complex application requirements. Especially when processing speech signals in noisy environments, traditional speech recognition usually relies on audio signal preprocessing and simple noise suppression methods, such as spectral subtraction and Wiener filtering.

[0003] However, these methods can only suppress noise globally and in a fixed way, making it difficult to adaptively distinguish effective speech features from complex environmental noise. In noisy scenarios, they are prone to losing speech details, resulting in low accuracy of speech recognition results and failing to meet the speech recognition needs of digital humans in complex environments. Summary of the Invention

[0004] This invention provides a speech recognition method and apparatus based on a self-attention mechanism, which can solve the problem that existing speech recognition methods rely on globally fixed noise suppression and are difficult to adaptively extract effective speech features, thereby improving the accuracy of speech recognition in complex environments.

[0005] One embodiment of the present invention provides a speech recognition method based on a self-attention mechanism, comprising: Acquire the speech data to be recognized; Spectral features are extracted from the speech data to obtain spectral features; The spectral features are input into a preset acoustic recognition model so that the acoustic recognition model can perform speech decoding on the spectral features to obtain decoded text; The decoded text is segmented into several sub-words; For each sub-word, predict the label and determine the corresponding label type; the label type includes: term start, term middle, or non-term. Using non-termal sub-words as separators, and taking the sub-word at the beginning of a term as the starting point, the sub-words of the starting point and the consecutive terms adjacent to the starting point are sequentially concatenated to form several professional terms. Each technical term is linearly projected to obtain the corresponding multi-head self-attention vector; Replace the corresponding subwords in the decoded text with all multi-head self-attention vectors to generate a mixed feature sequence; Linear decoding of the mixed feature sequence yields the speech recognition text.

[0006] Furthermore, after acquiring the speech data to be recognized, the process also includes: The speech data is input into a preset denoising model to generate denoised speech data.

[0007] Furthermore, spectral features are extracted from the speech data to obtain spectral features, including: The speech data is filtered using a Mel filter to obtain Mel features; Taking the logarithm of the Mel features yields the log-Mel energy features; The spectral characteristics are obtained by performing a discrete cosine transform on the log-Mel energy characteristics.

[0008] Furthermore, speech decoding is performed on the spectral features to obtain the decoded text, including: The spectral features are decoded to obtain the acoustic probability distribution of each speech time step; Repeat the bundle search process until all speech time steps have been traversed to obtain the decoded text; The beam search process includes: Expand all terms in the preset vocabulary on the current decoding path to generate a candidate path set for the current stage; where the initial current decoding path is an empty sequence; Based on the acoustic probability distribution, the cumulative joint probability of each candidate path is calculated; Based on the cumulative joint probability, the candidate paths in the current stage candidate path set are sorted in descending order, and the top k candidate paths with the highest cumulative joint probability are retained as the decoding paths for the next stage; where k is a preset beamwidth parameter. Determine whether to traverse all speech time steps. If yes, select the word sequence corresponding to the path with the highest cumulative joint probability from the next stage decoding path as the decoded text; otherwise, use the next stage decoding path as the current decoding path for the next execution of the bundle search process.

[0009] Furthermore, the decoded text is segmented to obtain several sub-words, including: Based on a pre-defined vocabulary list of specialized terms, the decoded text is segmented using the longest-fit matching strategy to obtain several terms.

[0010] Furthermore, the pre-defined vocabulary sub-word list is constructed in the following way: Acquire a number of terms from professional materials related to power systems; All terms in power system technical materials are broken down into single-character sequences to obtain an initial vocabulary; Repeat the sub-word merging process to obtain a pre-defined sub-word list of professional terms; The sub-word merging process includes: Calculate the joint probability of all adjacent character pairs in the current vocabulary and their individual character probabilities; where the initial vocabulary is the initial vocabulary. Calculate the corresponding association score based on the joint probability of each adjacent character pair and its individual character probability; The adjacent character pairs with the highest association scores are merged into new subwords, and the new subwords replace the corresponding adjacent character pairs in the current vocabulary to obtain the updated vocabulary. The size of the vocabulary is obtained by counting the non-single-character sequences in the updated vocabulary. Determine if the vocabulary size has reached the preset size. If so, use the updated vocabulary as the preset professional terminology sub-vocabulary. Otherwise, use the updated vocabulary as the current vocabulary for the next sub-vocabulary merging process.

[0011] Furthermore, the acoustic recognition model is obtained in the following way: Acquire several local speech data and their locally recognized text; Based on several local speech data and their local recognized text, a local speech recognition model is trained to obtain a trained local speech recognition model. The model parameters of the trained local speech recognition model are sent to the central server, which then uses a federated averaging algorithm to perform a weighted average aggregation of the model parameters of all local speech recognition models to obtain the global model parameters. The global model parameters are then pruned and quantized to obtain lightweight global model parameters. Receive lightweight global model parameters from the central server; An acoustic recognition model is deployed based on the lightweight global model parameters.

[0012] Furthermore, based on several local speech data and their local recognized text, a local speech recognition model is trained to obtain a trained local speech recognition model, including: Data augmentation is performed on each local speech data to obtain the corresponding augmented speech data; All local speech data and all augmented speech data are used together as training samples; For each training sample, spectral features are extracted to obtain the corresponding local spectral features; All local spectral features and their corresponding local recognized texts are input into the local speech recognition model to be trained, so that the local speech recognition model is trained with local spectral features as input and predicted recognized text as output. In each training process, the loss function is calculated based on the predicted recognized text and the local recognized text, and the model parameters of the local speech recognition model are adjusted according to the loss function until the local speech recognition model converges, and the trained local speech recognition model is obtained.

[0013] Based on the above method embodiments, the present invention provides corresponding device embodiments, including: a speech data acquisition module, a spectral feature extraction module, a text generation module, a text segmentation module, a tag prediction module, a term combination module, an attention projection module, a hybrid feature generation module, and a linear decoding module; The voice data acquisition module is used to acquire the voice data to be recognized; The spectral feature extraction module is used to extract spectral features from speech data to obtain spectral features; The text generation module is used to input spectral features into a preset acoustic recognition model so that the acoustic recognition model can perform speech decoding on the spectral features to obtain decoded text. The text segmentation module is used to segment the decoded text into several sub-words; The tag prediction module is used to predict the tag for each subword and determine the corresponding tag type; the tag types include: term start, term middle, or non-term; The terminology combination module is used to construct several professional terms by using non-terminal sub-words as separators and starting from the sub-word of the term as the starting point, and sequentially concatenating the starting point and the sub-words in the middle of the consecutive terms adjacent to the starting point. The attention projection module is used to linearly project each technical term to obtain the corresponding multi-head self-attention vector. The hybrid feature generation module is used to replace the corresponding sub-words in the decoded text with all multi-head self-attention vectors to generate a hybrid feature sequence. The linear decoding module is used to perform linear decoding on the mixed feature sequence to obtain speech recognition text.

[0014] Furthermore, the speech recognition device enhanced by the self-attention mechanism also includes: a speech data denoising module; The speech data denoising module is used to input speech data into a preset denoising model and generate denoised speech data.

[0015] Compared with the prior art, the beneficial effects of this embodiment are as follows: This invention acquires speech data to be identified; extracts spectral features from the speech data to obtain spectral features, enhancing the representation of effective speech information; inputs the spectral features into a preset acoustic recognition model, enabling the acoustic recognition model to decode the spectral features to obtain decoded text; segments the decoded text to obtain several sub-words; predicts a label for each sub-word to determine the corresponding label type; the label type includes: term start, term middle, or non-term; using non-term sub-words as separators, and using term start sub-words as starting points, sequentially concatenates the starting point and the consecutive term middle sub-words adjacent to the starting point to form several terms. By accurately locating and extracting professional terms, feature shifts caused by noise are suppressed. Each professional term is linearly projected to obtain a corresponding multi-head self-attention vector, which strengthens the core semantic features of the professional terms. The multi-head self-attention vector enhanced by self-attention replaces words in the decoded text that may be affected by noise, generating a hybrid feature sequence, which strengthens the effective semantic features and weakens the noise interference features. The hybrid feature sequence is linearly decoded, so that the decoding process can focus on the effective semantic features enhanced by self-attention, reducing misjudgments and omissions in the decoding process, and finally obtaining the speech recognition text.

[0016] In summary, this invention extracts technical terms and linearly projects each term into a multi-head self-attention vector before replacing subwords to construct a hybrid feature sequence. This enables the speech recognition model to adaptively focus on the semantic features of technical terms, thereby suppressing feature shifts caused by noise. This solves the problem that existing speech recognition methods rely on globally fixed noise suppression and are difficult to adaptively extract effective speech features, thus improving the accuracy of speech recognition in complex environments. Attached Figure Description

[0017] Figure 1 This is a schematic flowchart of a speech recognition method based on self-attention mechanism enhancement provided in an embodiment of the present invention; Figure 2 This is a comparison chart of speech recognition accuracy under different noise environments provided by an embodiment of the present invention; Figure 3 This is a comparison chart of the convergence of the loss function during the training process provided in an embodiment of the present invention; Figure 4 This is a comparison chart of the accuracy of professional term recognition provided in an embodiment of the present invention; Figure 5 This is a comparison chart of speech recognition delay times provided in an embodiment of the present invention; Figure 6 This is a comparison chart of user privacy protection provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a speech recognition device based on a self-attention mechanism enhancement according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0020] like Figure 1 As shown, in order to address the problem that existing speech recognition methods rely on globally fixed noise suppression and are difficult to adaptively extract effective speech features, an embodiment of the present invention provides a speech recognition method enhanced by a self-attention mechanism, which includes at least the following steps: Step S101: Obtain the speech data to be recognized; In step S101, the digital human collects the voice sound wave signal emitted by the user through a built-in or external microphone, which serves as the voice data to be recognized. .

[0021] It should be noted that the embodiments of the present invention are all illustrated using digital human interaction scenarios as typical application scenarios, but should not be construed as meaning that this method can only be applied to digital human scenarios.

[0022] Preferably, after acquiring the speech data to be recognized, the method further includes: The speech data is input into a preset denoising model to generate denoised speech data.

[0023] Specifically, in order to improve the accuracy of subsequent feature extraction and recognition, this invention acquires the speech data to be recognized... Then, a denoising model built using a deep neural network (DNN) is used to denoise the speech data to be recognized. Processing: Converting voice data The input is fed into a preset denoising model. By learning the mapping relationship between a large number of noisy speech and clean speech samples, the denoising model achieves effective separation of noise signals and speech signals, filters out interference components such as environmental noise and channel noise, and retains key acoustic features in the speech signal, ultimately generating denoised speech data. .

[0024] It should be noted that the preset denoising model is obtained in the following way: Acquire several ideal clean speech signals Furthermore, random environmental noise, equipment background noise, and other interference components are added to the ideal clean speech signal to obtain the corresponding noisy speech signal. All noisy speech signals and their corresponding ideal clean speech signals are input into the denoising model to be trained, so that the denoising model uses the noisy speech signals as input to predict the clean speech signals. Iterative training is performed on the input; In each iteration, the noise suppression loss function is calculated based on the ideal clean speech signal and the predicted clean speech signal: ; in, Indicates the first An ideal clean speech signal Indicates the first The predicted clean speech signal is obtained by predicting a noisy speech signal. This represents the noise suppression loss function. This represents the number of ideal, clean speech signals; Based on the noise suppression loss function, the model parameters of the denoising model are adjusted through the backpropagation algorithm until the noise suppression loss function converges or reaches the preset training rounds, thus obtaining the preset denoising model.

[0025] Step S201: Extract spectral features from the speech data to obtain spectral features; In a preferred embodiment, spectral features are extracted from the speech data to obtain spectral features, including: The speech data is filtered using a Mel filter to obtain Mel features; Taking the logarithm of the Mel features yields the log-Mel energy features; The spectral characteristics are obtained by performing a discrete cosine transform on the log-Mel energy characteristics.

[0026] For step S201, Mel Frequency Cepstral Coefficients (MFCCs) are used to analyze the speech data. Or denoised speech data Spectral feature extraction is performed to obtain spectral features, thereby converting the original speech signal into a high-dimensional feature vector that is easier for acoustic recognition models to process, and then inputting it into the subsequent acoustic recognition model.

[0027] Specifically, firstly, the voice data Or denoised speech data The input is fed into a Mel filter bank for filtering to simulate the high-resolution perception characteristics of the human ear for low-frequency signals, thus obtaining Mel features.

[0028] Subsequently, logarithmic operations are performed on the Mel features to further enhance the logarithmic Mel energy features and energy change features in the speech signal by compressing the numerical dynamic range through logarithmic operations, thus obtaining the logarithmic Mel energy features.

[0029] Finally, the log-Mel energy features are subjected to discrete cosine transform (DCT) to remove redundant correlations between feature dimensions and achieve dimensionality reduction, thereby extracting the final spectral features.

[0030] Based on the above processing flow, the specific formula for spectral feature extraction is as follows: ; in, Indicates spectral characteristics, This indicates the filtering process of the Mel filter. This indicates logarithmic operations. This represents the discrete cosine transform processing.

[0031] Through the above MFCC feature extraction, the obtained spectral features have higher resolution for low-frequency speech signals, effectively highlighting key formants and phoneme information in speech, while suppressing interference from irrelevant noise and redundant frequency components, providing an input basis for achieving high-precision speech recognition.

[0032] Step S301: Input the spectral features into the preset acoustic recognition model so that the acoustic recognition model can perform speech decoding on the spectral features to obtain the decoded text; In a preferred embodiment, speech decoding is performed on the spectral features to obtain decoded text, including: The spectral features are decoded to obtain the acoustic probability distribution of each speech time step; Repeat the bundle search process until all speech time steps have been traversed to obtain the decoded text; The beam search process includes: Expand all terms in the preset vocabulary on the current decoding path to generate a candidate path set for the current stage; where the initial current decoding path is an empty sequence; Based on the acoustic probability distribution, the cumulative joint probability of each candidate path is calculated; Based on the cumulative joint probability, the candidate paths in the current stage candidate path set are sorted in descending order, and the top k candidate paths with the highest cumulative joint probability are retained as the decoding paths for the next stage; where k is a preset beamwidth parameter. Determine whether to traverse all speech time steps. If yes, select the word sequence corresponding to the path with the highest cumulative joint probability from the next stage decoding path as the decoded text; otherwise, use the next stage decoding path as the current decoding path for the next execution of the bundle search process.

[0033] For step S301, the spectral features obtained in step S201 are input into a preset acoustic recognition model. In the acoustic recognition model, the spectral features are decoded into speech using a beam search algorithm to obtain the decoded text. The specific formula is as follows: ; in, This represents the target word sequence, i.e., the decoded text. Representing the spectral features of the input, the goal of speech decoding is to maximize the conditional probability. This allows us to select the word sequence that best matches the acoustic features.

[0034] Specifically, the acoustic recognition model first analyzes the spectral features of each time step to obtain the acoustic probability distribution corresponding to each speech time step, which is the model's confidence assessment of the possible terms at each time step.

[0035] To efficiently find the optimal solution in the vast word sequence combination space, this invention employs a beam search algorithm for decoding. This algorithm starts with an empty sequence as the initial decoding path. At each speech time step, it expands all possible next words in a preset vocabulary for the current k candidate paths, generating k × |V| candidate paths (where |V| is the size of the preset vocabulary), forming a new set of candidate paths. Then, for each newly generated candidate path, its cumulative joint probability is equal to the cumulative probability of the path in the previous stage, multiplied by the acoustic probability corresponding to the new word at the current time step, thus obtaining the overall confidence of the path from the initial to the current state. Next, all candidate paths are sorted in descending order according to the cumulative joint probability, and only the top k paths in terms of probability are retained for the next stage. This controls computational overhead while avoiding local optima problems caused by greedy single-step selection.

[0036] After traversing all speech time steps, the word sequence with the highest cumulative joint probability is selected from the final retained path as the decoded text obtained in this decoding.

[0037] It should be noted that k is a preset beamwidth parameter, which is generally a positive integer greater than or equal to 1. Its value determines the number of candidate paths retained in each round of the decoding process. The larger the value of k, the wider the search space and the higher the recognition accuracy, but the corresponding computational overhead is also greater. In practical applications, it should be set reasonably in combination with hardware computing power and recognition requirements.

[0038] This invention employs a pruning strategy based on beam search, which, compared to greedy decoding that independently selects the word with the highest probability at each position, effectively avoids global recognition bias caused by local optimal selection. While ensuring decoding accuracy, it reduces the originally exponentially increasing computational complexity to a controllable range. At the same time, by evaluating the global cumulative probability, it effectively reduces the risk of error accumulation, enabling the generated text to more accurately restore the semantic information of the original speech.

[0039] Step S302: Segment the decoded text to obtain several sub-words; In a preferred embodiment, the decoded text is segmented to obtain several sub-words, including: Based on a pre-defined vocabulary list of specialized terms, the decoded text is segmented using the longest-fit matching strategy to obtain several terms.

[0040] For step S302, preferably, before segmenting the decoded text, the decoded text is cleaned by matching and replacing it using a preset rule expression to remove irrelevant noise, such as punctuation marks and HTML tags, to obtain cleaned decoded text, ensuring that subsequent segmentation and modeling operations are not interfered with.

[0041] The cleaned decoded text is segmented. Based on a pre-defined vocabulary list of professional terms, the text is scanned and segmented using the longest-fit matching strategy. That is, starting from the beginning of the decoded text, the longest valid sub-word sequence in the vocabulary list is matched. After a successful match, the process is repeated from the beginning of the remaining text until the entire text is segmented, thus obtaining several sub-word units that are highly compatible with professional terms.

[0042] It should be noted that the above-mentioned specialized vocabulary sub-vocabulary was constructed offline based on the WordPiece algorithm. The specific construction process is as follows: The preferred, pre-defined vocabulary sub-word list is constructed in the following way: Acquire a number of terms from professional materials related to power systems; All terms in power system technical materials are broken down into single-character sequences to obtain an initial vocabulary; Repeat the sub-word merging process to obtain a pre-defined sub-word list of professional terms; The sub-word merging process includes: Calculate the joint probability of all adjacent character pairs in the current vocabulary and their individual character probabilities; where the initial vocabulary is the initial vocabulary. Calculate the corresponding association score based on the joint probability of each adjacent character pair and its individual character probability; The adjacent character pairs with the highest association scores are merged into new subwords, and the new subwords replace the corresponding adjacent character pairs in the current vocabulary to obtain the updated vocabulary. The size of the vocabulary is obtained by counting the non-single-character sequences in the updated vocabulary. Determine if the vocabulary size has reached the preset size. If so, use the updated vocabulary as the preset professional terminology sub-vocabulary. Otherwise, use the updated vocabulary as the current vocabulary for the next sub-vocabulary merging process.

[0043] Specifically, firstly, based on all the vocabulary in the power system professional materials, all vocabulary is broken down into single-character sequences to form an initial vocabulary list.

[0044] Next, the word merging process begins. By statistically analyzing the joint frequency and individual character frequency of adjacent character pairs, and based on the joint probability of each adjacent character pair and its individual character probability, the corresponding association score is calculated. This quantifies the tendency of characters to combine into stable words. The specific formula is as follows: ; in, Character and characters The adjacent character pairs formed Indicates adjacent character pairs The joint probability, Character The probability of a single character. Character The probability of a single character. Indicates adjacent character pairs The correlation score.

[0045] In each iteration, the association score of all adjacent character pairs is calculated. Select the character pair with the highest association score. Merged into new sub-word units And by replacing the corresponding adjacent character pairs in the vocabulary, higher-level specialized sub-words are gradually constructed.

[0046] The above sub-word merging process is repeated continuously. After each update of the vocabulary, the number of non-single character sequences is counted to determine whether the vocabulary size has reached the preset size. If the vocabulary size has reached the preset size, the iteration is terminated and the currently updated vocabulary is used as the final professional vocabulary sub-word vocabulary. If the vocabulary size has not reached the preset size, the updated vocabulary is used as the basis for a new round of iteration, and the sub-word merging process continues until the generated vocabulary size meets the requirements.

[0047] It should be noted that in actual speech recognition applications, the word segmenter is based on a professional vocabulary sub-word list that has been built offline. It uses a global longest first matching strategy to segment the input text in real time. There is no need to repeatedly perform character frequency statistics and association score calculations in the vocabulary building stage, avoiding the additional latency caused by online calculations. It also does not require the use of device computing resources for complex iterative operations, which greatly improves the text processing efficiency in real-time voice interaction scenarios and ensures low latency and high stability in the word segmentation process.

[0048] Step S303: Perform label prediction for each sub-word and determine the corresponding label type; the label type includes: term start, term middle, or non-term; For step S302, the linear classification layer built into the acoustic recognition model performs BIO tag prediction for each subword to determine the corresponding tag type. Specifically, the linear classification layer first maps the subword to an unnormalized prediction score, as shown in the following formula: ; in, This represents the unnormalized prediction result of the subword, indicating the score of the subword belonging to the B / I / O label classes. This represents the weight matrix of the linear classification layer. This represents the bias term of the linear classification layer. Indicates a subword.

[0049] Unnormalized predicted scores ( The subword is converted into a probability value using the Softmax function, and the label with the highest probability is selected as the label type of the subword. In this invention, the label type includes term start B, term middle I, or non-term O. Term start B indicates that the subword is the beginning part of a professional term. Term middle I indicates that the subword is part of a professional vocabulary, but it is not the beginning part of the vocabulary, but is located inside the vocabulary. Non-term O indicates that the subword is not part of any professional term.

[0050] Step S304: Using non-termal sub-words as separators, and taking the sub-word at the beginning of a term as the starting point, sequentially concatenate the starting point and the sub-words in the middle of consecutive terms adjacent to the starting point to form several professional terms. For step S304, extract all sub-words marked as term start B and term middle I. These sub-words will constitute the technical term. During the extraction process, the integrity of the technical term is restored according to the following rules: In the BIO tag sequence, the sub-words that are not term O are used as separators, and the sub-words of term start B are used as the starting point. The starting point and the sub-words of term middle I that appear consecutively after the starting point are concatenated in sequence to restore the complete technical term.

[0051] It should be noted that for WordPiece subwords with the ## prefix generated during splitting, the ## prefix is ​​removed during merging and then directly concatenated with the previous subword.

[0052] Step S305: Perform linear projection on each technical term to obtain the corresponding multi-head self-attention vector; For step S305, a linear projection is performed on each identified term to map the term's sub-word feature vectors to the dimensional space required by the attention mechanism. Based on this, a multi-head self-attention mechanism is constructed to generate the corresponding multi-head self-attention vector. This is specifically divided into two stages: single-head self-attention computation and parallel fusion of multi-head self-attention. In the single-head self-attention computation stage, the technical terms are linearly transformed to generate the query matrix corresponding to each attention head. Key matrix Sum matrix We use standard dot product attention to calculate the weights of single-head self-attention, using the following formula: ; in, , and These represent the query matrix, key matrix, and value matrix, respectively. This represents the attention head dimension. Key matrix transpose, This represents the standard dot product self-attention vector.

[0053] In the multi-head self-attention parallel fusion stage, the feature vectors output by multiple attention heads are aggregated through a concatenation operation, and then linearly mapped through the output weight matrix to obtain the multi-head self-attention vector. The specific formula is as follows: ; ; in, Represents a multi-head self-attention vector. This indicates a splicing operation. This represents the output weight matrix, used to map the concatenated result to the overall dimensions of the model. Indicates the first The output of each attention head, Indicates the first The weight matrix of the query vector in each attention head. Indicates the first The weight matrix of the key vectors in each attention head. Indicates the first The weight matrix of the value vector in each attention head.

[0054] This invention transforms identified technical terms into multi-head self-attention vectors by linearly projecting them, thereby achieving multi-dimensional capture and fusion of different semantic features within the technical terms. Compared to single-head attention, the multi-head structure allows different attention heads to focus on different domain knowledge features of the terms. While improving the accuracy of semantic understanding of terms, it also optimizes processing efficiency through parallel computing, ultimately obtaining a multi-head self-attention vector that can comprehensively represent the semantic associations of technical terms.

[0055] Step S306: Replace the corresponding sub-words in the decoded text with all multi-head self-attention vectors to generate a hybrid feature sequence; For step S306, in order to generate the enhanced speech recognition text, the multi-head self-attention vector obtained in step S305 is used to replace the original word representations at the corresponding positions in the decoded text according to the position boundaries of each term in the original word sequence. The words at non-term positions remain unchanged, thereby constructing a hybrid feature sequence containing enhanced term features and original ordinary vocabulary features.

[0056] Step S307: Perform linear decoding on the mixed feature sequence to obtain the speech recognition text.

[0057] For step S307, the mixed feature sequence is linearly decoded by the built-in linear decoding layer, each feature vector in the sequence is mapped to the word space, and then the argmax operation is performed on the word distribution at each position. The word with the highest probability is selected as the final predicted word at that position. Finally, the predicted words at all positions are concatenated one by one in the original sequence order to output the complete speech recognition text.

[0058] Taking "The 110 kV main transformer substation needs to be inspected" as an example, step S305 enhances the vectors of "110 kV" and "main transformer substation". This step replaces the two enhanced term vectors back to the corresponding positions in the whole sentence word sequence. Together with the original vectors of non-terms such as "of", "need", "to be", and "inspection", they form a complete sequence. Finally, the complete recognized text is decoded and output, realizing the targeted enhancement of the accuracy of professional vocabulary in the whole sentence speech recognition result.

[0059] To avoid sending user voice data to a central server and protect user privacy, the acoustic recognition model of this invention adopts a privacy-preserving training framework of local edge training, federated learning optimization, and lightweight deployment: First, edge training is performed on a local device using local voice data to obtain local model parameters; then, the local model parameters are uploaded to a central server for federated learning optimization to obtain lightweight global model parameters; finally, each device receives the lightweight global model parameters and deploys them to obtain the acoustic recognition model of this invention. The specific process is as follows: In a preferred embodiment, the acoustic recognition model is obtained in the following manner: Acquire several local speech data and their locally recognized text; Based on several local speech data and their local recognized text, a local speech recognition model is trained to obtain a trained local speech recognition model. The model parameters of the trained local speech recognition model are sent to the central server, which then uses a federated averaging algorithm to perform a weighted average aggregation of the model parameters of all local speech recognition models to obtain the global model parameters. The global model parameters are then pruned and quantized to obtain lightweight global model parameters. Receive lightweight global model parameters from the central server; An acoustic recognition model is deployed based on the lightweight global model parameters.

[0060] In one embodiment of the present invention, in order to improve the accuracy and versatility of the model, local speech data covering multiple languages, dialects and accents and their corresponding local recognition text are collected to construct a local speech dataset containing multiple samples. Each sample includes a local speech data and the local recognition text corresponding to the local speech data, thereby ensuring that the model can adapt to changes in different languages ​​and accents.

[0061] Specifically, the local speech dataset is as follows: ; in, Represents a local speech dataset. Indicates the first One sample, each sample Including sampled local speech data and local voice data Corresponding local recognition text , This indicates the amount of local voice data.

[0062] It should be noted that voice data contains a large amount of user privacy information. Directly uploading it to a central server for training poses a risk of data leakage. To protect user voice data privacy and prevent the raw data from being leaked, this invention, after constructing a local voice dataset, uses an edge computing architecture to complete model training locally. Only the trained model parameters are uploaded to the central server for federated aggregation, ensuring data privacy and security while simultaneously achieving global model optimization. The specific local training steps are as follows: Preferably, a local speech recognition model is trained based on several local speech data and their local recognized text to obtain a trained local speech recognition model, including: Data augmentation is performed on each local speech data to obtain the corresponding augmented speech data; All local speech data and all augmented speech data are used together as training samples; For each training sample, spectral features are extracted to obtain the corresponding local spectral features; All local spectral features and their corresponding local recognized texts are input into the local speech recognition model to be trained, so that the local speech recognition model is trained with local spectral features as input and predicted recognized text as output. In each training process, the loss function is calculated based on the predicted recognized text and the local recognized text, and the model parameters of the local speech recognition model are adjusted according to the loss function until the local speech recognition model converges, and the trained local speech recognition model is obtained.

[0063] Specifically, in order to expand the diversity of training samples and improve the model's generalization ability in complex real-world scenarios, data augmentation is performed on each local speech data. In this embodiment, data augmentation includes pitch shifting, time stretching, and noise injection. By transforming the original speech data, the diversity of training data is increased, thereby enhancing the model's adaptability in real-world scenarios.

[0064] The specific formula for pitch shift is as follows: ; in, Indicates pitch frequency, This represents the speech signal before the pitch shift. This represents the speech signal after pitch shift; The specific formula for time stretching is as follows: ; in, Indicates the time stretching factor. This represents the audio signal before time stretching. This represents the time-stretched audio signal. The specific formula for noise injection is as follows: ; in, Indicates noise intensity. Indicates random noise. This represents the speech signal before noise injection. This represents the speech signal after noise injection.

[0065] After completing the augmentation processing of the local speech data, all local speech data and all augmented speech data are used together as training samples to multiply the size of the training samples and the diversity of acoustic features without collecting additional real data.

[0066] Subsequently, Mel frequency cepstral coefficients (MFCC) were used to extract spectral features from each training sample to obtain the corresponding local spectral features.

[0067] All local spectral features and their corresponding local recognized texts are input into the local speech recognition model to be trained, so that the local speech recognition model takes local spectral features as input and predicted recognized text as output for iterative training. In each training process, the model first performs acoustic encoding and sequence decoding based on the input local spectral features to generate the corresponding predicted recognized text. Then, by comparing the predicted recognized text and the local recognized text, the loss function is calculated to quantify the error between the model output and the real label. According to the loss function, the error gradient is passed layer by layer through the backpropagation algorithm to adaptively adjust the model parameters of the local speech recognition model, continuously optimizing the model's feature mapping and sequence decoding capabilities. The above training loop of forward prediction, error calculation and parameter update is repeated until the loss function of the local speech recognition model tends to stabilize, and a trained local speech recognition model is obtained. This model has fully learned the correspondence between the acoustic features of local speech data and text, and has the ability to accurately recognize local languages, dialects and accents.

[0068] After training, each edge digital human device only uploads the model parameters of its local speech recognition model to the central server, without transmitting any raw speech data, thus protecting user privacy and security at the source and avoiding the risk of data leakage.

[0069] After receiving the model parameters uploaded by each device, the central server uses a federated averaging algorithm to perform a weighted average aggregation of the model parameters of all local speech recognition models for digital humans, obtaining the global model parameters. The specific formula is as follows: ; in, Represents global model parameters. Indicates the number of digital human devices. Indicates the first Local model parameters uploaded by a digital human device.

[0070] To reduce model size and computational overhead, and to enable the model to adapt to low-power edge devices, the central server prunes and quantizes the global model parameters to obtain lightweight global model parameters. The specific formula for the quantization process is as follows: ; in, This represents the model weights before quantization. This represents the quantized model weights. Indicates the quantization step size. This indicates rounding.

[0071] Finally, the central server distributes the lightweight global model parameters to each edge digital human device; after receiving the lightweight global model parameters, each edge digital human device completes the deployment of the acoustic recognition model based on the lightweight global model parameters.

[0072] Preferably, after obtaining the speech-recognized text, the process further includes: Obtain the true label type for each word in the decoded text; Based on the true label type and the label type obtained by label prediction for each sub-word, the cross-entropy loss function is calculated, and the linear classification layer is fine-tuned according to the cross-entropy loss function to improve the model's label prediction accuracy for power system professional terms.

[0073] Specifically, based on the true label type and the label type obtained from label prediction for each sub-word, the cross-entropy loss function is calculated using the following formula: ; in, This represents the cross-entropy loss value. Indicates the first The true label type of each sub-word Indicates the first The label type obtained by predicting the label of each sub-word.

[0074] Based on the cross-entropy loss function Fine-tuning the weight matrix of the linear classification layer With bias term This enables the linear classification layer to more accurately identify the boundaries and composition of power system terminology.

[0075] The application effects of this invention in the context of power system terminology are as follows: Figure 2 The diagram shows a comparison of speech recognition accuracy under different noise environments. It illustrates the accuracy of the speech recognition method of the present invention compared with traditional signal processing-based speech recognition methods under different noise environments. The speech recognition method of the present invention exhibits higher recognition accuracy under noise-free, white noise, traffic noise, and street noise conditions. In particular, it can extract speech information from noise more effectively in complex noise environments. In contrast, traditional methods usually rely on simple noise filtering and speech enhancement techniques, which leads to a significant drop in recognition performance in noisy environments.

[0076] Figure 3The graph shows a comparison of the convergence of the loss function during training, illustrating the convergence of the acoustic recognition model of this invention and the traditional Hidden Markov Model (HMM) during training. As can be seen from the graph, the acoustic recognition model of this invention converges much faster than the HMM during training and can achieve a lower loss value in a shorter training time. The HMM relies on relatively complex state transitions and observation probability calculations, resulting in a slower training process. In contrast, the acoustic recognition model of this invention, through an efficient gradient descent algorithm, can optimize network weights more quickly and reduce training time.

[0077] Figure 4 This chart compares the accuracy of professional term recognition, demonstrating the difference between acoustic recognition models and traditional rule-based word matching methods in specific fields such as wind power and energy storage. The acoustic recognition model shows a clear advantage in term recognition in professional fields such as wind power, energy storage, and frequency regulation, accurately understanding the context of professional terms and effectively recognizing them. In contrast, traditional rule-based word matching methods typically rely on manually compiled dictionaries and rules, resulting in lower accuracy when faced with new words or complex contexts.

[0078] Figure 5 The graph shows a comparison of speech recognition latency, illustrating the difference in latency between DNN-based speech recognition and traditional Hidden Markov Model (HMM)-based speech recognition. The acoustic recognition model of this invention exhibits significantly lower latency in speech recognition tasks compared to traditional HMMs. Especially in real-time speech recognition applications, the acoustic recognition model of this invention can quickly process input signals and provide real-time feedback. Traditional HMMs, due to their complex state transitions and high computational load, result in longer response times in practical applications. Furthermore, the parallel computing and end-to-end training capabilities of the acoustic recognition model of this invention give it a significant advantage in latency control.

[0079] Figure 6 The comparison chart illustrates the difference in user privacy protection between federated learning-based speech recognition and traditional centralized speech recognition. Federated learning, by training the model on a local device, avoids uploading user data to a central server, thus effectively protecting user privacy. In contrast, traditional centralized speech recognition systems require sending user voice data to a server for processing, posing a risk of data leakage. Federated learning not only ensures high-performance speech recognition but also optimizes the model without disclosing sensitive data, ensuring full protection of user privacy.

[0080] like Figure 7 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; An embodiment of the present invention provides a speech recognition device based on a self-attention mechanism, comprising: a speech data acquisition module, a spectral feature extraction module, a text generation module, a text segmentation module, a label prediction module, a term combination module, an attention projection module, a hybrid feature generation module, and a linear decoding module; The voice data acquisition module is used to acquire the voice data to be recognized; The spectral feature extraction module is used to extract spectral features from speech data to obtain spectral features; The text generation module is used to input spectral features into a preset acoustic recognition model so that the acoustic recognition model can perform speech decoding on the spectral features to obtain decoded text. The text segmentation module is used to segment the decoded text into several sub-words; The tag prediction module is used to predict the tag for each subword and determine the corresponding tag type; the tag types include: term start, term middle, or non-term; The terminology combination module is used to construct several professional terms by using non-terminal sub-words as separators and starting from the sub-word of the term as the starting point, and sequentially concatenating the starting point and the sub-words in the middle of the consecutive terms adjacent to the starting point. The attention projection module is used to linearly project each technical term to obtain the corresponding multi-head self-attention vector. The hybrid feature generation module is used to replace the corresponding sub-words in the decoded text with all multi-head self-attention vectors to generate a hybrid feature sequence. The linear decoding module is used to perform linear decoding on the mixed feature sequence to obtain speech recognition text.

[0081] In a preferred embodiment, the speech recognition device based on the self-attention mechanism further includes: a speech data denoising module; The speech data denoising module is used to input speech data into a preset denoising model and generate denoised speech data.

[0082] In a preferred embodiment, the spectral feature extraction module includes: a filtering unit, a logarithmic processing unit, and a frequency domain transformation unit; The filtering unit is used to filter speech data through a Mel filter to obtain Mel features; The logarithmic processing unit is used to take the logarithm of the Mel features to obtain the log-Mel energy features; The frequency domain transformation unit is used to perform discrete cosine transform on the logarithmic Mel energy characteristics to obtain the spectral characteristics.

[0083] In a preferred embodiment, the text generation module performs speech decoding on the spectral features to obtain decoded text, including: The spectral features are decoded to obtain the acoustic probability distribution of each speech time step; Repeat the bundle search process until all speech time steps have been traversed to obtain the decoded text; The beam search process includes: Expand all terms in the preset vocabulary on the current decoding path to generate a candidate path set for the current stage; where the initial current decoding path is an empty sequence; Based on the acoustic probability distribution, the cumulative joint probability of each candidate path is calculated; Based on the cumulative joint probability, the candidate paths in the current stage candidate path set are sorted in descending order, and the top k candidate paths with the highest cumulative joint probability are retained as the decoding paths for the next stage; where k is a preset beamwidth parameter. Determine whether to traverse all speech time steps. If yes, select the word sequence corresponding to the path with the highest cumulative joint probability from the next stage decoding path as the decoded text; otherwise, use the next stage decoding path as the current decoding path for the next execution of the bundle search process.

[0084] In a preferred embodiment, the text segmentation module segments the decoded text to obtain several sub-words, including: Based on a pre-defined vocabulary list of specialized terms, the decoded text is segmented using the longest-fit matching strategy to obtain several terms.

[0085] In a preferred embodiment, the pre-defined vocabulary sub-word list is constructed in the following manner: Acquire a number of terms from professional materials related to power systems; All terms in power system technical materials are broken down into single-character sequences to obtain an initial vocabulary; Repeat the sub-word merging process to obtain a pre-defined sub-word list of professional terms; The sub-word merging process includes: Calculate the joint probability of all adjacent character pairs in the current vocabulary and their individual character probabilities; where the initial vocabulary is the initial vocabulary. Calculate the corresponding association score based on the joint probability of each adjacent character pair and its individual character probability; The adjacent character pairs with the highest association scores are merged into new subwords, and the new subwords replace the corresponding adjacent character pairs in the current vocabulary to obtain the updated vocabulary. The size of the vocabulary is obtained by counting the non-single-character sequences in the updated vocabulary. Determine if the vocabulary size has reached the preset size. If so, use the updated vocabulary as the preset professional terminology sub-vocabulary. Otherwise, use the updated vocabulary as the current vocabulary for the next sub-vocabulary merging process.

[0086] In a preferred embodiment, the acoustic recognition model is obtained in the following manner: Acquire several local speech data and their locally recognized text; Based on several local speech data and their local recognized text, a local speech recognition model is trained to obtain a trained local speech recognition model. The model parameters of the trained local speech recognition model are sent to the central server, which then uses a federated averaging algorithm to perform a weighted average aggregation of the model parameters of all local speech recognition models to obtain the global model parameters. The global model parameters are then pruned and quantized to obtain lightweight global model parameters. Receive lightweight global model parameters from the central server; An acoustic recognition model is deployed based on the lightweight global model parameters.

[0087] In a preferred embodiment, a local speech recognition model is trained based on several local speech data and their local recognized text to obtain a trained local speech recognition model, including: Data augmentation is performed on each local speech data to obtain the corresponding augmented speech data; All local speech data and all augmented speech data are used together as training samples; For each training sample, spectral features are extracted to obtain the corresponding local spectral features; All local spectral features and their corresponding local recognized texts are input into the local speech recognition model to be trained, so that the local speech recognition model is trained with local spectral features as input and predicted recognized text as output. In each training process, the loss function is calculated based on the predicted recognized text and the local recognized text, and the model parameters of the local speech recognition model are adjusted according to the loss function until the local speech recognition model converges, and the trained local speech recognition model is obtained.

[0088] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the speech recognition method based on self-attention mechanism enhancement provided by any of the above-described method embodiments of the present invention.

[0089] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0090] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A speech recognition method based on self-attention mechanism enhancement, characterized in that, include: Acquire the speech data to be recognized; Spectral features are extracted from the speech data to obtain spectral features; The spectral features are input into a preset acoustic recognition model so that the acoustic recognition model performs speech decoding on the spectral features to obtain decoded text; The decoded text is segmented to obtain several sub-words; For each sub-word, a label prediction is performed to determine the corresponding label type; the label type includes: term start, term middle, or non-term. Using non-termal sub-words as separators, and taking the sub-word at the beginning of a term as the starting point, the sub-words of the starting point and the consecutive terms adjacent to the starting point are sequentially concatenated to form several professional terms. Each technical term is linearly projected to obtain the corresponding multi-head self-attention vector; Replace the corresponding subwords in the decoded text with all multi-head self-attention vectors to generate a mixed feature sequence; The mixed feature sequence is linearly decoded to obtain the speech recognition text.

2. The speech recognition method based on self-attention mechanism enhancement according to claim 1, characterized in that, After acquiring the speech data to be recognized, the process also includes: The speech data is input into a preset denoising model to generate denoised speech data.

3. The speech recognition method based on self-attention mechanism enhancement according to claim 1, characterized in that, Spectral features are extracted from the speech data to obtain spectral features, including: The speech data is filtered using a Mel filter to obtain Mel features; Taking the logarithm of the Mel features yields the logarithmic Mel energy features; The spectral characteristics are obtained by performing a discrete cosine transform on the logarithmic Mel energy features.

4. The speech recognition method based on self-attention mechanism enhancement according to claim 1, characterized in that, Speech decoding of spectral features yields decoded text, including: The spectral features are decoded to obtain the acoustic probability distribution of each speech time step; Repeat the bundle search process until all speech time steps have been traversed to obtain the decoded text; The beam search process includes: Expand all terms in the preset vocabulary on the current decoding path to generate a candidate path set for the current stage; where the initial current decoding path is an empty sequence; Based on the acoustic probability distribution, the cumulative joint probability of each candidate path is calculated; Based on the cumulative joint probability, the candidate paths in the current stage candidate path set are sorted in descending order, and the top k candidate paths with the highest cumulative joint probability are retained as the decoding paths for the next stage; where k is a preset beamwidth parameter. Determine whether all speech time steps have been traversed. If so, select the word sequence corresponding to the path with the highest cumulative joint probability from the next stage decoding path as the decoded text; otherwise, use the next stage decoding path as the current decoding path for the next execution of the bundle search process.

5. The speech recognition method based on self-attention mechanism enhancement according to claim 1, characterized in that, The decoded text is segmented to obtain several sub-words, including: Based on a pre-defined vocabulary list of specialized terms, the decoded text is segmented using the longest-preferred matching strategy to obtain several sub-words.

6. The speech recognition method based on self-attention mechanism enhancement according to claim 5, characterized in that, The preset vocabulary sub-word list is constructed in the following way: Acquire a number of terms from professional materials related to power systems; All terms in power system technical materials are broken down into single-character sequences to obtain an initial vocabulary; Repeat the sub-word merging process to obtain the preset professional vocabulary sub-word list; The sub-word merging process includes: Calculate the joint probability of all adjacent character pairs in the current vocabulary and their individual character probabilities; where the initial vocabulary is the initial vocabulary. Calculate the corresponding association score based on the joint probability of each adjacent character pair and its individual character probability; The adjacent character pairs with the highest association scores are merged into new subwords, and the new subwords replace the corresponding adjacent character pairs in the current vocabulary to obtain the updated vocabulary. The size of the vocabulary is obtained by counting the non-single-character sequences in the updated vocabulary. Determine whether the vocabulary size has reached the preset size. If so, use the updated vocabulary as the preset professional vocabulary sub-vocabulary. Otherwise, use the updated vocabulary as the current vocabulary for the next sub-vocabulary merging process.

7. The speech recognition method based on self-attention mechanism enhancement according to claim 1, characterized in that, The acoustic recognition model is obtained in the following way: Acquire several local speech data and their locally recognized text; Based on several local speech data and their local recognized text, a local speech recognition model is trained to obtain a trained local speech recognition model. The model parameters of the trained local speech recognition model are sent to the central server, so that the central server can perform a weighted average aggregation of the model parameters of all local speech recognition models using a federated averaging algorithm to obtain the global model parameters. The global model parameters are pruned and quantized to obtain lightweight global model parameters; Receive the lightweight global model parameters sent by the central server; The acoustic recognition model is deployed based on the lightweight global model parameters.

8. The speech recognition method based on self-attention mechanism enhancement according to claim 7, characterized in that, Based on several local speech data and their local recognized text, a local speech recognition model is trained to obtain a trained local speech recognition model, including: Data augmentation is performed on each local speech data to obtain the corresponding augmented speech data; All local speech data and all augmented speech data are used together as training samples; For each training sample, spectral features are extracted to obtain the corresponding local spectral features; All local spectral features and their corresponding local recognized texts are input into the local speech recognition model to be trained, so that the local speech recognition model is trained with local spectral features as input and predicted recognized text as output. In each training process, the loss function is calculated based on the predicted recognized text and the local recognized text, and the model parameters of the local speech recognition model are adjusted according to the loss function until the local speech recognition model converges, and the trained local speech recognition model is obtained.

9. A speech recognition device based on a self-attention mechanism, characterized in that, include: The system includes a speech data acquisition module, a spectral feature extraction module, a text generation module, a text segmentation module, a label prediction module, a term combination module, an attention projection module, a hybrid feature generation module, and a linear decoding module. The voice data acquisition module is used to acquire the voice data to be recognized; The spectral feature extraction module is used to extract spectral features from speech data to obtain spectral features; The text generation module is used to input the spectral features into a preset acoustic recognition model, so that the acoustic recognition model performs speech decoding on the spectral features to obtain decoded text; The text segmentation module is used to segment the decoded text to obtain several sub-words; The tag prediction module is used to predict the tag for each subword and determine the corresponding tag type; the tag type includes: term start, term middle, or non-term; The terminology combination module is used to construct several professional terms by using non-terminology sub-words as separators and the sub-words starting from the term as the starting point, and sequentially concatenating the starting point and the sub-words in the middle of the consecutive terms adjacent to the starting point. The attention projection module is used to linearly project each technical term to obtain the corresponding multi-head self-attention vector. The hybrid feature generation module is used to replace the corresponding sub-words in the decoded text with all multi-head self-attention vectors to generate a hybrid feature sequence; The linear decoding module is used to perform linear decoding on the mixed feature sequence to obtain speech recognition text.

10. The speech recognition device based on self-attention mechanism enhancement according to claim 9, characterized in that, Also includes: Voice data noise reduction module; The speech data denoising module is used to input speech data into a preset denoising model to generate denoised speech data.

Images