Speech recognition system based on improved Transform architecture

Abstract

The invention belongs to the field of artificial intelligence and voice recognition, and particularly relates to a voice recognition system based on an improved Transform architecture, which comprises a self-positioning module used for receiving an original audio signal, outputting a self-supervised voice feature vector and a traditional audio feature vector in parallel, and sending the self-supervised voice feature vector and the traditional audio feature vector to a feature normalization conversion module; the feature normalization conversion module is used for mapping the self-supervised voice feature vector and the traditional audio feature vector to a standard speaker feature space and outputting a normalized feature; the perception modeling module performs multi-scale time sequence coding through an improved Transform structure, and outputs a voice semantic probability distribution sequence; the CTC loss module is used for optimizing the acoustic model according to the voice semantic probability distribution sequence; the collaboration unit is used for receiving multiple paths of original audio features, screening credible channels from the obtained synchronization features, and outputting corrected features; and the fusion filtering module is used for receiving the local features and the corrected features, generating global probability distribution through attention weight fusion, and decoding the global probability distribution into a final text sequence.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and speech recognition, and in particular to a speech recognition system based on an improved Transformer architecture. Background Art

[0002] With the rapid development of deep learning technology in the field of speech processing, automatic speech recognition (ASR) technology has made significant progress and is widely used in multiple practical scenarios such as intelligent assistants, voice search, human-computer interaction, and subtitle generation. Traditional speech recognition systems usually adopt a multi-stage modular architecture, such as a combination of Gaussian mixture model-hidden Markov model (GMM-HMM) and deep neural network (DNN). This type of architecture divides the speech processing process into multiple stages such as feature extraction, acoustic modeling, language modeling, and decoding. Although the structure is clear, it relies on a large number of manually designed features (such as MFCC, PLP, etc.) and complex prior structure configurations. The overall training process is complex, making it difficult to achieve end-to-end optimization, and the recognition accuracy is limited in non-ideal environments.

[0003] To further improve speech recognition performance, deep neural network models centered on end-to-end concepts have become increasingly mainstream in recent years. Sequence modeling techniques based on the Transformer architecture are particularly prominent. The Transformer, with its multi-head attention mechanism and parallel processing capabilities, has achieved breakthroughs in natural language processing (NLP) and has been introduced into speech recognition tasks for contextual modeling and semantic extraction of audio sequences. Compared to traditional RNN or CNN models, the Transformer possesses stronger global modeling capabilities and is better able to capture long-range dependencies.

[0004] However, the original Transformer structure still has several shortcomings when applied in the field of speech recognition. First, the Transformer has poor robustness to the original audio features, especially under non-ideal conditions such as noisy environments, speakers with strong accents, or uneven speaking speeds, the recognition accuracy drops significantly. Second, the Transformer lacks an adaptive processing mechanism for speaker characteristics and is easily affected by speaker variability. Third, the Transformer is usually trained using standard cross-entropy loss, and the label sequence of speech recognition is often not aligned with the input audio length, which can lead to difficulties in model training convergence or unstable performance.

[0005] To address these issues, current research is incorporating technologies such as self-supervised learning, feature space transformations (such as FMLLR), and the Connectionist Temporal Classification (CTC) loss function into end-to-end speech recognition architectures to improve the model's adaptability and generalization capabilities for complex speech scenarios. For example, self-supervised pre-training models like Wav2Vec 2.0 can learn rich contextual features from large amounts of unlabeled audio; the FMLLR feature transformation technique can normalize the feature distributions of different speakers; and the CTC loss effectively addresses the misalignment between audio and text label sequences.

[0006] Therefore, there is an urgent need to design a speech recognition system architecture that integrates highly robust audio feature extraction, speaker adaptation capabilities and timing alignment optimization strategies, which can not only utilize the powerful modeling capabilities of Transformer but also overcome its limitations in practical applications, thereby improving the recognition accuracy and practicality of the system in complex speech environments. Summary of the Invention

[0007] The purpose of the present invention is to provide an end-to-end speech recognition method and system based on an improved Transformer structure. By integrating a self-supervised learning feature extraction mechanism, a feature space normalization conversion mechanism, and a temporal alignment loss optimization mechanism, the robustness and recognition accuracy of the speech recognition system in complex acoustic environments are effectively improved.

[0008] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a speech recognition system based on an improved Transformer architecture, comprising:

[0009] The self-localization module receives the original audio signal, outputs the self-supervised speech feature vector and the traditional audio feature vector in parallel, and sends them to the feature normalization conversion module;

[0010] a feature normalization conversion module, connected to the self-localization module, configured to receive the self-supervised speech feature vector and the traditional audio feature vector, map them to a standard speaker feature space, and output normalized features;

[0011] The perceptual modeling module is connected to the feature normalization conversion module, which is used to receive normalized features and perform multi-scale temporal encoding through the improved Transformer structure to output a speech semantic probability distribution sequence;

[0012] The CTC loss module is connected to the perceptual modeling module and is used to receive the speech semantic probability distribution sequence, calculate the alignment loss between the label sequence and the input sequence through the CTC loss function, and update the model parameters in reverse order to obtain the optimized acoustic model;

[0013] The collaborative unit is used to receive multiple channels of original audio features, perform time shift correction on the feature sequence, filter the obtained synchronization features for trusted channels, and output the corrected features to the fusion filter module;

[0014] The fusion filtering module is connected to the self-positioning module and the collaborative unit optimization module respectively. It is used to receive local features and corrected features, generate global probability distribution through attention weight fusion, and decode them into the final text sequence.

[0015] The self-positioning module comprises:

[0016] The self-supervised speech feature extraction submodule uses the Wav2Vec 2.0 or HuBERT model to extract context-dependent deep temporal feature vectors using a neural network based on contrastive learning or mask modeling.

[0017] Audio feature extraction submodule, used to extract MFCC feature vectors;

[0018] The vector splicing module is used to map the deep time series feature vector and the MFCC feature vector to the standard speaker feature space through the FMLLR transformation matrix and input it into the feature normalization conversion module.

[0019] The perception modeling module includes:

[0020] A multi-scale attention mechanism module is used to extract context-related features at different time spans from the input speech feature sequence and output the context feature vector to the gated feedforward network unit;

[0021] The gated feedforward network unit is used to dynamically filter features by introducing a gating mechanism, thereby generating a gated feature vector and sending it to the post-processing unit;

[0022] The gated feedforward network unit includes: a first sub-network and a second sub-network;

[0023] The first sub-network is used to perform linear transformation and nonlinear activation operations to extract the main semantic representation;

[0024] The second sub-network serves as a gating path, used to generate a set of gating weight coefficients to represent the importance of each channel or feature dimension;

[0025] The outputs of the first and second sub-network paths are element-by-element weighted fused, that is, the semantic features of the main path are multiplied by the weights of the gating path point by point to achieve feature regulation and information screening.

[0026] The post-processing unit is used to fuse the absolute position coding of each frame of the gated feature vector with the relative position coding of the frame in the entire sequence, generate a speech feature vector after position embedding processing, and extract the language semantic content contained in the current speech frame. Finally, the semantic probability distribution P corresponding to each frame is generated through the output module.

[0027] The post-processing unit includes: a hybrid position embedding unit and a Transformer decoder;

[0028] When the gated feature vector is in the hybrid position embedding unit, it serves as the core input for the joint embedding of speech content features and temporal position information;

[0029] The hybrid position embedding unit includes: a position fusion module for fusing the absolute position code of each frame with the relative position code of the frame in the entire sequence, generating a speech feature vector after position embedding, and sending it to the Transformer decoder;

[0030] In the Transformer decoder, it is used to perform cross-attention calculation with the speech feature vector after position embedding processing, extract the language semantic content contained in the current speech frame, and finally generate the semantic probability distribution corresponding to each frame through the output module.

[0031] The collaborative unit includes:

[0032] The delay compensation module is connected to the self-positioning module and is used to receive multiple channels of original audio features, perform time shift correction on the feature sequence based on the timestamp deviation, and output synchronization features;

[0033] The optimization module is connected to the delay compensation module and is used to receive multi-channel synchronous features, screen the trusted channels according to the signal-to-noise ratio and delay stability score, and output the corrected features to the fusion filtering module.

[0034] A recognition method of a speech recognition system based on an improved Transformer architecture includes the following steps:

[0035] S1: The raw audio signal received from the voice acquisition device is input into the self-supervised feature extraction module, which performs unsupervised feature encoding and extracts context-aware temporal feature vectors;

[0036] S2: Input the time series feature vector into the feature normalization conversion module, map its audio features to the standard speech feature space, obtain normalized speech features, and send them to the improved Transformer modeling module;

[0037] S3: The improved Transformer modeling module processes the normalized features and outputs a speech and semantic probability distribution sequence to the CTC loss module;

[0038] S4: Calculate the alignment loss between the label sequence and the semantic probability distribution P through the CTC loss function, update the model parameters in reverse, and the CTC loss module outputs the optimized acoustic model;

[0039] S5: When there are multiple speech input sources, the delay compensation module corrects the time deviation and generates synchronization features. The collaborative unit optimization module selects the credible feature subset and outputs the corrected feature set.

[0040] S6: The fusion filter module obtains the modified feature set and acoustic model M, generates a probability distribution through the model M, fuses the multi-source probabilities, and performs Beam Search decoding to output the final recognized text T.

[0041] The improved Transformer modeling module processes the normalized features and outputs a speech and semantic probability distribution sequence, including the following steps:

[0042] S3-1: The perception modeling module receives the context feature vector output by the multi-scale attention mechanism module and inputs the feature vector into two parallel sub-networks;

[0043] S3-2: The first subnetwork is used to perform linear transformation and nonlinear activation operations to extract the main semantic representation, that is, H = ReLU(X·W1·b1); where X is the feature vector matrix, W1 is the weight matrix of the first linear transformation, and b1 is the bias vector of the first linear transformation;

[0044] The second subnetwork serves as a gating path to generate a set of gating weight coefficients that characterize the importance of each channel or feature dimension, i.e., G = σ(X·W2·b2); where X is the eigenvector matrix, W2 is the weight matrix of the second linear transformation, and b2 is the bias vector of the second linear transformation.

[0045] S3-3: Perform element-by-element weighted fusion on the outputs of the first and second sub-network paths, that is, perform point-by-point multiplication of the semantic features of the main path and the weights of the gated path, namely:

[0046] Gated output Output = H⊙G, where ⊙ is a point-by-point multiplication operation; this achieves feature control and information screening;

[0047] S3-4: The gated feedforward network unit inputs the fused gated feature vector to the post-processing unit, which fuses the absolute position code of each frame with the relative position code of the frame in the entire sequence to generate a speech feature vector after position embedding, and extracts the language semantic content contained in the current speech frame. Finally, the semantic probability distribution corresponding to each frame is generated through the output module.

[0048] The step S4 comprises the following steps:

[0049] S4-1: The CTC loss module receives the probability distribution sequence output by the perceptual modeling module, which contains the predicted probability of each character at each time step, and receives the annotated real text label sequence.

[0050] S4-2: insert special blank tags at the beginning, end, and between every two characters of the real tag sequence;

[0051] S4-3: Execute forward algorithm:

[0052] S4-3-1: Initialize the probability of the first time step, that is, only allow blank labels or the first true label to be valid;

[0053] S4-3-2: Calculate the subsequent time steps frame by frame;

[0054] S4-3-3: If the current extended label is blank or contains repeated characters, the probability of maintaining the previous state or moving to the current label is accumulated;

[0055] S4-3-4: If the current label is a new character, the probability of skipping the blank label is additionally accumulated.

[0056] S4-4: Loss calculation and back propagation:

[0057] S4-4-1: Summarize the sum of probabilities of all valid paths;

[0058] S4-4-2: Take the negative logarithm of the sum of probabilities as the CTC loss;

[0059] S4-4-3: Reversely update the acoustic model parameters through the gradient descent algorithm to optimize the alignment capability.

[0060] The step S5 comprises the following steps:

[0061] S5-1: Perform feature extraction on each input audio channel independently and generate multiple sets of normalized feature sequences;

[0062] S5-2: Select one audio channel as the reference channel and calculate the cross-correlation function between the target channel and the reference channel. That is, compare the similarity of the two feature sequences by sliding them over time. The time offset corresponding to the highest similarity point is the delay value.

[0063] S5-3: Shift the target channel feature sequence forward or backward as a whole according to the detected delay value;

[0064] S5-4: Calculate the comprehensive credibility score of each channel:

[0065] Calculate the signal-to-noise ratio, which is the ratio of pure speech energy to noise energy;

[0066] Calculate stability, which is the inverse of the variance of the characteristic sequence. The smaller the fluctuation, the higher the score;

[0067] S5-5: Channels with a score ≥ 0.8 and a delay ≤ 200 milliseconds are retained, and out-of-tolerance or out-of-delay channels are discarded and an alarm is triggered to obtain a set of synchronization features that pass the screening.

[0068] The step S6 comprises the following steps:

[0069] S6-1: Input the features of each trusted channel into the optimized acoustic model and output the character probability distribution sequence of the corresponding channel;

[0070] S6-2: Calculate the similarity between the probability distribution of each channel and the current global distribution, and generate the fusion weight based on the channel credibility score;

[0071] The probability distribution of all channels is weighted and summed according to the weights to update the global probability distribution;

[0072] S6-3: Beam Search decoding:

[0073] Keep the current five best text paths and their cumulative scores, add new characters or keep blank labels at the end of each path; new characters include: all recognizable characters and blank labels;

[0074] S6-4: Calculate the cumulative total score and select the path with the highest cumulative score as the final recognized text.

[0075] Acoustic score: the logarithmic probability of the character in the global probability distribution;

[0076] The cumulative total is calculated as follows:

[0077] Language model score: the probability of the character being continued given by the external language model;

[0078] Cumulative total score = original path score + acoustic score + 30% × language model score.

[0079] The present invention has the following beneficial effects and advantages:

[0080] 1. The present invention is highly robust: the present invention improves the model's adaptability to noise, accent, and speech rate variations through self-supervised deep speech features and FMLLR feature normalization processing.

[0081] 2. The present invention does not require precise alignment: The introduction of the CTC loss function in the present invention frees the model training process from the dependence on frame-level alignment, greatly reducing the training threshold and data preparation costs.

[0082] 3. The recognition accuracy of the present invention is high: the present invention adopts an improved Transformer structure to effectively fuse local and global information and enhance the semantic consistency modeling capability.

[0083] 4. The modular design of the system of the present invention: each module has independent functions and can be freely combined and tailored according to actual applications, and is suitable for deployment in multiple scenarios.

[0084] 5. The fusion decoding mechanism of the present invention is flexible: through multi-source fusion and trusted path selection mechanism, the stability and interpretability of the system's overall recognition output are improved.

[0085] 6. The present invention has strong security: the introduction of collaborative unit optimization modules can effectively eliminate low-credibility or potential attack sources, and improve the system's anti-interference ability and application reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 Flowchart of the enhanced speech recognition method based on the Whisper model of the present invention;

[0087] Figure 2 This is a schematic diagram of the structure of the improved Transformer modeling module of the present invention;

[0088] Figure 3 This is a schematic diagram of the coding and decoding structure of the present invention. DETAILED DESCRIPTION

[0089] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0090] like Figure 1 As shown, this is a flow chart of the enhanced speech recognition method based on the Whisper model of the present invention. The present invention is a speech recognition system based on an improved Transformer architecture, and designs a modular, multi-source information fusion, and high-confidence decoding output speech recognition system architecture. The system integrates a self-positioning module, a perception module, a feature normalization module, a deep modeling module, a delay compensation module, a collaborative optimization module and a fusion filtering module to support multi-dimensional information modeling under different speech source input conditions. It has the advantages of strong scalability, high environmental adaptability, and excellent decoding accuracy.

[0091] 1. System Architecture

[0092] (1) Self-positioning module

[0093] The self-localization module receives the original audio signal, outputs the self-supervised speech feature vector and the traditional audio feature vector in parallel, and sends them to the feature normalization conversion module;

[0094] Self-positioning module, including:

[0095] The self-supervised speech feature extraction submodule uses the Wav2Vec 2.0 or HuBERT model to extract context-dependent deep temporal feature vectors using a neural network based on contrastive learning or mask modeling.

[0096] Audio feature extraction submodule, used to extract MFCC feature vectors;

[0097] (2) Feature Normalization Conversion Module

[0098] Improving the robustness of cross-speaker recognition.

[0099] a feature normalization conversion module, connected to the self-localization module, configured to receive the self-supervised speech feature vector and the traditional audio feature vector, perform speaker-adaptive normalization transformation on the speech features using FMLLR (Feature-space Maximum Likelihood Linear Regression) technology, map the transformed speech features to a standard speaker feature space, and output normalized features;

[0100] (3) Perception Modeling Module

[0101] like Figure 2 As shown in the figure, it is a schematic diagram of the structure of the improved Transformer modeling module of the present invention. The perception modeling module is connected to the feature normalization conversion module, which is used to receive normalized features and perform multi-scale temporal coding through the improved Transformer structure to output a speech semantic probability distribution sequence;

[0102] The perception modeling module is an improved Transformer modeling module, which includes:

[0103] The multi-scale attention mechanism module first extracts context-related features at different receptive fields (i.e., different time spans) from the input speech feature sequence. This mechanism can simultaneously focus on local details and global structure, thereby generating multi-dimensional and multi-granular attention representations. The output context feature vector is sent to the gated feedforward network unit;

[0104] The gated feedforward network unit is used to dynamically filter features by introducing a gating mechanism, thereby generating a gated feature vector and sending it to the post-processing unit;

[0105] Gated feedforward network performs feature regulation on attention output

[0106] The feature vectors output by the multi-scale attention model are then fed into a gated feedforward network unit. This unit incorporates a gating mechanism that controls channel weights, dynamically filtering, reweighting, or selectively enhancing incoming semantic information based on the importance of different channels. This mechanism improves the model's ability to focus on key information while suppressing irrelevant or redundant features.

[0107] A gated feedforward network unit, comprising: a first sub-network and a second sub-network;

[0108] The first sub-network is used to perform linear transformation and nonlinear activation operations to extract the main semantic representation;

[0109] The second sub-network serves as a gating path, used to generate a set of gating weight coefficients to represent the importance of each channel or feature dimension;

[0110] The outputs of the first and second sub-network paths are element-by-element weighted fused, that is, the semantic features of the main path are multiplied by the weights of the gating path point by point to achieve feature regulation and information screening.

[0111] Feature transfer path and target after gating

[0112] After filtering and reconstruction, the gated feature vector is passed to the post-processing unit: the hybrid position embedding unit or the decoder module as an enhanced semantic representation.

[0113] The post-processing unit is used to fuse the absolute position coding of each frame of the gated feature vector with the relative position coding of the frame in the entire sequence, generate a speech feature vector after position embedding processing, and extract the language semantic content contained in the current speech frame. Finally, the semantic probability distribution P corresponding to each frame is generated through the output module.

[0114] Its functions include but are not limited to:

[0115] When the gated feature vector is in the hybrid position embedding unit, it serves as the core input for the joint embedding of speech content features and temporal position information;

[0116] The hybrid position embedding unit includes: a position fusion module for fusing the absolute position code of each frame with the relative position code of the frame in the entire sequence, generating a speech feature vector after position embedding, and sending it to the Transformer decoder;

[0117] In the Transformer decoder, it is used to perform cross-attention calculation with the speech feature vector after position embedding processing, extract the language semantic content contained in the current speech frame, and finally generate the semantic probability distribution corresponding to each frame through the output module, such as Figure 3 shown.

[0118] The above hybrid position is embedded in the unit and the following operations are performed:

[0119] 1. Generation of absolute position code

[0120] First, the system generates an encoding vector based on the time step of each frame in the speech sequence, representing the frame's absolute position within the overall speech sequence. This vector is typically constructed based on fixed rules, such as periodic functions or predefined numerical patterns, and therefore does not rely on model training parameters. This is publicly available existing technology.

[0121] 2. Generation of relative position encoding

[0122] The system then calculates the relative position between any two time frames in the speech sequence and converts this into an encoding vector that can be used in the attention mechanism. This encoding reflects the distance or order difference between one time frame and another. This approach is often used for long-distance modeling and is one of the existing extensions of the attention mechanism.

[0123] 3. Fusion of Positional Encoding

[0124] The absolute position code of each frame is fused with the relative position code of its related frames in the entire sequence. This fusion method involves first concatenating or weighting the speech feature vector with the absolute position code to generate an initial position enhancement vector. This vector is then further adjusted based on its relative position information with other time frames to incorporate more context-aware information. This fusion process is accomplished by the position fusion module in the structure of the present invention and is an innovative aspect of the present invention.

[0125] 4. Semantic Modeling and Output

[0126] After positional embedding, the speech feature vector is fed into the subsequent attention encoder or decoder for context modeling. The model uses this information to extract the semantic content of the current speech frame, ultimately generating a semantic probability distribution for each frame through the output module. For example, the probability of a phoneme, pinyin, word, subword, or other unit appearing in the current frame can be output.

[0127] Summary of the overall action chain

[0128] In short, the gated feedforward network acts as a "semantic information regulator," bridging the gap between multi-scale attention and semantic modeling. Its output not only preserves the rich contextual information extracted by attention but also enhances the accuracy and robustness of semantic representation through a gating mechanism. This output is subsequently used as the input for key tasks such as temporal-aware modeling, target output prediction, or alignment calculation.

[0129] (4) CTC loss module

[0130] The CTC loss module is connected to the temporal classification loss module and the perceptual modeling module. It receives the speech semantic probability distribution sequence, calculates the alignment loss between the label sequence and the input sequence through the CTC loss function, and updates the model parameters in reverse order to obtain the optimized acoustic model.

[0131] (5) Collaborative unit

[0132] The collaborative unit is used to receive multiple channels of original audio features, perform time shift correction on the feature sequence, filter the obtained synchronization features for trusted channels, and output the corrected features to the fusion filter module;

[0133] Collaborative units, including:

[0134] The delay compensation module is connected to the self-positioning module and is used to receive multiple channels of raw audio features (such as remote microphones, voice sensors, etc.), perform time shift correction on the feature sequence based on the timestamp deviation, and output synchronization features;

[0135] The optimization module is connected to the delay compensation module and is used to receive multi-channel synchronous features, screen the trusted channels according to the signal-to-noise ratio and delay stability score, and output the corrected features to the fusion filtering module.

[0136] (6) Fusion filter module

[0137] The fusion filtering module is connected to the self-positioning module and the collaborative unit optimization module respectively. It is used to receive local features and modified features, generate a global probability distribution through attention weight fusion, integrate multiple speech feature sources, and finally output the globally optimal text sequence decoding result.

[0138] 2. Speech Recognition Method and Steps

[0139] To achieve the coordinated operation of the above systems, the speech recognition method based on the improved Transformer structure proposed in this invention includes the following key steps:

[0140] Step 1: Self-supervised speech feature extraction

[0141] The original audio signal received from the voice acquisition device is input into the self-supervised feature extraction module, which performs unsupervised feature encoding through a neural network based on contrastive learning or mask modeling (such as Wav2Vec 2.0, HuBERT, etc.) to extract context-aware temporal feature vectors to replace traditional low-level perceptual features.

[0142] Step 2: Speaker feature normalization

[0143] The features from step 1 are fed into the FMLLR feature conversion module. During the training phase, this module estimates the linear transformation matrix for each speaker and then maps their audio features into a standard speech feature space, effectively eliminating distribution bias between speakers and improving cross-speaker consistency of the model.

[0144] Step 3: Deep Speech Sequence Modeling

[0145] The normalized speech features are input into the improved Transformer modeling module. This module includes the following innovative structures:

[0146] Multi-scale attention mechanism: Adopting attention mechanisms with different window sizes in the temporal dimension to capture both local pronunciation and global semantics;

[0147] Gated feedforward network: Improves the ability to express nonlinear information by introducing a gating mechanism (such as GLU); Position embedding hybrid strategy: Fusion of absolute position encoding and relative position perception makes the model more sensitive to the temporal order in speech sequences.

[0148] Finally, the speech semantic probability distribution sequence is output to the CTC loss module;

[0149] Step 4: End-to-end training without alignment (CTC optimization)

[0150] The probability distribution sequence output in step 3 is trained and optimized using the Connectionist Temporal Classification (CTC) loss function. CTC allows blank labels in the output sequence and supports parallel modeling of multiple possible paths. It is suitable for speech recognition tasks where the lengths of speech and text sequences are inconsistent.

[0151] S4-1: The CTC loss module receives the probability distribution sequence output by the perceptual modeling module, which contains the predicted probability of each character at each time step, and receives the annotated real text label sequence.

[0152] S4-2: insert special blank tags at the beginning, end, and between every two characters of the real tag sequence;

[0153] S4-3: Execute forward algorithm:

[0154] S4-3-1: Initialize the probability of the first time step, that is, only allow blank labels or the first true label to be valid;

[0155] S4-3-2: Calculate the subsequent time steps frame by frame;

[0156] S4-3-3: If the current extended label is blank or contains repeated characters, the probability of maintaining the previous state or moving to the current label is accumulated;

[0157] S4-3-4: If the current label is a new character, the probability of skipping the blank label is additionally accumulated.

[0158] S4-4: Loss calculation and back propagation:

[0159] S4-4-1: Summarize the sum of probabilities of all valid paths;

[0160] S4-4-2: Take the negative logarithm of the sum of probabilities as the CTC loss;

[0161] S4-4-3: Reversely update the acoustic model parameters through the gradient descent algorithm to optimize the alignment capability.

[0162] Step 5: Multi-source information sharing and collaborative selection

[0163] When there are multiple voice input sources (such as multiple microphone arrays, front and rear vehicle voice input, etc.), the collaborative unit optimization module scores the quality, stability and source of the input signal, automatically filters out trusted input channels, eliminates abnormal sources and malicious interference, and improves the security of the recognition system.

[0164] S5-1: Perform feature extraction on each input audio channel independently and generate multiple sets of normalized feature sequences;

[0165] S5-2: Select one audio channel as the reference channel and calculate the cross-correlation function between the target channel and the reference channel. That is, compare the similarity of the two feature sequences by sliding them over time. The time offset corresponding to the highest similarity point is the delay value.

[0166] S5-3: Shift the target channel feature sequence forward or backward as a whole according to the detected delay value;

[0167] S5-4: Calculate the comprehensive credibility score of each channel:

[0168] Calculate the signal-to-noise ratio, which is the ratio of pure speech energy to noise energy;

[0169] Calculate stability, which is the inverse of the variance of the characteristic sequence. The smaller the fluctuation, the higher the score;

[0170] S5-5: Channels with a score ≥ 0.8 and a delay ≤ 200 milliseconds are retained, and out-of-tolerance or out-of-delay channels are discarded and an alarm is triggered to obtain a set of synchronization features that pass the screening.

[0171] Step 6: Delay Alignment and Correction

[0172] For the speech feature input selected in step 5, the delay compensation module is used to correct its acquisition time to achieve timing synchronization and avoid recognition errors caused by input lag.

[0173] S6-1: Input the features of each trusted channel into the optimized acoustic model and output the character probability distribution sequence of the corresponding channel;

[0174] S6-2: Calculate the similarity between the probability distribution of each channel and the current global distribution, and generate the fusion weight based on the channel credibility score;

[0175] The probability distribution of all channels is weighted and summed according to the weights to update the global probability distribution;

[0176] S6-3: Beam Search decoding:

[0177] Keep the current five best text paths and their cumulative scores, add new characters or keep blank labels at the end of each path; new characters include: all recognizable characters and blank labels;

[0178] S6-4: Calculate the cumulative total score and select the path with the highest cumulative score as the final recognized text.

[0179] Acoustic score: the logarithmic probability of the character in the global probability distribution;

[0180] The cumulative total is calculated as follows:

[0181] Language model score: the probability of the character being continued given by the external language model;

[0182] Cumulative total score = original path score + acoustic score + 30% × language model score.

[0183] Step 7: Fusion decoding and recognition output

[0184] The speech features from step 1 and the delayed, corrected, trusted features from step 6 are fed into the fusion filter module. A robust Kalman filter or attention fusion mechanism is used to output a fused unified probability distribution. Finally, the Beam Search decoder, combined with an external language model, decodes and outputs the final recognized text.

[0185] Example:

[0186] This example proposes a speech recognition system based on the Transformer architecture. By integrating autocorrelation learning features, speaker-adaptive feature conversion technology, and a CTC-constrained temporal alignment loss mechanism, it achieves high-precision, high-inclination, and high-selectivity speech recognition capabilities. The specific steps are as follows:

[0187] S10: Audio Input and Wav2Vec 2.0 Feature Extraction

[0188] The input audio data undergoes format processing to ensure that the format meets the input requirements. Then, the pre-trained Wav2Vec2.0 network is used to extract autocorrelation features of the audio. This feature can demonstrate strong speech encoding capabilities in local and time domain dimensions, supporting the subsequent Transformer network to achieve more efficient baseline structure decoding.

[0189] S20: FMLLR feature space conversion control

[0190] In order to balance the irregularities among various speakers, application scenarios, acoustic models and input feature distributions, the present invention applies FMLLR technology to feature processing, quantitatively calculates the transformation matrix of the input features based on the training results, and ensures that the feature input structures of different speakers form a regular distribution that conforms to the system learning method.

[0191] S30: Improve Transformer acoustic model encoding and decoding

[0192] Input features are passed to the improved Transformer acoustic component to implement Multi-Scale Attention. This attention is based on the mean attention of various kernel sizes and integrates the itemized channels of short-term and long-term scales to enhance the Transformer network's understanding of actual audio.

[0193] At the same time, a gated feedforward network (GatedFeedforwardNetwork) is used to combine two layers of nonlinear output and a layer of Sigmoid gate value to enhance the selectivity of features and promote the preference analysis capability of the Transformer network.

[0194] S40: CTC loss timing constraints and aligned decoding

[0195] To address the alignment issue between very long audio input and very short output text, the system is trained using a Connectionist Temporal Classification (CTC) loss constraint. This allows for encoding and decoding of temporal features to character sequences without the need for positioning markers, significantly reducing reliance on alignment data.

[0196] S50: Beam Search + Language Model Decoding and Compilation

[0197] Beam Search is implemented on the probability sequence output by Transformer to select multiple optimal paths. At the same time, an external language model is integrated to perform the final text encoding and decoding. The output results are stored in the form of a data link to facilitate the verification and evaluation of each audio input segment.

[0198] S60: Reasonable system calls and application strategies

[0199] Depending on the application scenario, the system can call different strategies to implement different recognition model combinations such as multi-model synchronization, layered fusion, and single-line optimal selection, to achieve high-precision recognition and rapid response in scenarios such as smart vehicles, smart sensing, voice search, and smart home interaction.

[0200] In step S10, the system needs to obtain basic input features in the speech recognition task, that is, to extract high-quality speech representation information for subsequent model processing and recognition optimization. This step includes the following processes S11 to S12:

[0201] S11: Voice input preprocessing

[0202] The system first receives the raw speech signal. To meet the input requirements of subsequent models, the audio signal undergoes format standardization, including sampling rate unification, channel processing, and amplitude normalization. This series of preprocessing measures ensures consistency in format, duration, and audio quality across audio samples from different sources, thereby improving the system's robustness and adaptability.

[0203] S12: Self-supervised speech feature extraction

[0204] After preprocessing, the audio is fed into a pretrained Wav2Vec 2.0 model for feature extraction. Wav2Vec 2.0, a self-supervised learning model, extracts semantically rich temporal features from raw speech. These extracted features not only contain essential acoustic information but also incorporate context-dependent semantic relationships, providing high-quality input for subsequent speech recognition.

[0205] In step S20, the system needs to perform speaker-adaptive conversion on the input speech features to improve the robustness and generalization ability of the model in multi-speaker scenarios. This step includes the following processes S21-S22: S21: Establish a speaker feature conversion model

[0206] To mitigate the impact of differences in speaking style, accent, and pitch between speakers, the system uses speaker-adaptive FMLLR technology. This method analyzes speaker characteristics in the audio and establishes personalized transformation parameters, thereby aligning and normalizing the input speech features of different speakers.

[0207] S22: Output fusion after feature conversion

[0208] Based on the transformation parameters, the original features are mapped into a unified feature space. This process effectively suppresses model performance fluctuations caused by speaker variability, making the input features more concentrated and more conducive to training and recognition. The transformed features are cached and input into the Transformer acoustic model for subsequent modeling.

[0209] In step S30, the system needs to perform deep modeling on the processed speech features and extract the temporal dependency and semantic association information in the speech sequence through the improved Transformer structure. This step includes the following processes S31 to S32:

[0210] S31: Multi-scale Attention Mechanism for Enhanced Speech Understanding

[0211] The features are fed into an improved Transformer acoustic model. To adapt to the varying timescales of speech, the model incorporates a multi-scale attention mechanism. This mechanism focuses on both short-term and long-term speech information, accurately capturing key speech details such as morphemes, pauses, and intonation, and enhancing the model's temporal perception.

[0212] S32: Introducing a gating mechanism to enhance expressiveness

[0213] A gating mechanism is further introduced into the model structure to effectively enhance the model's ability to express speech information through automatic screening and control of feature information. This mechanism allows the model to dynamically adjust its focus on different speech features, thereby improving overall recognition accuracy and adaptability to complex speech environments.

[0214] S40: In step S40: the system needs to solve the misalignment problem between speech and text sequences and implement effective speech-to-text mapping by introducing an unsupervised alignment mechanism. This step includes the following processes S41-S42:

[0215] S41: Alignment-free modeling using CTC

[0216] To address the common problem of speech recognition where the length of speech and text differs, the system uses the CTC method for training. CTC eliminates the need for precise label alignment and directly maps speech input to character sequences, greatly simplifying the annotation process while enhancing the model's ability to handle long speech and irregular speech rates.

[0217] S42: Output preliminary recognition text sequence

[0218] Through the training and inference of the CTC algorithm, the system can initially output a series of character sequences as recognition results. This sequence will then be entered into the language model module as a candidate for further screening and optimization to obtain a higher confidence recognition output.

[0219] In step S50: the system needs to perform multi-path reasoning and language model-assisted optimization on the recognition results to output more accurate text results that conform to language semantics. This step includes the following processes S51~S52: S51: Multi-path search improves recognition accuracy

[0220] To select the optimal result from multiple possible recognition paths, the system uses a beam search algorithm, retaining multiple high-scoring paths for parallel analysis. This method can significantly reduce the recognition error rate and improve the accuracy of the final result.

[0221] S52: Fusion language model to improve semantic rationality

[0222] Based on the recognition path, the system uses a language model to perform a secondary verification of the output results. The language model uses context to determine the linguistic fluency and semantic rationality of the output content, effectively avoiding problems such as spelling errors and grammatical incoherence. Ultimately, the result with the highest language model support is selected as the system's recognition output.

[0223] In step S60: the system needs to call the optimal recognition strategy according to the actual application scenario, and standardize the management and data storage of the recognition output. This step includes the following processes S61~S62: S61: Automatically switch the recognition scheme according to the application scenario

[0224] This system supports speech recognition in a variety of application scenarios, including in-vehicle voice control, remote voice command recognition, and home voice interaction. To meet different application requirements, the system automatically selects matching recognition models and processing flows, enabling flexible deployment with on-demand layering and modular combinations.

[0225] S62: Standardize output results and record data

[0226] After recognition is complete, the system encapsulates the output results into a standardized data format, including timestamp, text content, recognition confidence, and other information. All results are stored in the system database to support subsequent verification, evaluation, and iterative model training.

[0227] Through the above steps, the system of the present invention not only has basic recognition functions, but also forms high-precision recognition capabilities that are strong against noise, cross-speakers, time sequence misalignment, and irregular input through the fusion of self-learning features + FMLLR + CTC + Transformer. It is a new Transformer speech recognition solution with application prospects and practical value.

[0228] In conjunction with this embodiment,

[0229] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0230] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A speech recognition system based on an improved Transformer architecture, characterized in that: include: The self-localization module receives the original audio signal, outputs the self-supervised speech feature vector and the traditional audio feature vector in parallel, and sends them to the feature normalization conversion module; a feature normalization conversion module, connected to the self-localization module, configured to receive the self-supervised speech feature vector and the traditional audio feature vector, map them to a standard speaker feature space, and output normalized features; The perception modeling module is an improved Transformer modeling module, which is connected to the feature normalization conversion module. It is used to receive normalized features and perform multi-scale temporal encoding through the improved Transformer structure to output a speech semantic probability distribution sequence; The CTC loss module is connected to the perceptual modeling module and is used to receive the speech semantic probability distribution sequence, calculate the alignment loss between the label sequence and the input sequence through the CTC loss function, and update the model parameters in reverse order to obtain the optimized acoustic model; The collaborative unit is used to receive multiple channels of original audio features, perform time shift correction on the feature sequence, filter the obtained synchronization features for trusted channels, and output the corrected features to the fusion filter module; The fusion filtering module is connected to the self-positioning module and the collaborative unit optimization module respectively. It is used to receive local features and corrected features, generate global probability distribution through attention weight fusion, and decode them into the final text sequence.

2. A speech recognition system based on an improved Transformer architecture according to claim 1, characterized in that: The self-positioning module comprises: The self-supervised speech feature extraction submodule uses the Wav2Vec 2.0 or HuBERT model to extract context-dependent deep temporal feature vectors using a neural network based on contrastive learning or mask modeling. Audio feature extraction submodule, used to extract MFCC feature vectors; The vector splicing module is used to map the deep time series feature vector and the MFCC feature vector to the standard speaker feature space through the FMLLR transformation matrix and input it into the feature normalization conversion module.

3. A speech recognition system based on an improved Transformer architecture according to claim 1, characterized in that: The perception modeling module includes: A multi-scale attention mechanism module is used to extract context-related features at different time spans from the input speech feature sequence and output the context feature vector to the gated feedforward network unit; The gated feedforward network unit is used to dynamically filter features by introducing a gating mechanism, thereby generating a gated feature vector and sending it to the post-processing unit; The gated feedforward network unit includes: a first sub-network and a second sub-network; The first sub-network is used to perform linear transformation and nonlinear activation operations to extract the main semantic representation; The second sub-network serves as a gating path, used to generate a set of gating weight coefficients to represent the importance of each channel or feature dimension; The outputs of the first and second sub-network paths are element-by-element weighted fused, that is, the semantic features of the main path are multiplied by the weights of the gating path point by point to achieve feature regulation and information screening. The post-processing unit is used to fuse the absolute position coding of each frame of the gated feature vector with the relative position coding of the frame in the entire sequence, generate a speech feature vector after position embedding processing, and extract the language semantic content contained in the current speech frame. Finally, the semantic probability distribution P corresponding to each frame is generated through the output module.

4. A speech recognition system based on an improved Transformer architecture according to claim 3, characterized in that: The post-processing unit includes: a hybrid position embedding unit and a Transformer decoder; When the gated feature vector is in the hybrid position embedding unit, it serves as the core input for the joint embedding of speech content features and temporal position information; The hybrid position embedding unit includes: a position fusion module for fusing the absolute position code of each frame with the relative position code of the frame in the entire sequence, generating a speech feature vector after position embedding, and sending it to the Transformer decoder; In the Transformer decoder, it is used to perform cross-attention calculation with the speech feature vector after position embedding processing, extract the language semantic content contained in the current speech frame, and finally generate the semantic probability distribution corresponding to each frame through the output module.

5. A speech recognition system based on an improved Transformer architecture according to claim 1, characterized in that: The collaborative unit includes: The delay compensation module is connected to the self-positioning module and is used to receive multiple channels of original audio features, perform time shift correction on the feature sequence based on the timestamp deviation, and output synchronization features; The optimization module is connected to the delay compensation module and is used to receive multi-channel synchronous features, screen the trusted channels according to the signal-to-noise ratio and delay stability score, and output the corrected features to the fusion filtering module.

6. The recognition method of a speech recognition system based on an improved Transformer architecture according to any one of claims 1 to 5, characterized in that: The following steps are involved: S1: The raw audio signal received from the voice acquisition device is input into the self-supervised feature extraction module, which performs unsupervised feature encoding and extracts context-aware temporal feature vectors; S2: Input the time series feature vector into the feature normalization conversion module, map its audio features to the standard speech feature space, obtain normalized speech features, and send them to the improved Transformer modeling module; S3: The improved Transformer modeling module processes the normalized features and outputs a speech and semantic probability distribution sequence to the CTC loss module; S4: Calculate the alignment loss between the label sequence and the semantic probability distribution P through the CTC loss function, update the model parameters in reverse, and the CTC loss module outputs the optimized acoustic model; S5: When there are multiple speech input sources, the delay compensation module corrects the time deviation and generates synchronization features. The collaborative unit optimization module selects the credible feature subset and outputs the corrected feature set. S6: The fusion filter module obtains the modified feature set and acoustic model M, generates a probability distribution through the model M, fuses the multi-source probabilities, and performs Beam Search decoding to output the final recognized text T.

7. The recognition method of the speech recognition system based on the improved Transformer architecture according to claim 6, characterized in that: The improved Transformer modeling module processes the normalized features and outputs a speech and semantic probability distribution sequence, including the following steps: S3-1: The perception modeling module receives the context feature vector output by the multi-scale attention mechanism module and inputs the feature vector into two parallel sub-networks; S3-2: The first subnetwork is used to perform linear transformation and nonlinear activation operations to extract the main semantic representation, that is, H = ReLU(X·W1·b1); where X is the feature vector matrix, W1 is the weight matrix of the first linear transformation, and b1 is the bias vector of the first linear transformation; The second subnetwork serves as a gating path to generate a set of gating weight coefficients that characterize the importance of each channel or feature dimension, i.e., G = σ(X·W2·b2); where X is the eigenvector matrix, W2 is the weight matrix of the second linear transformation, and b2 is the bias vector of the second linear transformation. S3-3: Perform element-by-element weighted fusion on the outputs of the first and second sub-network paths, that is, perform point-by-point multiplication of the semantic features of the main path and the weights of the gated path, namely: Gated output Output = H⊙G, where ⊙ is a point-by-point multiplication operation; this achieves feature control and information screening; S3-4: The gated feedforward network unit inputs the fused gated feature vector to the post-processing unit, which fuses the absolute position code of each frame with the relative position code of the frame in the entire sequence to generate a speech feature vector after position embedding, and extracts the language semantic content contained in the current speech frame. Finally, the semantic probability distribution corresponding to each frame is generated through the output module.

8. The recognition method of a speech recognition system based on an improved Transformer architecture according to claim 6, characterized in that: The step S4 comprises the following steps: S4-1: The CTC loss module receives the probability distribution sequence output by the perceptual modeling module, which contains the predicted probability of each character at each time step, and receives the annotated real text label sequence. S4-2: Insert special blank tags at the beginning, end, and between every two characters of the real tag sequence; S4-3: Execute the forward algorithm: S4-3-1: Initialize the probability of the first time step, that is, only allow blank labels or the first true label to be valid; S4-3-2: Calculate the subsequent time steps frame by frame; S4-3-3: If the current extended label is blank or contains repeated characters, the probability of maintaining the previous state or moving to the current label is accumulated; S4-3-4: If the current label is a new character, the probability of skipping the blank label is additionally accumulated. S4-4: Loss calculation and back propagation: S4-4-1: Summarize the sum of probabilities of all valid paths; S4-4-2: Take the negative logarithm of the sum of probabilities as the CTC loss; S4-4-3: Reversely update the acoustic model parameters through the gradient descent algorithm to optimize the alignment capability.

9. The recognition method of the speech recognition system based on the improved Transformer architecture according to claim 6, characterized in that: The step S5 comprises the following steps: S5-1: Perform feature extraction on each input audio channel independently and generate multiple sets of normalized feature sequences; S5-2: Select one audio channel as the reference channel and calculate the cross-correlation function between the target channel and the reference channel. That is, compare the similarity of the two feature sequences by sliding them over time. The time offset corresponding to the highest similarity point is the delay value. S5-3: Shift the target channel feature sequence forward or backward as a whole according to the detected delay value; S5-4: Calculate the comprehensive credibility score of each channel: Calculate the signal-to-noise ratio, which is the ratio of pure speech energy to noise energy; Calculate stability, which is the inverse of the variance of the characteristic sequence. The smaller the fluctuation, the higher the score; S5-5: Channels with a score ≥ 0.8 and a delay ≤ 200 milliseconds are retained, and out-of-tolerance or out-of-delay channels are discarded and an alarm is triggered to obtain a set of synchronization features that pass the screening.

10. The recognition method of a speech recognition system based on an improved Transformer architecture according to claim 6, characterized in that: The step S6 comprises the following steps: S6-1: Input the features of each trusted channel into the optimized acoustic model and output the character probability distribution sequence of the corresponding channel; S6-2: Calculate the similarity between the probability distribution of each channel and the current global distribution, and generate the fusion weight based on the channel credibility score; The probability distribution of all channels is weighted and summed according to the weights to update the global probability distribution; S6-3: Beam Search decoding: Keep the current five best text paths and their cumulative scores, add new characters or keep blank labels at the end of each path; new characters include: all recognizable characters and blank labels; S6-4: Calculate the cumulative total score and select the path with the highest cumulative score as the final recognized text. Acoustic score: the logarithmic probability of the character in the global probability distribution; The cumulative total is calculated as follows: Language model score: the probability of the character being continued given by the external language model; Cumulative total score = original path score + acoustic score + 30% × language model score.

Images