End-to-end speech recognition method and system based on keyword enhanced attention mechanism

Abstract

The application relates to the technical field of speech recognition, and provides an end-to-end speech recognition method and system based on a keyword-enhanced attention mechanism, which comprises the following steps: a keyword retriever extracts entity keywords; a speech buffer manager receives continuous air traffic control audio segments, converts the audio segments into audio mel spectrum graphs, and then converts the audio mel spectrum graphs into audio embedding sequences; a keyword encoder unit maps a word embedding representation sequence into a keyword embedding sequence; an audio transcription decoder unit performs keyword-enhanced attention calculation, converts an audio embedding sequence, a transcription start flag and a splicing vector of a word embedding representation sequence into a predicted text sequence; a new word embedding representation sequence is input into a text transcriptioner through self-recurrence until a transcription end flag is output by the audio transcription decoder unit; and the predicted text sequence is output as a speech recognition text. The application realizes real-time recognition of streaming input speech, and has the advantages of high recognition accuracy, low response delay and flexible deployment.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to an end-to-end speech recognition method and system based on a keyword-enhanced attention mechanism. Background Technology

[0002] Speech recognition technology is a key tool for ensuring flight safety and improving air traffic control efficiency, providing core technical support for automatic detection of command compliance, simulated training of controller capabilities, and comprehensive intelligent upgrades of air traffic control systems. However, air traffic control commands have unique characteristics such as dense technical terminology, strong context dependence, and high precision and second-level response requirements. These characteristics expose many insurmountable shortcomings in the practical application of existing speech recognition technologies, making it impossible to fully meet the accuracy and real-time standards of the air traffic control field.

[0003] In existing technologies, traditional streaming speech recognition models based on Long Short-Term Memory (LSTM) networks typically employ an architecture that trains acoustic and language models separately, matching acoustic features through statistical probabilities. However, their inherent limitations are amplified in air traffic control scenarios. First, their acoustic modeling capabilities are limited, exhibiting poor robustness when faced with background noise (such as radio static interference and wind noise), accent differences, or sudden changes in speech rate, resulting in a high misrecognition rate for key information (such as flight number, altitude, and heading). Second, the separate language models struggle to effectively model rare terms and dynamically changing contexts in air traffic control scenarios, failing to effectively integrate real-time flight schedules, weather, and other information into the recognition process, leading to a disconnect between the recognition results and the facts.

[0004] End-to-end speech recognition models based on the Transformer architecture, such as OpenAI's Whisper, have significantly improved the accuracy and robustness of general speech recognition. However, their "end-to-end" nature becomes a limitation in the specific domain of air traffic control. Specifically, these models primarily rely on audio input for information decoding and lack the ability to dynamically guide or constrain the decoding process using prompts. Therefore, they cannot inject external real-time database information as prior knowledge into the recognition process. When encountering ambiguous speech or similar instructions, the model cannot accurately distinguish based on context, resulting in deficiencies in flexibility and final accuracy, making them unsuitable for real-time, high-information-density air traffic control speech recognition tasks.

[0005] General-purpose large language models for speech modalities, with their massive number of parameters and training on huge amounts of data, have demonstrated excellent performance in general speech understanding tasks. However, these models were not originally designed for the ultra-low latency, streaming processing scenarios in air traffic control. Typically, these large language models require receiving complete speech segments before processing, and real-time speech transcription cannot meet the real-time requirements of air traffic control speech recognition. Furthermore, these models have high computational resource requirements and are mostly deployed in data centers, providing services externally via interfaces. This makes them difficult to deploy on resource-constrained edge devices for controller terminals, failing to meet the stringent requirements of air traffic control systems for economy, data security, and reliability.

[0006] Therefore, there is an urgent need to develop a new speech recognition solution that can integrate external prompting information, and is characterized by high accuracy, lightweight design, low latency, and support for streaming decoding, in order to fill the current technological gap. Summary of the Invention

[0007] This invention aims to address at least one of the technical problems existing in related technologies. To this end, this invention provides an end-to-end speech recognition method and system based on a keyword-enhanced attention mechanism, solving the technical problem that existing speech recognition models cannot simultaneously meet the comprehensive requirements of air traffic control speech recognition in terms of "high accuracy," "low latency," and "edge deployment." Addressing the high dependence on real-time flight data (including current route information, historical control instructions, etc.) during air traffic control instruction parsing, and the technical characteristics of needing to instantly complete sentence boundary detection and context association in continuous speech streams, this invention improves the accuracy of air traffic control speech recognition by dynamically introducing external keyword signals and introducing special tags such as speech timestamps and start / end markers into the word segmenter, thereby achieving real-time recognition processing of streaming input speech. This results in advantages such as high recognition accuracy, low response latency, and flexible deployment.

[0008] This invention provides an end-to-end speech recognition method based on a keyword-enhanced attention mechanism, comprising:

[0009] S1: Extract entity keywords that conform to air traffic control standardization terminology from the real-time aviation database through a keyword retrieval device, and parse the entity keywords into a word embedding representation sequence through a text embedding model;

[0010] S2: Receive continuous empty tube audio segments through the voice buffer manager, convert the empty tube audio segments into audio Mel spectrograms, and convert the audio Mel spectrograms into audio embedding sequences through the audio embedding model;

[0011] S3: The text transcriber fuses the features of the word embedding representation sequence and the audio embedding sequence to obtain the predicted text sequence; the predicted text sequence is fed back to the text embedding model through autoregression and concatenated with entity keywords to generate a new word embedding representation sequence.

[0012] Among them, the text transcriber is an end-to-end speech recognition architecture based on a transformer, and introduces a keyword-enhanced attention mechanism;

[0013] The text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The keyword encoder unit maps the word embedding representation sequence to a keyword embedding sequence. The audio transcriber decoder unit uses a hybrid attention mechanism to calculate the keyword enhancement attention between the keyword embedding sequence and the audio embedding sequence. Based on the keyword enhancement attention, the audio transcriber decoder unit converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the word embedding representation sequence into a predicted text sequence.

[0014] S4: Input the new lexical embedding representation sequence into the text transcriber, and repeat step S3 to convert the concatenated vector of the audio embedding sequence, the transcription start flag, and the new lexical embedding representation sequence into the predicted text sequence; during the continuous input of the audio stream, the audio transcription decoder unit outputs the predicted text sequence at the current moment in real time; when the audio transcription decoder unit outputs the transcription end flag, the predicted text sequence is output as the speech recognition text to the speech buffer manager.

[0015] Furthermore, the keyword retrieval device includes an input interface unit, a semantic filtering unit, and a result caching unit;

[0016] The input interface unit extracts current flight plan information from the real-time aviation database and reads the air traffic control standardized terminology set from the knowledge base;

[0017] The semantic filtering unit identifies and filters entity keywords that conform to air traffic control standardized terminology from the current flight plan information through a time validity filtering mechanism;

[0018] The result caching unit stores the entity keywords and updates the entity keywords.

[0019] Furthermore, the semantic filtering unit includes a flight number detection component, a waypoint detection component, and a term recognition component;

[0020] The flight number detection component identifies the flight call sign from the current flight plan information; the waypoint detection component matches valid waypoints with geographic point names maintained in the aviation navigation database; and the terminology recognition component identifies terms and passwords in the current flight plan information that conform to the air traffic control standardized terminology set.

[0021] The flight number detection component, the flight path detection component, and the terminology recognition component use predefined grammar templates and key pattern matching algorithms, combined with a time validity filtering mechanism, to perform multi-level filtering of words and obtain entity keywords that conform to air traffic control standardized terminology.

[0022] Furthermore, the result caching unit maintains a semantic priority index table and a time validity index table, and the result caching unit performs joint sorting based on priority and time validity to output entity keywords;

[0023] The semantic priority index table is used to identify the importance of entity keywords. The priority of entity keywords from high to low is built-in keywords, call signs, landmarks and locations, and instruction keywords. The time validity index table is used to identify the time validity of entity keywords.

[0024] Furthermore, the voice cache manager includes a circular buffer unit, a window control unit, and a time management unit;

[0025] The circular buffer unit has parallel read and write functions, receives continuous empty tube audio segments and stores them frame by frame according to the sampling rate, and generates real-time timestamps at the same time.

[0026] The window control unit extracts the empty tube audio segment from the loop buffer unit based on the window length, sliding step size, and overlap ratio; the window length, sliding step size, and overlap ratio of the window control unit are dynamically adjusted in real time based on the recognition feedback of the text transcriber.

[0027] The time management unit continuously tracks and records the absolute time offset of the empty tube audio segments in the audio stream.

[0028] Furthermore, the keyword encoder unit includes a six-layer self-attention layer network. Each self-attention layer network includes a cascaded position encoding and self-attention module and a fully connected layer module. The position encoding and self-attention module and the fully connected layer module all include residual connections and layer normalization.

[0029] Furthermore, the audio transcription decoder unit includes a bottom six-layer cross-attention layer structure and a top six-layer self-attention layer structure; each cross-attention layer includes a cascaded position encoding and self-attention module, a cross-attention module, and a fully connected layer, and each self-attention layer includes a cascaded position encoding and self-attention module and a fully connected layer module, and the position encoding and self-attention module and the fully connected layer module all include residual connections and layer normalization.

[0030] Furthermore, the audio embedding model is a Whisper-V3-Turbo model fine-tuned for air traffic control speech scenarios;

[0031] The entity keywords include flight call signs, waypoints, airport codes, and aviation terms.

[0032] Furthermore, while generating the predicted text sequence, the audio transcription decoder unit simultaneously outputs a local timestamp, which is aligned with the absolute time offset recorded by the time management unit.

[0033] The present invention also provides an end-to-end speech recognition system based on a keyword-enhanced attention mechanism to execute the above-mentioned end-to-end speech recognition method based on a keyword-enhanced attention mechanism, including a keyword retrieval unit, a speech cache manager, and a text transcriber;

[0034] Among them, the keyword retrieval device extracts entity keywords that conform to air traffic control standardization terminology from the real-time aviation database, and parses the entity keywords into a word embedding representation sequence through the text embedding model;

[0035] The voice buffer manager receives continuous empty tube audio segments and converts them into audio Mel spectrograms. Then, it uses an audio embedding model to convert the audio Mel spectrograms into audio embedding sequences.

[0036] The text transcriber is an end-to-end speech recognition architecture based on a transformer, and introduces a keyword-enhanced attention mechanism;

[0037] The text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The keyword encoder unit maps the word embedding representation sequence to a keyword embedding sequence. The audio transcriber decoder unit uses a hybrid attention mechanism to calculate keyword-enhanced attention between the keyword embedding sequence and the audio embedding sequence. Based on the keyword-enhanced attention, the audio transcriber decoder unit converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the word embedding representation sequence into a predicted text sequence. The predicted text sequence is fed back to the text embedding model via autoregression and concatenated with entity keywords to generate a new word embedding representation sequence.

[0038] A new lexical embedding representation sequence is input to the text transcriber. The text transcriber converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the new lexical embedding representation sequence into a predicted text sequence. During the continuous input of the audio stream, the audio transcription decoder unit outputs the predicted text sequence at the current moment in real time. When the audio transcription decoder unit outputs the transcription end flag, it outputs the predicted text sequence as the speech recognition text to the speech buffer manager.

[0039] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0040] This invention introduces a keyword enhancement mechanism to achieve focused modeling of technical terms and high-confidence entities during speech recognition. By introducing lexical embedding representation sequences in the encoding stage of the text transcription network and using a cross-attention module to establish the association between entity keywords and audio features, the model's recognition ability for air traffic control terminology such as flight call signs, route names, and airport codes is enhanced. Compared to traditional speech recognition systems that rely on pure acoustic or language models for inference, this invention can significantly improve recognition robustness and accuracy in environments with high noise, overlapping pronunciations, and complex accents. It enables real-time recognition and processing of streaming input speech and has advantages such as high recognition accuracy, low response latency, and flexible deployment.

[0041] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating an end-to-end speech recognition method based on a keyword-enhanced attention mechanism provided by the present invention.

[0044] Figure 2 This is a schematic diagram of the text transcriber architecture provided by the present invention.

[0045] Figure 3 This is a schematic diagram of the overall architecture of an end-to-end speech recognition system based on a keyword-enhanced attention mechanism provided by the present invention.

[0046] Figure 4 This is a schematic diagram illustrating the recognition accuracy of the present invention on the test set of dataset A. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0048] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0049] The following is combined with Figures 1 to 4 This invention describes an end-to-end speech recognition method and system based on a keyword-enhanced attention mechanism.

[0050] like Figure 1 As shown, an end-to-end speech recognition method based on a keyword-enhanced attention mechanism includes:

[0051] S1: Extract entity keywords that conform to air traffic control standardization terminology from the real-time aviation database through a keyword retrieval device, and parse the entity keywords into a word embedding representation sequence through a text embedding model;

[0052] The keyword retrieval unit is a key component responsible for inputting semantic auxiliary information. Its function is to extract entity keywords closely related to the current speech recognition task from external real-time aviation databases and knowledge bases, based on the real-time task scenario, time window, and air traffic control communication environment. Entity keywords include flight call signs, waypoints, airport codes, and aviation terminology. By providing a context-related auxiliary vocabulary set, the keyword retrieval unit effectively enhances the recognition accuracy of subsequent text transcription and the accuracy of air traffic control terminology. The keyword retrieval unit incorporates a time validity filtering mechanism to ensure that the output keywords possess actual air traffic control semantic meaning within the current time window.

[0053] The keyword search engine includes an input interface unit, a semantic filtering unit, and a result caching unit;

[0054] The input interface unit extracts current flight plan information from the real-time aviation database and reads the air traffic control standardized terminology set from the knowledge base;

[0055] The input interface unit is primarily responsible for interacting with external database services, including accessing and synchronizing real-time databases and knowledge bases. The real-time database provides dynamic data such as current flight scheduling and plan execution status; the knowledge base contains standard aviation navigation terminology, fixed waypoints, airport names, and common air traffic control command phrases, and is preprocessed according to a unified format to ensure data consistency in subsequent processing.

[0056] The semantic filtering unit identifies and filters entity keywords that conform to air traffic control standardized terminology from the current flight plan information through a time validity filtering mechanism;

[0057] The semantic filtering unit includes a flight number detection component, a waypoint detection component, and a terminology recognition component;

[0058] The flight call sign is identified from the current flight plan information by the flight number detection component; valid waypoints are matched from the geographic point names maintained in the air navigation database by the waypoint detection component; and terms and passwords that conform to the air traffic control standardized terminology set in the current flight plan information are identified by the terminology recognition component.

[0059] The flight number detection component, the waypoint detection component, and the terminology recognition component use predefined grammar templates and key pattern matching algorithms, combined with a time validity screening mechanism, to perform multi-level screening of words and obtain entity keywords that conform to air traffic control standardized terminology.

[0060] The semantic filtering unit is used to identify and extract keywords that conform to air traffic control semantic features from the original text. The time validity filtering mechanism uses the system's current timestamp and the valid time window in the flight plan to determine whether the keywords still have air traffic control business significance in the current time period, thereby achieving dynamic filtering.

[0061] The result cache unit stores entity keywords and updates entity keywords.

[0062] The result cache unit is used to store and update the filtered entity keyword result set. The result cache unit maintains a semantic priority index table and a time validity index table, and outputs entity keywords based on a joint sorting of priority and time validity.

[0063] The semantic priority index table is used to identify the importance of entity keywords. The priority of entity keywords from high to low is: built-in keywords, call signs, landmarks and locations, and instruction keywords. The time validity index table is used to identify the time validity of entity keywords.

[0064] In the final output stage, the result caching unit will sort the results based on priority and timeliness, select no more than 256 keywords, generate a keyword output sequence, and provide it to the text transcription module.

[0065] The keyword search engine's workflow begins with the system triggering a real-time data update request. When the system enters a new speech recognition window, the input interface unit retrieves current flight schedule information from the real-time database, including departure and arrival airports, flight routes, and scheduling tables related to the current time. Simultaneously, it reads a set of standardized terms from the knowledge base, such as frequently used control commands and fixed waypoints in air traffic control communications.

[0066] After the raw text data enters the semantic filtering unit, the flight number detection component first identifies the flight call sign. Then, the waypoint detection component matches valid waypoints based on the geographic point names maintained in the aviation navigation database; the terminology recognition component identifies commonly used air traffic control terms and passwords.

[0067] Subsequently, the time validity filtering mechanism compares candidate keywords with the system time and flight status table, eliminating flight numbers not in the current active flight segment or expired flight paths. The filtered results are sent to the result cache unit for indexing and sorting, retaining no more than 256 keywords as the final output while ensuring semantic coverage. This output sequence is then passed to the text transcriber to improve keyword attention within specific time periods and operational contexts.

[0068] S2: Receive continuous empty tube audio segments through the voice buffer manager, convert the empty tube audio segments into audio Mel spectrograms, and convert the audio Mel spectrograms into audio embedding sequences through the audio embedding model;

[0069] The speech buffer manager is a key component in a fast ATC (Modern Air Traffic Control) speech recognition system, responsible for connecting the front-end audio input with the back-end text transcription process. Its core function is to buffer, segment, and manage the timing of the input speech signal, ensuring that the speech stream is delivered to the recognition engine in a real-time, smooth, and accurate manner during continuous recognition. Through a dynamic windowing mechanism, timestamp alignment strategy, and sliding control linked to the end marker, the speech buffer manager effectively improves the real-time performance and accuracy of the system's recognition, forming a crucial foundation for achieving continuous speech recognition and sentence-level segmentation.

[0070] The design philosophy of the voice buffer manager is to establish a voice data buffer system that can continuously receive, dynamically process, and precisely align voice data to ensure that continuous voice input can be structurally managed during the recognition process.

[0071] The voice buffer manager includes a circular buffer unit, a window control unit, and a time management unit; the units collaborate efficiently through an ordered data flow interface.

[0072] The circular buffer unit receives continuous empty tube audio segments and stores them frame by frame according to the sampling rate, while generating real-time timestamps;

[0073] The circular buffer unit receives continuous audio signals from input devices, such as voice communication systems or microphones, and stores the data in an orderly manner. The circular buffer unit features parallel read / write capabilities, enabling it to provide fast read services for subsequent recognition tasks while data is continuously being written, thus avoiding recognition delays caused by data congestion.

[0074] The window control unit extracts the empty tube audio segment from the circular buffer unit based on the window length, sliding step size, and overlap ratio; the window length, sliding step size, and overlap ratio of the window control unit are dynamically adjusted in real time based on the recognition feedback of the text transcriber.

[0075] The window control unit is responsible for "windowing" the audio signals in the buffer, that is, extracting audio segments from the buffer according to a certain time length and overlap ratio. The length of the window and the sliding step size can be dynamically adjusted in real time based on recognition feedback, enabling the entire system to balance high response speed with the contextual integrity required for accurate recognition. The window control unit works in conjunction with the text transcriber to achieve streaming recognition by continuously moving the recognition window.

[0076] The time management unit continuously tracks and records the absolute time offset of the empty tube audio segments in the audio stream.

[0077] Absolute time offset is crucial in speech recognition result annotation, speech event localization, and subsequent data playback. It ensures precise time alignment between the recognition output and the audio input, providing the system with a traceable time reference.

[0078] During system operation, the voice buffer manager first receives continuous audio input and stores it frame by frame into the circular buffer unit according to the sampling rate, while simultaneously generating and attaching a corresponding real-time timestamp. As the buffered data accumulates, an audio segment of a specified length is extracted according to the window control logic and sent to the text transcriber for speech recognition.

[0079] S3: The text transcriber fuses the features of the word embedding representation sequence and the audio embedding sequence to obtain the predicted text sequence; the predicted text sequence is fed back to the text embedding model through autoregression and concatenated with entity keywords to generate a new word embedding representation sequence.

[0080] Among them, the text transcriber is an end-to-end speech recognition architecture based on a transformer, and a keyword-enhanced attention mechanism is introduced;

[0081] The text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The keyword encoder unit maps the word embedding representation sequence to a keyword embedding sequence. The audio transcriber decoder unit uses a hybrid attention mechanism to calculate the keyword enhancement attention between the keyword embedding sequence and the audio embedding sequence. Based on the keyword enhancement attention, the audio transcriber decoder unit converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the word embedding representation sequence into a predicted text sequence.

[0082] The text transcription generator outputs a sequence of results, including the recognized text, local timestamps, and a transcription end marker. If the transcription end marker is not yet present in the output, it indicates that the sentence is not yet fully completed. The speech buffer manager then continuously slides the window, gradually adding new input audio to achieve real-time streaming recognition.

[0083] When a transcription end marker appears in the transcription result, the starting position of the window is dynamically adjusted based on the timestamp corresponding to the transcription end marker, so that the next recognition begins from the audio position after the end point of that sentence. At the same time, the system updates the offset to ensure that the recognition window is synchronized with the global time. This loop repeats, enabling the system to accurately extract semantically complete sentence-level speech segments from the real-time access speech stream and maintain a smooth speech recognition process.

[0084] The text transcription engine is the core recognition engine of the air traffic control speech recognition system, responsible for accurately converting continuous audio segments dynamically extracted by the speech buffer manager into text sequences. In addition to outputting the recognized text, it simultaneously provides relative timestamps and sentence end markers to accurately identify speech sentence boundaries and assist in the automatic control of the speech buffer manager's window sliding. The text transcription engine adopts an end-to-end speech recognition architecture based on a transformer, upon which a keyword-enhanced attention mechanism is introduced. The entity keywords processed by this mechanism are dynamically extracted from the database by the system. The model can significantly optimize the accuracy of speech recognition in the aviation field based on these dynamically loaded keywords, achieving high-confidence recognition of speech content and preservation of semantic continuity.

[0085] The text transcriber employs a hierarchical and functional decoupling strategy to cope with high-load, low-latency air traffic control voice stream scenarios.

[0086] like Figure 2 As shown, the text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The overall hidden layer dimension of the model is set to 1280 dimensions, and the hidden layer dimension of the fully connected network is expanded to 5120 dimensions.

[0087] The keyword encoder unit is responsible for mapping the token embedding representation sequence to a high-dimensional text vector sequence. The input entity keywords are first parsed into a discrete token sequence T, and then transformed into a continuous token embedding representation sequence TE by the text embedding model. TE enters the keyword encoder unit, which consists of a six-layer self-attention network. Each self-attention layer includes a cascaded positional encoding and self-attention module, as well as a fully connected layer module. Both the positional encoding and self-attention module and the fully connected layer module contain residual connections and layer normalization, calculated as follows:

[0088]

[0089] in, For output features, For layer normalization, As input features, The output features or positional encodings of the attention layer module are compared with the output features of the self-attention module. This is a residual connection.

[0090] Residual connections are used to effectively prevent gradient vanishing due to network deepening. In the positional encoding and self-attention modules, the calculation expression for the attention mechanism is:

[0091]

[0092] in, For the output features of the attention mechanism, for Activation function For querying the matrix, The key matrix, For value matrices, The scaling dimension of the feature. for The transpose of .

[0093] The association weights within a feature are calculated using an attention mechanism, where , , All are derived from the input features through linear transformation.

[0094] A fully connected layer consists of two linear transformations and activation functions, and its calculation expression is as follows:

[0095]

[0096] in, For the output of the fully connected layer, It is a non-linear activation function. This is the weight matrix of the first linear layer. For the first linear layer bias, This is the weight matrix of the second linear layer. For the second linear layer bias;

[0097] After six layers of stacked encoding, the unit outputs a keyword embedding sequence containing semantic enhancement information, which serves as auxiliary information for subsequent decoding.

[0098] The audio transcription decoder unit includes a bottom six-layer cross-attention layer structure and a top six-layer self-attention layer structure. Each cross-attention layer includes a cascaded positional encoding and self-attention module, a cross-attention module, and a fully connected layer. Each self-attention layer includes a cascaded positional encoding and self-attention module and a fully connected layer module. The positional encoding and self-attention module and the fully connected layer module all include residual connections and layer normalization.

[0099] The audio transcription decoder unit receives dual input sources: audio and text. At the audio input, the raw audio is first converted into an audio Mel-spectrum, which is then fed into the audio embedding model. This audio segment embedding model originates from the encoder of the "Whisper-V3-Turbo model," fine-tuned for air traffic control scenarios. During the training phase of the text transcriber, parameters are frozen to ensure feature extraction stability, thus outputting a discrete audio embedding sequence AE. The audio transcription decoder unit consists of a bottom six-layer cross-attention layer structure and a top six-layer self-attention layer structure; the input sequence is formed by concatenating the audio embedding sequence, the transcription start marker SOT, and a new word embedding representation sequence generated through autoregression into a channel-dimensional vector.

[0100] In the bottom cross-attention layer, each layer sequentially contains a positional encoding and self-attention module, a cross-attention module, and a fully connected layer. The cross-attention module is the core of keyword enhancement recognition. Its calculation is also based on the attention mechanism, but the query matrix Q comes from the previous output of the current layer of the decoder, while the key matrix K and value matrix V come from the keyword extraction sequence representing dynamic knowledge in the database output by the keyword encoder unit on the right. The role of this module is to enhance the model's decision selectivity in key term recognition by calculating the interaction between acoustic features introduced by the environment and external text features. After crossing the cross-attention layer, the feature vector is uploaded to the top six self-attention layers, eliminating the cross-attention mechanism, and used for further deep modeling of the internal dependencies of the context. All attention modules in the decoder adopt a grouped query attention mechanism, with the number of query heads uniformly defined as 16 groups, and the number of key and value heads reduced to 8 groups to improve memory efficiency. The final output of the model is the predicted text sequence, the end-of-transmission marker EOT, and the relative timestamp TS.

[0101] The overall workflow of the text transcription system follows a temporal logic from dual-stream feature extraction to cross-modal fusion, and then to autoregressive generation. This process begins with the interaction between the raw, continuous audio segments input from the speech buffer manager and an external database. On one hand, the input audio is processed in the front end and accurately converted into an audio Mel-spectrum reflecting the frequency distribution, which is then fed into the audio embedding model for feature extraction, forming an AE sequence with acoustic representation capabilities. On the other hand, the system extracts entity keywords related to current flight dynamics and air traffic control context from the database in real time. These non-fixed words are converted into basic tokens and input into the text embedding model, where a six-layer self-attention network on the right encodes a structured high-dimensional TE sequence.

[0102] During the initial decoding phase, the system inputs the AE sequence and SOT flag into the audio transcription decoder. The feature sequence flows from bottom to top through a six-layer cross-attention network, where the acoustic context features in the decoder and the keywords encoded by the encoder undergo cross-attention calculation. This cross-modal matching process allows the model to focus on ambiguous or easily confused technical terms in the early stages of decoding, generating reinforced representations using externally injected prior knowledge and avoiding the generation of non-existent content such as flight call signs. Subsequently, this reinforced representation continues to flow upward through a six-layer self-attention network, completing the semantic coherence reconstruction in complex contexts through global calculation of the logical relationships between preceding and following sequences.

[0103] After the decoder completes its computation, the fully connected layer at the top of the decoder is responsible for outputting the specific character predicted in the current step. This is illustrated by the dashed path from top to bottom in the diagram.

[0104] The system employs a rigorous autoregressive generation model. The predicted text sequence generated in the current step is not only recorded as a stage of transcription results, but is also synchronously fed back to the bottom text embedding model along the autoregressive path. After being transformed into a new TE, it is added to the decoder's input sequence queue, serving as known conditions for predicting the next audio segment in the next iteration of the calculation.

[0105] S4: Input the new lexical embedding representation sequence into the text transcriber, and repeat step S3 to convert the concatenated vector of the audio embedding sequence, the transcription start flag, and the new lexical embedding representation sequence into the predicted text sequence; during the continuous input of the audio stream, the audio transcription decoder unit outputs the predicted text sequence at the current moment in real time; when the audio transcription decoder unit outputs the transcription end flag, the predicted text sequence is output as the speech recognition text to the speech buffer manager.

[0106] During the continuous iteration of forward computation, when the neural network confirms that the semantic expression corresponding to the current audio segment has completely ended, the generator outputs a specific End-of-Translation (EOT) marker and a timestamp (TS) at the end of the sequence. At this point, the text transcriber determines that the current transcription loop is closed and returns the complete recognized sequence, including the relative time stamp, to the speech buffer manager module. Upon receiving the EOT loop closure marker, the speech buffer manager automatically moves the starting point of the recognition window forward to the actual end position of the sentence and accurately updates the time offset. Thanks to the efficient transmission of the internal hidden state and the incremental attention computation mechanism, the model can respond instantly to audio buffer updates even in a continuous stream of unattended command speech, achieving uninterrupted recognition computation and completely avoiding semantic loss and accuracy degradation caused by manual truncation.

[0107] The audio embedding model is a Whisper-V3-Turbo model that has been fine-tuned for air traffic control speech scenarios;

[0108] While generating the predicted text sequence, the audio transcription decoder unit simultaneously outputs a local timestamp, which is aligned with the absolute time offset recorded by the time management unit.

[0109] like Figure 3 As shown, an end-to-end speech recognition system based on a keyword-enhanced attention mechanism is used to execute the aforementioned end-to-end speech recognition method based on a keyword-enhanced attention mechanism, including a keyword retrieval unit, a speech cache manager, and a text transcriber;

[0110] Among them, the keyword retrieval device extracts entity keywords that conform to air traffic control standardization terminology from the real-time aviation database, and parses the entity keywords into a word embedding representation sequence through the text embedding model;

[0111] The voice buffer manager receives continuous empty tube audio segments and converts them into audio Mel spectrograms. Then, it uses an audio embedding model to convert the audio Mel spectrograms into audio embedding sequences.

[0112] The text transcriber is an end-to-end speech recognition architecture based on a transformer, and introduces a keyword-enhanced attention mechanism;

[0113] The text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The keyword encoder unit maps the word embedding representation sequence to a keyword embedding sequence. The audio transcriber decoder unit uses a hybrid attention mechanism to calculate keyword-enhanced attention between the keyword embedding sequence and the audio embedding sequence. Based on the keyword-enhanced attention, the audio transcriber decoder unit converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the word embedding representation sequence into a predicted text sequence. The predicted text sequence is fed back to the text embedding model via autoregression and concatenated with entity keywords to generate a new word embedding representation sequence.

[0114] A new lexical embedding representation sequence is input to the text transcriber. The text transcriber converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the new lexical embedding representation sequence into a predicted text sequence. During the continuous input of the audio stream, the audio transcription decoder unit outputs the predicted text sequence at the current moment in real time. When the audio transcription decoder unit outputs the transcription end flag, it outputs the predicted text sequence as the speech recognition text to the speech buffer manager.

[0115] This invention introduces a keyword enhancement mechanism to achieve focused modeling of technical terms and high-confidence entities during speech recognition. By introducing lexical embedding representation sequences in the encoding stage of the text transcription network and using a cross-attention module to establish the association between entity keywords and audio features, the model's recognition ability for air traffic control terminology such as flight call signs, route names, and airport codes is enhanced. Compared to traditional speech recognition systems that rely on pure acoustic or language models for inference, this invention can significantly improve recognition robustness and accuracy in environments with high noise, overlapping pronunciations, and complex accents. It enables real-time recognition and processing of streaming input speech and has advantages such as high recognition accuracy, low response latency, and flexible deployment.

[0116] The text-to-text neural network training process aims to achieve both high-precision recognition and strong robustness in air traffic control speech recognition scenarios through a multi-stage training mechanism. The entire training process includes three stages: a pre-training stage, a keyword recognition-enhanced fine-tuning stage, and an air traffic control-specific fine-tuning stage. The audio embedding model remains frozen throughout the training process. Each stage maintains a unified model framework, differing only in the construction of the keyword-weighted loss function, dataset composition, and parameter freezing strategy. The implementation process and key technical details of each stage are described below.

[0117] The goal of the pre-training phase is to establish the model's basic speech-to-text mapping capability and to initially cultivate the model's keyword perception ability. In this phase, the model uses multi-source training corpora consisting of multiple open-source speech datasets and synthetic speech datasets (text generated by a large language model and then converted into speech by a text-to-speech service). The composition of the multi-source speech datasets is shown in Table 1.

[0118] Table 1 Composition of the multi-source speech dataset

[0119]

[0120] In each training batch, the labeled text is processed by a vocabulary-based noun extraction algorithm to identify a set of nouns that form the keyword set for the current batch. Simultaneously, several task-irrelevant nouns are randomly extracted from text within the same domain as noise keywords to improve the model's ability to distinguish non-critical semantic words. This keyword set is then fed into the keyword encoder unit, and the encoded result is used by the audio transcription decoder unit for cross-attention calculation.

[0121] Introducing keyword weights during training This stage uses a fixed keyword weighting ratio. Furthermore, a scheduler is used for dynamic and smooth decay, ensuring that keyword weights dominate in the early stages of training and gradually revert to ordinary weights in the later stages, thus achieving a balance in keyword perception. The loss function is calculated as follows:

[0122]

[0123] in, For loss, For the actual transcribed text in the first Tokens for each location Represents a sequence of speech features. For the model in the first The predicted probability of each position. To output the sequence length, For the first The weight coefficient of each position, Determined by the following formula:

[0124]

[0125] in, This is a keyword indicator function; its value is 1 when the output word is a keyword, and 0 otherwise. For predefined keywords, this mechanism allows the model to receive a larger gradient penalty when it identifies incorrect keywords, thereby enhancing its keyword sensitivity.

[0126] The datasets used for training in the keyword recognition enhancement and fine-tuning stage are high-quality speech datasets, including WenetSpeech, AliMeeting, and AISHELL-1. The keyword recognition enhancement and fine-tuning stage aims to further enhance the model's ability to capture target keywords and its semantic discrimination capabilities. This stage employs a soft-weighted loss mechanism, and the keyword sources are the same as in the pre-training stage, consisting of nouns extracted from the actual transcribed text, with a certain proportion of noisy nouns added.

[0127] Unlike the pre-training phase, the soft weighting mechanism does not rely entirely on the presence or absence of keywords, but rather dynamically adjusts the weighting based on the model's predicted probabilities. Specifically, the keyword weighting multiplier is set to... 5, It is determined by the following formula:

[0128]

[0129] in, for The probability of.

[0130] This weighting method allows the model to generate a larger gradient for predicting keywords with low confidence, thereby significantly improving the model's ability to focus on keywords while maintaining overall recognition stability.

[0131] The goal of the air traffic control domain-specific fine-tuning phase is to enable the model to possess a highly stable ability to recognize proprietary command formats, flight call signs, waypoint names, and place names in air traffic control communications. In this phase, all parameters of the keyword encoder unit and the text embedding model are frozen, and only the audio transcription decoder unit's parameters are updated.

[0132] Training dataset A consists of real air traffic control communication data, totaling 55,000 speech samples, including 35,000 Chinese air traffic control dialogues and 20,000 English air traffic control dialogues. Each speech sample is accompanied by a manually transcribed text. Regular expressions are used to extract flight call signs, waypoints, and control instructions from the tags to form a keyword set. Irrelevant flight numbers and instructions are sampled from the same corpus as noise keywords to maintain a balanced distribution of keyword categories in the data. In this stage, the standard cross-entropy loss function is used.

[0133] The accuracy criteria for recognition results include word accuracy, call sign entity accuracy, location entity accuracy, instruction entity accuracy, and numeral entity accuracy.

[0134] Character accuracy measures the system's recognition precision at the single-character level, reflecting the error rate of the model at the basic speech recognition level. Assume the test set contains... The number of characters correctly recognized in the system output is [number]. Then the accuracy rate of the characters Defined as:

[0135]

[0136] This metric is calculated uniformly after each training stage is completed, and is used to evaluate the overall transcription accuracy, acoustic modeling ability, and language modeling consistency of the model.

[0137] Call sign entity accuracy measures the system's success rate in recognizing aircraft call signs, a crucial piece of information. Call sign entities are the core target of semantic recognition in air traffic control communications, and errors in their recognition directly impact mission accuracy. Let the test set contain a total of [number missing] call sign entities. The number of correctly identified call sign entities in the system output is The system only determines a correct recognition if the call sign matches exactly in the transcribed text (including identical letter, number, and structural order). Call sign entity accuracy. Defined as:

[0138]

[0139] This metric is used to directly measure the model's stability and fault tolerance in critical aviation identification tasks.

[0140] Location entity accuracy measures the system's accuracy in identifying geographical names such as waypoints, airports, and controlled areas. Let the total number of location entities in the test set be... The number of location entities correctly identified by the system is The system determines a location to be correct if the output string matches the reference place name semantically or symbolically. Location entity accuracy. Defined as:

[0141]

[0142] This metric is used to reflect the model's performance in entity type discrimination and geographic proper name recognition scenarios.

[0143] Command entity accuracy is used to evaluate the system's performance in recognizing air traffic control commands. Command entities include standard operational phrases such as "climb to," "turn left / right," or "cleared for takeoff," which are the crucial information component in air traffic control speech recognition. Let the total number of command entities in the test set be... The number of instruction entities correctly identified by the system is Then the accuracy of the instruction entity Defined as:

[0144]

[0145] This indicator is used to quantitatively analyze the improvement of the model's identification during the domain-specific fine-tuning stage.

[0146] Numerical entity accuracy measures the system's precision in recognizing numerical semantics such as altitude, speed, and heading. Let the test set contain a total of [number] numeral entities. The number of correctly identified numeral entities in the system output is Then the accuracy of numeral entities Defined as:

[0147]

[0148] This metric is used to evaluate the model's recognition stability and parsing ability when processing numerical information in speech.

[0149] This invention introduces a keyword enhancement mechanism to achieve focused modeling of specialized terms and high-confidence entities during speech recognition. This mechanism introduces keyword embedding vectors during the encoding stage of the text transcription network and utilizes a cross-attention module to establish the association between keywords and audio features, thereby enhancing the model's recognition ability for air traffic control terminology such as flight call signs, route names, and airport codes. Compared to traditional speech recognition systems that rely on pure acoustic or language models for inference, this invention significantly improves robustness and accuracy in environments with high noise levels, overlapping pronunciations, and complex accents. In an experimental environment with an i7-12700K CPU and a 3060Ti 8GB GPU, the recognition accuracy is compared to that of whisper-v3-turbo (whisper for short), whisper-v3-turbo fine-tuned using dataset A (whisper+fine-tuned), this invention (without keyword guidance), and this invention (with keyword guidance). The diagram illustrates the comparison of recognition accuracy. Figure 4 As shown.

[0150] Figure 4 In this study, the present invention (with keyword guidance) achieved the highest accuracy rates for word accuracy, call sign entity accuracy, location entity accuracy, instruction entity accuracy, and numeral entity accuracy. The present invention (without keyword guidance) also significantly outperformed Whisper and Whisper+ fine-tuning. Comparing the present invention (with keyword guidance) and the present invention (without keyword guidance), the present invention (with keyword guidance) showed further improvements in all indicators, especially in call sign entity accuracy and location entity accuracy recognition.

[0151] Whisper performed the worst in call sign entity recognition. Although there was some improvement after fine-tuning, it was still far inferior to the present invention (with keyword guidance) and the present invention (without keyword guidance). This indicates that the proposed solution has a stronger optimization effect on the recognition of key semantic entities.

[0152] The text transcription neural network of this invention has an overall parameter count of approximately 1 billion (1B). Under a context of 1024 tokens and single-precision weight loading, the model itself and its key-value cache occupy only 3.5GB. This design significantly reduces computational complexity while ensuring sufficient expressive power, enabling the model to have a faster response speed and lower computational resource consumption during the inference phase compared to traditional large-scale models. As a result, the model described in this invention can not only run efficiently on high-performance servers but also achieve real-time recognition on edge computing devices, thereby meeting the rapid speech processing needs of edge nodes such as airport control towers or air traffic control centers.

[0153] This invention supports inference using two domestic deep learning frameworks, MindSpore and PaddlePaddle, and provides full-process support for model training, inference, and optimization within the MindSpore framework. It also supports the open-source computing framework GGML, ensuring compatibility with mainstream domestic computing cards such as Moore's Threads and Ascend, fully utilizing domestic computing resources, meeting information security and self-control requirements, and providing reliable technical support for aviation speech recognition applications in a domestically developed environment.

[0154] Compared to other models using the Transformer architecture, this invention supports a complete streaming speech recognition process. The speech cache manager employs a sliding window data processing mechanism, enabling real-time interception and dynamic recognition of continuous speech input. Combined with the autoregressive generation strategy of the text transcriber, in a test environment with four batches of concurrent input, the average recognition latency is only about 800ms, sufficient to ensure that the flow of control instructions is uninterrupted and does not back up.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end speech recognition method based on a keyword-enhanced attention mechanism, characterized in that, include: S1: Extract entity keywords that conform to air traffic control standardized terminology from the real-time aviation database through a keyword retrieval device, and parse the entity keywords into a word embedding representation sequence through a text embedding model. The entity keywords include flight call signs, waypoints, airport codes, and aviation terms. S2: Receive continuous empty tube audio segments through the voice buffer manager, convert the empty tube audio segments into audio Mel spectrograms, and convert the audio Mel spectrograms into audio embedding sequences through the audio embedding model; S3: The text transcriber fuses the features of the word embedding representation sequence and the audio embedding sequence to obtain the predicted text sequence; the predicted text sequence is fed back to the text embedding model through autoregression and concatenated with entity keywords to generate a new word embedding representation sequence. Among them, the text transcriber is an end-to-end speech recognition architecture based on a transformer, and introduces a keyword-enhanced attention mechanism; The text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The keyword encoder unit maps the word embedding representation sequence to a keyword embedding sequence. The hybrid attention mechanism of the audio transcription decoder unit calculates the keyword enhancement attention between the keyword embedding sequence and the audio embedding sequence. Based on the keyword enhancement attention, the audio transcription decoder unit converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the word embedding representation sequence into the predicted text sequence. The keyword encoder unit includes a six-layer self-attention layer network. Each self-attention layer network includes a cascaded position encoding and self-attention module and a fully connected layer module. The position encoding and self-attention module and the fully connected layer module all include residual connections and layer normalization. The audio transcription decoder unit includes a bottom six-layer cross-attention layer structure and a top six-layer self-attention layer structure; each cross-attention layer includes a cascaded positional encoding and self-attention module, a cross-attention module and a fully connected layer, and each self-attention layer includes a cascaded positional encoding and self-attention module and a fully connected layer module. The positional encoding and self-attention module and the fully connected layer module all include residual connections and layer normalization. S4: Input the new lexical embedding representation sequence into the text transcriber, and repeat step S3 to convert the concatenated vector of the audio embedding sequence, the transcription start flag, and the new lexical embedding representation sequence into the predicted text sequence; during the continuous input of the audio stream, the audio transcription decoder unit outputs the predicted text sequence at the current moment in real time; when the audio transcription decoder unit outputs the transcription end flag, the predicted text sequence is output as the speech recognition text to the speech buffer manager.

2. The end-to-end speech recognition method based on keyword-enhanced attention mechanism according to claim 1, characterized in that, The keyword search engine includes an input interface unit, a semantic filtering unit, and a result caching unit; The input interface unit extracts current flight plan information from the real-time aviation database and reads the air traffic control standardized terminology set from the knowledge base; The semantic filtering unit identifies and filters entity keywords that conform to air traffic control standardized terminology from the current flight plan information through a time validity filtering mechanism; The result caching unit stores the entity keywords and updates the entity keywords.

3. The end-to-end speech recognition method based on keyword-enhanced attention mechanism according to claim 2, characterized in that, The semantic filtering unit includes a flight number detection component, a waypoint detection component, and a terminology recognition component; The flight number detection component identifies the flight call sign from the current flight plan information; the waypoint detection component matches valid waypoints with geographic point names maintained in the aviation navigation database; and the terminology recognition component identifies terms and passwords in the current flight plan information that conform to the air traffic control standardized terminology set. The flight number detection component, the flight path detection component, and the terminology recognition component use predefined grammar templates and key pattern matching algorithms, combined with a time validity filtering mechanism, to perform multi-level filtering of words and obtain entity keywords that conform to air traffic control standardized terminology.

4. The end-to-end speech recognition method based on keyword-enhanced attention mechanism according to claim 2, characterized in that, The result caching unit maintains a semantic priority index table and a time validity index table. The result caching unit outputs entity keywords by jointly sorting them according to priority and time validity. The semantic priority index table is used to identify the importance of entity keywords. The priority of entity keywords from high to low is as follows: built-in keywords, call signs, landmarks and locations, and instruction keywords. The time validity index table is used to identify the time validity of entity keywords.

5. The end-to-end speech recognition method based on keyword-enhanced attention mechanism according to claim 1, characterized in that, The voice cache manager includes a circular buffer unit, a window control unit, and a time management unit; The circular buffer unit has parallel read and write functions, receives continuous empty tube audio segments and stores them frame by frame according to the sampling rate, and generates real-time timestamps at the same time. The window control unit extracts the empty tube audio segment from the loop buffer unit based on the window length, sliding step size, and overlap ratio; the window length, sliding step size, and overlap ratio of the window control unit are dynamically adjusted in real time based on the recognition feedback of the text transcriber. The time management unit continuously tracks and records the absolute time offset of the empty tube audio segments in the audio stream.

6. The end-to-end speech recognition method based on keyword-enhanced attention mechanism according to claim 1, characterized in that, The audio embedding model is a Whisper-V3-Turbo model that has been fine-tuned for air traffic control speech scenarios.

7. The end-to-end speech recognition method based on keyword-enhanced attention mechanism according to claim 5, characterized in that, While generating the predicted text sequence, the audio transcription decoder unit simultaneously outputs a local timestamp, which is aligned with the absolute time offset recorded by the time management unit.

8. An end-to-end speech recognition system based on a keyword-enhanced attention mechanism, characterized in that, An end-to-end speech recognition method based on a keyword-enhanced attention mechanism as described in any one of claims 1 to 7, comprising a keyword retrieval unit, a speech cache manager, and a text transcriber; Among them, the keyword retrieval device extracts entity keywords that conform to air traffic control standardization terminology from the real-time aviation database, and parses the entity keywords into a word embedding representation sequence through the text embedding model; The voice buffer manager receives continuous empty tube audio segments and converts them into audio Mel spectrograms. Then, it uses an audio embedding model to convert the audio Mel spectrograms into audio embedding sequences. The text transcriber is an end-to-end speech recognition architecture based on a transformer, and introduces a keyword-enhanced attention mechanism; The text transcriber includes a keyword encoder unit and an audio transcriber decoder unit. The keyword encoder unit maps the word embedding representation sequence to a keyword embedding sequence. The audio transcriber decoder unit uses a hybrid attention mechanism to calculate keyword-enhanced attention between the keyword embedding sequence and the audio embedding sequence. Based on the keyword-enhanced attention, the audio transcriber decoder unit converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the word embedding representation sequence into a predicted text sequence. The predicted text sequence is fed back to the text embedding model via autoregression and concatenated with entity keywords to generate a new word embedding representation sequence. A new lexical embedding representation sequence is input to the text transcriber. The text transcriber converts the concatenated vector of the audio embedding sequence, the transcription start flag, and the new lexical embedding representation sequence into a predicted text sequence. During the continuous input of the audio stream, the audio transcription decoder unit outputs the predicted text sequence at the current moment in real time. When the audio transcription decoder unit outputs the transcription end flag, it outputs the predicted text sequence as the speech recognition text to the speech buffer manager.

Images