A Single-Event Upset Resistant Reinforcement Method for Transformer-Based Speech Recognition Models

Abstract

The present invention discloses a method for enhancing the resistance to single event upset of a Transformer speech recognition model, which is used to improve the efficiency and accuracy of speech recognition for astronauts in space missions, including: constructing a training set and preprocessing the data in the training set; inputting the preprocessed training set into the Transformer speech recognition model to obtain a trained Transformer speech recognition model; modifying the parameters in the trained Transformer speech recognition model to simulate single event upset events and possible errors in a high-radiation environment, and establishing a SEU impact model; based on the SEU impact model, performing a robustness analysis on the Transformer speech recognition model to determine sensitive modules, and strengthening the sensitive modules with triple modular redundancy technology to obtain a speech recognition model with enhanced resistance to single event upset.

Description

Technical Field

[0001] The present invention relates to the technical fields of aerospace, aviation, deep learning, and radiation tolerance, and particularly to a single event upset resistant reinforcement method based on a Transformer speech recognition model. Background Art

[0002] With the continuous advancement of deep space exploration missions, the importance of intelligent and autonomous systems in the aerospace field has become increasingly prominent. In a high-radiation environment, especially in deep space missions, when astronauts perform complex tasks, they need to rely on speech recognition technology for human-computer interaction to improve the efficiency and accuracy of task execution. However, due to the significantly higher radiation level in the space environment than on the ground, it will cause single event upsets (SEUs) in semiconductor components, which may lead to problems such as parameter errors and inaccurate outputs when deep learning models face radiation effects. This poses a severe test to the robustness and reliability of existing models in a high-radiation environment.

[0003] Currently, the reinforcement methods for single event upsets mainly include triple modular redundancy (TMR) and error detection and correction code (ECC). Triple modular redundancy copies the computing unit three times and performs parallel processing to implement a majority voting mechanism to prevent errors caused by single event upsets. Although TMR is effective in improving system reliability, its hardware overhead is large, especially in complex deep learning models, and this overhead is often unbearable. Error detection and correction code (ECC) is another common reinforcement scheme. By encoding data, it ensures that correct data can be restored when an error occurs. However, the application of ECC in deep learning models may increase the computational latency, and the design and implementation for complex models are also relatively complex. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the prior art and propose a single event upset resistant reinforcement method based on a Transformer speech recognition model.

[0005] To achieve the above purpose, the present invention proposes a single event upset resistant reinforcement method based on a Transformer speech recognition model for improving the efficiency and accuracy of speech recognition of astronauts in space missions, including:

[0006] Step 1) Construct a training set and preprocess the data in the training set;

[0007] Step 2) Input the preprocessed training set into the Transformer speech recognition model to obtain a trained Transformer speech recognition model;

[0008] Step 3) Modify the parameters in the trained Transformer speech recognition model to simulate single-event upsets and errors that may occur in a high-radiation environment, and establish an SEU impact model;

[0009] Step 4) Based on the SEU impact model, perform a robustness analysis on the Transformer speech recognition model to identify sensitive modules, and reinforce the sensitive modules with triple modular redundancy technology to obtain a single-event upset-resistant reinforced speech recognition model.

[0010] Preferably, the step 1) includes:

[0011] Adopt the open-source dataset LibriSpeech including speech and corresponding text transcripts;

[0012] For speech, use the torchaudio library to read the audio files in LibriSpeech, extract the audio signals; calculate the filter bank energy features of the audio signals;

[0013] For text, adopt a subword-based tokenization method to split the text data into smaller units.

[0014] Preferably, the Transformer speech recognition model in the step 2) includes:

[0015] A feature extractor for extracting meaningful feature information from the input audio;

[0016] Six encoders, each encoder consisting of a multi-head self-attention mechanism and a feed-forward neural network, for encoding the input features;

[0017] Six decoders for decoding the features output by the encoders and finally generating the corresponding text output.

[0018] Preferably, the step 3) includes:

[0019] Convert the weight parameter values of the trained Transformer speech recognition model into 32-bit binary form to simulate single-event upsets. The 32-bit binary includes: 1 sign bit, 8 exponent bits, and 32 mantissa bits;

[0020] Randomly change a certain bit in the binary representation from "1 to 0" or "0 to 1" to simulate errors that may occur in a high-radiation environment, and thus obtain the SEU impact model.

[0021] Preferably, the sensitive modules in the step 4) include: the multi-head self-attention mechanisms of the first two encoders.

[0022] Preferably, the triple modular redundancy technology is used to reinforce the sensitive module to obtain a speech recognition model with single event upset (SEU) resistance, including:

[0023] Copy the computing unit of the sensitive module three times to form three computing units running in parallel;

[0024] Implement a majority voting mechanism for the three computing units running in parallel. When one of the computing units is affected by single event upset, the correct output is selected by a two-out-of-three method;

[0025] Obtain a speech recognition model with single event upset resistance.

[0026] Compared with the prior art, the advantages of the present invention are as follows:

[0027] 1. Targeted reinforcement: The present invention conducts a detailed robustness analysis on the modules of the Transformer speech recognition model, identifies the modules sensitive to single event upset (SEU), and thus conducts targeted reinforcement. This method avoids the waste of resources in traditional reinforcement methods such as check codes and triple modular redundancy (TMR), and achieves more precise protection;

[0028] 2. Resource optimization: Compared with the traditional comprehensive reinforcement method, the present invention only applies the triple modular redundancy technology to key modules, significantly reducing the hardware overhead and computational complexity. Thus, while ensuring the system performance, the demand for system resources is reduced. This makes it more practical in aerospace applications with limited resources;

[0029] 3. Performance improvement: By separately reinforcing the modules with greater impact on SEU, the stability and reliability of the model in a high-radiation environment are ensured, thus effectively improving the accuracy and response speed of speech recognition, and providing safer and more efficient support for astronauts' operations;

[0030] 4. Scalability and compatibility: The method of the present invention has good scalability. According to the requirements of different application scenarios, corresponding reinforcement configurations can be made for different modules, providing a flexible solution for applying the Transformer speech recognition model in various extreme environments in the future. The reinforcement method of the present invention is compatible with existing deep learning models and their training frameworks, without requiring major structural changes to the model, facilitating integration and implementation, and reducing the difficulty of deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flowchart of the method for reinforcing the Transformer speech recognition model against single event upset of the present invention;

[0032] Figure 2 is the Transformer model structure;

[0033] Figure 3 It is the modified Transformer model structure. Detailed implementation manners

[0034] The present invention provides a method for strengthening the anti-single event upset of a Transformer speech recognition model. By analyzing the robustness of the parameters of each module in the model under single event upset, the modules that are more sensitive to SEU and have a greater impact on the model output results are identified, and triple modular redundancy (TMR) strengthening is applied to these modules alone. This method improves the stability and reliability of the Transformer speech recognition model in a high-radiation environment with less resource consumption while ensuring the model performance.

[0035] The main steps include:

[0036] 1. Module robustness analysis: For different modules such as the embedding layer, multi-head attention mechanism, and feed-forward network layer in the Transformer speech recognition model, evaluate the sensitivity of their parameters under single event upset conditions to determine which modules are more vulnerable to SEU.

[0037] 2. Selective strengthening: Perform triple modular redundancy strengthening on the sensitive modules identified in the robustness analysis (such as the weight parameters in the attention mechanism or the computing nodes in the feed-forward network), and keep the other modules in their original structures to reduce the overhead.

[0038] 3. Strengthening verification: Verify the performance of the strengthened model through simulation verification and experimental evaluation to ensure the robustness of the model in a high-radiation environment.

[0039] The technical solution of the present invention will be described in detail below in conjunction with the drawings and embodiments to illustrate how to use the anti-single event upset strengthening method of the present invention to improve the stability and reliability of the Transformer speech recognition model in a high-radiation environment.

[0040] Embodiment

[0041] As Figure 1 shown, the embodiment of the present invention provides a method for strengthening the anti-single event upset of a Transformer speech recognition model, aiming to improve the efficiency and accuracy of speech recognition of astronauts in space missions with less performance resource consumption. It includes the following steps:

[0042] Step 1) Construct a training set and preprocess the training set data; specifically:

[0043] The training set data adopted by the present invention is the open-source dataset LibriSpeech, which mainly includes a large amount of speech and corresponding text transcripts. The specific steps of data preprocessing are as follows:

[0044] 1. Audio processing: Use the torchaudio library to read audio files in LibriSpeech and extract audio signals.

[0045] 2. Feature extraction: Calculate the FBank (filter bank energy) features of the audio signal for subsequent model training. FBank features can effectively capture the audio features in the speech signal, helping to improve the recognition accuracy.

[0046] 3. Text processing: For the text part, adopt a subword-based tokenization method to split the text data into smaller units to meet the input requirements of the Transformer model. This method not only reduces the size of the vocabulary but also enhances the model's ability to handle unseen words.

[0047] Step 2) Input the preprocessed training set into the Transformer speech recognition model to obtain the trained Transformer speech recognition model;

[0048] The Transformer model structure used in the present invention is as follows Figure 2 as shown:

[0049] 1. Feature extractor: Used to extract meaningful feature information from the input audio.

[0050] 2. Encoder: Consists of 6 layers of encoders, each layer composed of a multi-head self-attention mechanism and a feed-forward neural network, responsible for encoding the input features.

[0051] 3. Decoder: Also consists of 6 layers of decoders, which decode the features output by the encoder and finally generate the corresponding text output.

[0052] During the model construction process, the torch library is used to manage and optimize the model parameters to ensure the efficient operation of the model.

[0053] Step 3) Modify the parameters in the trained Transformer speech recognition model to simulate single-event upsets and errors that may occur in a high-radiation environment, and establish an SEU impact model;

[0054] Considering that the parameters of the Transformer model are stored in SRAM memory cells and their storage form is essentially binary, according to the IEEE754 protocol, the representation of a 32-bit floating-point number is as follows:

[0055] Sign bit (S): 1 bit

[0056] Exponent bit (E): 8 bits

[0057] Least significant digit (M): 23 bits

[0058] Therefore, when a single event upset occurs, a certain bit of the model weight may change. For this reason, the steps to establish the SEU impact model in the present invention are as follows:

[0059] 1. Convert the model weight parameter value into a 32-bit binary form to simulate a single event upset.

[0060] 2. Randomly change a certain bit in the binary representation from "1 to 0" or "0 to 1" to simulate possible errors in a high-radiation environment.

[0061] Step 4) Based on the SEU impact model, perform a robustness analysis on the Transformer speech recognition model, determine the sensitive modules, and reinforce the sensitive modules with the triple modular redundancy technology to obtain a speech recognition model with enhanced resistance to single event upsets.

[0062] Steps of the robustness analysis:

[0063] 1. Deploy the Transformer speech recognition model to a simulation environment, and analyze the sensitivity of the parameters of each module to errors by injecting single event upset faults.

[0064] 2. Analyze each main module (embedding layer, multi-head attention mechanism, feed-forward network layer, decoder, etc.) in the model one by one, and evaluate the impact of parameter flipping on the speech recognition performance.

[0065] Steps of the triple modular redundancy reinforcement:

[0066] For the sensitive modules (multi-head self-attention mechanisms of the first two layers of the encoder) identified in the robustness analysis, use the triple modular redundancy technology for reinforcement, as Figure 3 shown. The specific implementation includes:

[0067] 1. Duplicate the computing units: Duplicate the computing units of the sensitive modules (multi-head self-attention modules of the first two layers of the encoder) three times to form three computing units running in parallel.

[0068] 2. Implement a majority voting mechanism: When one of the computing modules is affected by a single event upset, select the correct output by a two-out-of-three method to ensure the accuracy of the final result. This mechanism effectively prevents speech recognition errors caused by single event upsets.

[0069] Steps of the simulation verification:

[0070] The simulation verification of the hardened Transformer speech recognition model is carried out by injecting single-event upset faults, and the differences in recognition accuracy, processing delay, and hardware overhead of the model before and after hardening are tested. At the same time, the performance of the model in the real radiation environment is evaluated through experimental data.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the embodiments, those of ordinary skill in the art should understand that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A method for enhancing the radiation hardness of a Transformer-based speech recognition model to improve the efficiency and accuracy of speech recognition for astronauts in space missions, comprising: Step 1) Construct a training set and preprocess the data in the training set; Step 2) Input the preprocessed training set into the Transformer speech recognition model to obtain a trained Transformer speech recognition model; Step 3) Modify the parameters in the trained Transformer speech recognition model to simulate single event upset (SEU) events and errors that may occur in a high-radiation environment, and establish an SEU impact model; Step 4) Based on the SEU impact model, perform a robustness analysis on the Transformer speech recognition model to identify sensitive modules, and apply triple modular redundancy (TMR) technology to reinforce the sensitive modules to obtain a radiation-hardened speech recognition model.

2. The anti-single event upset reinforcement method for a Transformer-based speech recognition model according to claim 1, wherein The said Step 1) includes: Adopt the open-source dataset LibriSpeech that includes speech and corresponding text transcripts; For speech, use the torchaudio library to read the audio files in LibriSpeech, extract the audio signals, and calculate the filter bank energy features of the audio signals; For text, adopt a subword-based tokenization method to segment the text data into smaller units.

3. The anti-single event upset reinforcement method for the Transformer-based speech recognition model according to claim 1, characterized in that In the said Step 2), the Transformer speech recognition model includes: A feature extractor for extracting meaningful feature information from the input audio; Six encoders, each encoder consisting of a multi-head self-attention mechanism and a feed-forward neural network, for encoding the input features; Six decoders for decoding the features output by the encoders and finally generating the corresponding text output.

4. The anti-single event upset reinforcement method for a Transformer-based speech recognition model according to claim 1, characterized in that The said Step 3) includes: Convert the weight parameter values of the trained Transformer speech recognition model into 32-bit binary form to simulate single event upset events. The 32-bit binary includes: 1 sign bit, 8 exponent bits, and 23 mantissa bits; Randomly change a certain bit in the binary representation from "1 to 0" or "0 to 1" to simulate errors that may occur in a high-radiation environment, thereby obtaining the SEU impact model.

5. The anti-single event upset reinforcement method for a Transformer-based speech recognition model according to claim 3, wherein The sensitive modules in the said Step 4) include: the multi-head self-attention mechanisms of the first two encoders.

6. The anti-single event upset reinforcement method based on the Transformer speech recognition model according to claim 5, characterized in that The said applying triple modular redundancy technology to reinforce the sensitive modules to obtain a radiation-hardened speech recognition model includes: Copy the computing units of the sensitive modules three times to form three computing units running in parallel; Implement a majority voting mechanism for the three computing units running in parallel. When one of the computing units is affected by a single event upset, select the correct output by a two-out-of-three method; Obtain a radiation-hardened speech recognition model.

Images