A helium-oxygen environment speech recognition method and system based on whisper

Abstract

This invention relates to a Whisper-based helium-oxygen environment speech recognition method and system: helium-oxygen speech data acquisition and construction; helium-oxygen speech feature extraction and logarithmic Mel spectrum construction; construction of a Whisper-LoRA model; model training and evaluation: the training set is input into the model for training and fine-tuned using LoRA low-rank increments, the loss value is calculated using a loss function, the evaluation set is input into the trained model for evaluation and CER is calculated, the model converges when the loss value is lower than a set threshold and the CER meets preset conditions; model prediction: the newly acquired helium-oxygen speech, after processing, has its logarithmic Mel spectrum features input into the converged model for speech recognition. During the recognition process, hot word bias guides the decoding path to prioritize coverage of diving-related terms, multi-speed ensemble technology enhances TTA during testing, and candidate re-ranking technology of the language model are used to perform feature recognition of helium-oxygen speech to generate high-precision text.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing, automatic speech recognition (ASR), and deep learning, and in particular to a Whisper-based speech recognition method and system for helium-oxygen environments. Designed for helium-oxygen saturated diving environments, it utilizes a combination of techniques including a large-scale pre-trained Whisper model, low-rank adaptation (LoRA) fine-tuning, hot word bias during the decoding stage, test-time enhancement (TTA), language model reordering, and scrolling contextual hints. Background Technology

[0002] 1. Speech distortion problem in helium-oxygen environment

[0003] In saturation diving operations, the density of the helium-oxygen mixture breathed by divers differs significantly from that of air. This increases the speed of sound and alters the acoustic impedance, causing the overall resonant peak of the vocal tract to shift upwards. High-frequency components of the speech spectrum are enhanced, while low-frequency energy is weakened. Speech rate, the boundary between voiced and unvoiced sounds, and the resonance structure all exhibit significant differences compared to atmospheric pressure air. Directly using a standard Mandarin ASR model to recognize this type of speech at depths of 12m and 25m often results in severe errors in spelling, word choice, and omissions of proper nouns.

[0004] 2. Advantages and limitations of large-scale Whisper models

[0005] Whisper, as a large-scale multilingual pre-trained Transformer speech recognition model, has strong robustness and multi-task capabilities. However, its pre-training corpus contains almost no data on helium-oxygen environments and diving scenarios, limiting its adaptability to acoustic distortion and specific domain terminology. If traditional full-parameter fine-tuning is used, not only will the computational and storage costs be enormous, but it will also easily damage the generalization performance of the original model in other scenarios.

[0006] 3. Problems with existing fine-tuning and decoding methods

[0007] In existing technologies, one approach only makes minor adjustments to the acoustic model, failing to fully utilize hot words, language models, and contextual information during the decoding stage, resulting in low recognition rates for domain-specific terms (such as "diving bell," "saturation diving," and "decompression chamber"). Another approach, while using n-gram or neural language models, lacks systematic hyperparameter tuning for helium-oxygen environments, often relying on human experience for decoding hyperparameters, making it difficult to achieve an optimal trade-off between recognition accuracy and latency. Therefore, a dedicated Whisper-LoRA fine-tuning method for helium-oxygen environments is needed, combined with a systematic decoding enhancement strategy, to significantly reduce the character error rate (CER) while ensuring controllable latency, especially improving the recognition accuracy of specialized terms and key instruction words. Summary of the Invention

[0008] This invention addresses the problems and shortcomings of existing technologies by providing a Whisper-based helium-oxygen environment speech recognition method and system. It utilizes LoRA to fine-tune the query / value projection matrix of the attention layer in the Whisper model, achieving helium-oxygen acoustic domain adaptation with minimal parameter overhead. During the prediction stage, it introduces hot word bias, TTA speech rate perturbation, character binary grammar language model reordering, and scrolling contextual cues, thereby significantly reducing CER in helium-oxygen environments at different depths and improving the accuracy of key diving terminology recognition, while keeping system latency within an acceptable engineering range.

[0009] The present invention solves the above-mentioned technical problems through the following technical solution:

[0010] This invention provides a Whisper-based speech recognition method for helium-oxygen environments, characterized by the following steps:

[0011] S1. Helium-oxygen speech data acquisition and construction: In the shipborne mobile saturation diving chamber, the pressure inside the chamber is controlled to the corresponding preset pressure, and helium-oxygen mixed gas is filled in. The speech of multiple divers at different water depths is collected. Each helium-oxygen speech is preprocessed to obtain the preprocessed helium-oxygen speech of each diver at different water depths. The preprocessed helium-oxygen speech is paired with manually annotated text to construct the first helium-oxygen speech dataset.

[0012] S2. Helium-oxygen speech feature extraction and log-Mel spectrum construction: The preprocessed helium-oxygen speech waveforms in the helium-oxygen speech dataset are converted into fixed-dimensional log-Mel spectrum features through short-time Fourier transform and Mel filter bank. A second helium-oxygen speech dataset is constructed, consisting of the log-Mel spectrum features of the preprocessed helium-oxygen speech and the corresponding labeled text. The second helium-oxygen speech dataset is divided into a training set and an evaluation set.

[0013] S3. Constructing the Whisper-LoRA model: Using the Whisper large model as the base model, while keeping the original parameters frozen, a low-rank incremental matrix is ​​introduced only on the query / value projection layer of multi-head attention to form the Whisper-LoRA model. The Whisper-LoRA model consists of an encoder and a decoder.

[0014] S4. Whisper-LoRA Model Training and Evaluation: The training set is input into the Whisper-LoRA model for model training and fine-tuned using LoRA low-rank increments. The encoder encodes each logarithmic Mel spectral feature in the training set to obtain the corresponding speech feature sequence. The decoder decodes the obtained speech feature sequence and generates a Chinese text token sequence autoregressively. The loss value is calculated using a loss function. The evaluation set is input into the trained Whisper-LoRA model for model evaluation and CER calculation. When the loss value is lower than a set threshold and the CER meets the preset conditions, the Whisper-LoRA model converges.

[0015] S5. Whisper-LoRA Model Prediction: The newly acquired helium-oxygen speech is preprocessed, converted into fixed-dimensional logarithmic Mel spectral features by short-time Fourier transform and Mel filter bank, and then input into the converged Whisper-LoRA model for speech recognition. During the recognition process, the hot word bias guides the decoding path to prioritize coverage of diving-related terms, enhances the multi-rate integration of TTA during testing to slightly scale the rate of the input helium-oxygen speech, uses candidate reordering technology of lightweight character bigram language model, and uses scrolling context prompts and long speech concatenation technology to perform feature recognition on the input helium-oxygen speech to generate high-precision text.

[0016] This invention also provides a Whisper-based helium-oxygen environment speech recognition system, characterized in that it includes:

[0017] A helium-oxygen speech acquisition module is used to control the pressure inside the shipborne mobile saturation diving chamber to a corresponding preset pressure, fill it with helium-oxygen mixed gas, and collect the speech of multiple divers at different water depths. Each helium-oxygen speech is preprocessed to obtain the preprocessed helium-oxygen speech of each diver at different water depths. The preprocessed helium-oxygen speech is then paired with manually annotated text to construct the first helium-oxygen speech dataset.

[0018] A feature extraction module is used to convert each preprocessed helium-oxygen speech waveform in the helium-oxygen speech dataset into fixed-dimensional log-Mel spectral features through short-time Fourier transform and Mel filter bank. The second helium-oxygen speech dataset is constructed by the log-Mel spectral features of the preprocessed helium-oxygen speech and the corresponding labeled text. The second helium-oxygen speech dataset is divided into a training set and an evaluation set.

[0019] A model building module is used to use the Whisper large model as the base model. While keeping the original parameters frozen, a low-rank incremental matrix is ​​introduced only on the query / value projection layer of multi-head attention to form the Whisper-LoRA model. The Whisper-LoRA model consists of an encoder and a decoder.

[0020] A model training and evaluation module is used to input the training set into the Whisper-LoRA model for model training and to fine-tune the model using LoRA low-rank increment. The encoder encodes each log Mel spectral feature in the training set to obtain the corresponding speech feature sequence. The decoder decodes the obtained speech feature sequence and generates a Chinese text token sequence autoregressively. The loss value is calculated using a loss function. The evaluation set is input into the trained Whisper-LoRA model for model evaluation and CER calculation. When the loss value is lower than a set threshold and the CER meets the preset conditions, the Whisper-LoRA model converges.

[0021] A model prediction module is used to preprocess newly acquired helium-oxygen speech, convert it into fixed-dimensional logarithmic Mel spectral features through short-time Fourier transform and Mel filter bank, and then input it into a converged Whisper-LoRA model for speech recognition. During the recognition process, the module combines hot word bias to guide the decoding path to prioritize coverage of diving-related terms, multi-rate integration to enhance TTA during testing to slightly scale the speech rate of the input helium-oxygen speech, candidate reordering of a lightweight character binary grammar language model, and scrolling contextual prompts and long speech concatenation techniques to perform feature recognition on the input helium-oxygen speech to generate high-precision text.

[0022] The positive and progressive effects of this invention are as follows:

[0023] (1) Low-cost helium-oxygen domain adaptation: LoRA is used to perform low-rank incremental learning only on the query / value linear transformation of the attention layer in the Whisper model. The parameters involved in training account for about 0.01% of the total number of parameters in the Whisper model, which significantly reduces memory usage and training time, while retaining the generalization ability of the original model in other scenarios.

[0024] (2) Hot word enhancement recognition for diving instructions: By hot word bias, high-risk instruction words such as "diving bell", "saturation diving" and "decompression chamber" are explicitly amplified in the decoding probability space, increasing the frequency of these words in the N-best sequence and reducing the potential safety risks caused by misidentification.

[0025] (3) Synergistic gains of TTA+ language model reordering: The generated candidate diversity is fully utilized, which further reduces CER by about 37% on the baseline, while the delay overhead is predictable and controllable.

[0026] (4) Scrolling context maintains consistency of long speech: By scrolling prompts, the last few characters of the recognition result of the previous segment are used as the text prefix of the current segment, which effectively avoids the splitting and repetition of names and terms caused by the cutting of long instructions, and improves the semantic coherence across segments. Attached Figure Description

[0027] Figure 1 This is a flowchart of a preferred embodiment of the helium-oxygen environment speech recognition method of the present invention.

[0028] Figure 2 This is a structural block diagram of a helium-oxygen environment speech recognition system according to a preferred embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] For ease of description, only the parts relevant to the present invention are shown in the accompanying drawings. The terms "first," "second," etc., used in this invention are merely for the convenience of describing the technical solutions of the invention and do not have a specific limiting effect; they are all general references and do not constitute a limitation on the technical solutions of the present invention.

[0031] like Figure 1 As shown, this embodiment of the invention provides a Whisper-based speech recognition method for helium-oxygen environments, comprising the following steps:

[0032] Step 101: Helium-Oxygen Speech Data Acquisition and Construction: In the shipborne mobile saturation diving chamber, the chamber pressure is controlled to the corresponding preset pressure, and a helium-oxygen mixture is introduced. Speech data from multiple divers at different water depths is collected. Each collected helium-oxygen speech is preprocessed to obtain preprocessed helium-oxygen speech data from each diver at different water depths. The preprocessed helium-oxygen speech data is then paired with manually annotated text to construct the first helium-oxygen speech dataset = {preprocessed helium-oxygen speech 1, annotated text 1; preprocessed helium-oxygen speech 2, annotated text 2; ...}.

[0033] 1. Helium-Oxygen Environment Construction: On a ship equipped with a mobile saturation diving chamber and a diving bell, the saturation chamber is pressurized to about 3MPa and filled with a helium-oxygen mixture to maintain the oxygen partial pressure required for medical use; four divers are organized to take turns performing inspection tasks at depths of 12m and 25m, and read out preset Chinese sentences through an underwater communication system during the mission.

[0034] 2. Voice and Text Annotation: The acquired helium-oxygen voice is preprocessed, including editing, de-silencing, and quality screening, to retain sentences with clear voice content and stable channel. Professionals then transcribe and proofread the audio in standard Mandarin, creating an annotation file consisting of "audio file + corresponding text".

[0035] Step 102, Helium-Oxygen Speech Feature Extraction and Log-Melbourne Spectrum Construction: The preprocessed helium-oxygen speech waveforms in the helium-oxygen speech dataset are converted into fixed-dimensional log-Melbourne spectrum features through short-time Fourier transform (STFT) and Mel filter bank. A second helium-oxygen speech dataset is constructed, consisting of the log-Melbourne spectrum features of the preprocessed helium-oxygen speech and the corresponding labeled text = {log-Melbourne spectrum features of preprocessed helium-oxygen speech 1, labeled text 1; log-Melbourne spectrum features of preprocessed helium-oxygen speech 2, labeled text 2; ...}, which provides a unified acoustic input representation for the Whisper model. The second helium-oxygen speech dataset is divided into a training set and an evaluation set.

[0036] Step 103: Construct the Whisper-LoRA model: Using the Whisper large model as the base model, while keeping the original parameters frozen, a low-rank incremental matrix is ​​introduced only on the query / value projection layer of multi-head attention to form the Whisper-LoRA model. The Whisper-LoRA model consists of an encoder and a decoder. A small number of parameters are updated through LoRA to adapt the model to helium-oxygen acoustic distortion and domain terminology.

[0037] Step 104, Whisper-LoRA Model Training and Evaluation: The training set is input into the Whisper-LoRA model for training, and the model is fine-tuned using LoRA low-rank increments. The encoder encodes each logarithmic Mel spectral feature in the training set to obtain the corresponding speech feature sequence, and the decoder decodes the obtained speech feature sequence to generate a Chinese text token sequence autoregressively. The loss value is calculated using a loss function. The evaluation set is input into the trained Whisper-LoRA model for model evaluation and CER is calculated. When the loss value is lower than a set threshold and the CER meets the preset condition (the decrease in CER over multiple consecutive evaluations is <0.001), the Whisper-LoRA model converges.

[0038] In step 104, the encoder encodes each logarithmic Mel-spectral feature in the training set to obtain the corresponding speech feature sequence:

[0039] Formula (1):

[0040] in, H is an encoder network consisting of multiple layers of self-attention and feedforward sub-layers stacked together; H is the latent acoustic representation of the encoder output, i.e., the speech feature sequence.

[0041] The decoder decodes the acquired speech feature sequence and autoregressively generates a Chinese text token sequence:

[0042] Given H, the decoder decodes the speech feature sequence and autoregressively generates the corresponding Chinese text token sequence y=(y1, ..., y...). N Its conditional probability decomposition is:

[0043] Formula (2):

[0044] in, This is the token sequence after adding start and end markers; y N For the token x, the speech feature sequence represents helium-oxygen speech; The probability of the entire helium-oxygen speech being this token sequence y; θ represents all historical tokens generated up to step t; θ is the parameter set of the Whisper–LoRA model.

[0045] During the training phase, teacher-mandated instruction is used, and the decoder uses the true prefix at each step. As a condition, minimize the token-by-token cross-entropy loss:

[0046] Formula (3):

[0047] in, Let L be the target token for step t; CE The log-likelihood loss at the character level is for the entire helium-oxygen speech.

[0048] Step 105, Whisper-LoRA Model Prediction: The newly acquired helium-oxygen speech is preprocessed, converted into fixed-dimensional logarithmic Mel spectral features by short-time Fourier transform and Mel filter bank, and then input into the converged Whisper-LoRA model for speech recognition. During the recognition process, the hot word bias guides the decoding path to prioritize coverage of diving-related terms, enhances the TTA multi-rate integration during testing to slightly scale the speech rate of the input helium-oxygen speech, uses the candidate reordering technique of the lightweight character bigram language model, and uses scrolling context prompts and long speech concatenation techniques to perform feature recognition on the input helium-oxygen speech to generate high-precision text.

[0049] In step 105, the hot word bias guides the decoding path to prioritize covering terms in the diving domain:

[0050] By organizing and statistically analyzing the data, key terms are extracted from the helium-oxygen diving mission scripts to form a set of hot words, including terms such as "diving bell," "saturation diving," "Diver No. 1," and "decompression chamber." These terms often have safety or mission instruction significance in diving communications. By importing the set of hot words, a positive logarithmic probability bias is applied to the sub-word tokens that can be expanded into hot words during the decoding process, guiding the decoding path to prioritize the coverage of key terms in the diving field.

[0051] Specifically, in the t-th decoding step, let the original logits be z. t ∈R |V| Where logits is the model's original score for each token, V is the sub-vocabulary, and H is a subset of tokens that can be expanded to include hot words. t ⊆V, add a bias to it:

[0052] Formula (4):

[0053] Where k is the index of the candidate token in the hot word set; z t,k This represents the raw log-odds of the model for token k without bias. The log-odds ratio after adding hot word bias; λ bias >0 represents the hot word bias strength, and this embodiment recommends a value of 1.2; II(·) is the indicator function, which takes the value 1 when the condition is true, and 0 otherwise; H t This is the set of tokens that can form hot words under the current text prefix.

[0054] Then, the weighted probability distribution for the next step is obtained by applying softmax:

[0055] Formula (5):

[0056] Where P(y) t = k | ·) represents the conditional probability that the decoder selects token k at step t under helium-oxygen conditions.

[0057] In step 105, a technique is used to enhance the multi-rate integration of TTA during testing to perform slight rate scaling on the input helium-oxygen speech:

[0058] To reduce sensitivity to individual speech rates, during the decoding process, a small speech rate scaling is applied to the speech feature sequence corresponding to the input helium-oxygen speech x, resulting in a set of perturbed speech φ. s(x) Where s∈S is the set of velocity coefficients, for example, S={0.97,1.00,1.03}. The same Whisper–LoRA model is applied to each velocity coefficient s to obtain the log-likelihood of the candidate transcriber y:

[0059] Formula (6): log P θ TTA (y | x) = (1 / |S|) · Σ_{s∈S} log P θ (y | φ s(x) )

[0060] Where, φ s(x) This is the speech sequence obtained by playing the original helium-oxygen speech at different speeds according to coefficient s. Physically, it can be understood as a simulation of a diver speaking quickly and slowly; |S| represents the number of speed coefficients; log P θ TTA (·) To enhance the mean log-likelihood of candidate transcriptions after TTA integration during testing, thereby improving robustness to speech rate fluctuations and transient distortions.

[0061] In step 105, the candidate reordering technique for the lightweight character bigram language model is as follows:

[0062] Using a lightweight character bigram language model trained on reference transcription, each candidate transcription is weighted and sorted based on acoustic log-likelihood, hot word coverage, and language model score to output the optimal transcription. Specifically:

[0063] We statistically analyze the unary and binary co-occurrence distributions of Chinese characters on reference transcribed texts, construct a lightweight character-binary grammar language model, and apply it to any transcribed text. The logarithmic probability is defined as:

[0064] Formula (7):

[0065] Among them, c i For the i-th Chinese character; P(c i+1 | c i ) represents the character c in a statistical sense i Followed by the character c i+1 The conditional probability, after smoothing, gives a non-zero probability for unseen combinations.

[0066] For the nth candidate transcription of helium-oxygen speech Its length is defined as L n Hot word coverage rate , representing the proportion of characters covered by hot words in the transcription, the comprehensive scoring function is:

[0067] Formula (8):

[0068] Wherein, α is the weight of the acoustic log-likelihood score, reflecting the model's confidence in the speech-text mapping under helium-oxygen conditions; β is the weight of hot word coverage, emphasizing the importance of correctly recognizing diving command words; γ is the weight of the language model score, measuring the naturalness of the transcribed sequence in terms of Chinese language statistical regularities; Cov_H(·) is the ratio of the number of matched characters to the total number of characters after matching the transcribed sequence with the hot word set.

[0069] The final output is transcribed to the one with the highest overall score:

[0070] Formula (9): .

[0071] In step 105, scrolling context prompts and long speech splicing technology are used:

[0072] For long helium-oxygen speech exceeding the single decoding length limit, during the decoding process, the encoded features of the entire long helium-oxygen speech segment are divided into C segments of fixed duration {x}. (1) , x (2) , ..., x (C)}, set a text hint p for the c-th segment. (c) Its decoding target is:

[0073] Formula (10):

[0074] Where p (c) This indicates the cue text at the end of the previous segment, which physically corresponds to the semantic continuation of the diver's unfinished sentence.

[0075] After segment c is decoded, it is transcribed from its best value. The last L characters of the middle segment are used as a hint for the next segment:

[0076] Formula (11):

[0077] The final transcription of the entire long helium-oxygen speech was obtained through simple concatenation:

[0078] Formula (12): .

[0079] In this embodiment, long speech is divided into adjacent overlapping segments with a fixed duration. Several characters at the end of the recognition result of the previous segment are used as prompts and are input into the Whisper-LoRA model along with the current segment to achieve cross-segment semantic continuity and consistent translation of proper nouns.

[0080] like Figure 2As shown, this embodiment of the invention also provides a Whisper-based helium-oxygen environment speech recognition system, including a helium-oxygen speech acquisition module 1, a feature extraction module 2, a model building module 3, a model training and evaluation module 4, and a model prediction module 5.

[0081] The helium-oxygen speech acquisition module 1 is used in a shipborne mobile saturation diving chamber to control the internal pressure to a corresponding preset pressure, fill the chamber with helium-oxygen mixed gas, and collect the speech of multiple divers at different water depths. Each helium-oxygen speech is preprocessed to obtain preprocessed helium-oxygen speech of each diver at different water depths. The preprocessed helium-oxygen speech is then paired with manually annotated text to construct the first helium-oxygen speech dataset = {preprocessed helium-oxygen speech 1, annotated text 1; preprocessed helium-oxygen speech 2, annotated text 2; ...}.

[0082] Feature extraction module 2 is used to convert each preprocessed helium-oxygen speech waveform in the helium-oxygen speech dataset into fixed-dimensional log-Mel spectral features through short-time Fourier transform and Mel filter bank. A second helium-oxygen speech dataset is constructed, consisting of the log-Mel spectral features of the preprocessed helium-oxygen speech and the corresponding labeled text = {log-Mel spectral features of preprocessed helium-oxygen speech 1, labeled text 1; log-Mel spectral features of preprocessed helium-oxygen speech 2, labeled text 2; ...}, which provides a unified acoustic input representation for the Whisper model. The second helium-oxygen speech dataset is divided into a training set and an evaluation set.

[0083] Model building module 3 is used to take the Whisper large model as the base model, and while keeping the original parameters frozen, introduce a low-rank incremental matrix only on the query / value projection layer of multi-head attention to form the Whisper-LoRA model. The Whisper-LoRA model consists of an encoder and a decoder, and a small number of parameters are updated through LoRA to make the model adapt to helium-oxygen acoustic distortion and domain terminology.

[0084] The model training and evaluation module 4 is used to input the training set into the Whisper-LoRA model for model training and to fine-tune the model using LoRA low-rank increments. The encoder encodes each logarithmic Mel spectral feature in the training set to obtain the corresponding speech feature sequence. The decoder decodes the obtained speech feature sequence and generates a Chinese text token sequence autoregressively. The loss value is calculated using a loss function. The evaluation set is input into the trained Whisper-LoRA model for model evaluation and CER calculation. When the loss value is lower than a set threshold and the CER meets the preset conditions, the Whisper-LoRA model converges.

[0085] Model prediction module 5 is used to preprocess the newly acquired helium-oxygen speech, convert it into fixed-dimensional logarithmic Mel spectral features through short-time Fourier transform and Mel filter bank, and then input it into the converged Whisper-LoRA model for speech recognition. During the recognition process, it combines hot word bias to guide the decoding path to prioritize coverage of diving-related terms, multi-rate integration to enhance TTA during testing to slightly scale the speech rate of the input helium-oxygen speech, candidate reordering of the lightweight character bigram language model, and scrolling context prompts and long speech concatenation techniques to perform feature recognition on the input helium-oxygen speech to generate high-precision text.

[0086] This invention also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the aforementioned method.

[0087] This invention also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the aforementioned method.

[0088] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0089] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A Whisper-based speech recognition method for helium-oxygen environments, characterized in that, Includes the following steps: S1. Helium-oxygen speech data acquisition and construction: In the shipborne mobile saturation diving chamber, the pressure inside the chamber is controlled to the corresponding preset pressure, and helium-oxygen mixed gas is filled in. The speech of multiple divers at different water depths is collected. Each helium-oxygen speech is preprocessed to obtain the preprocessed helium-oxygen speech of each diver at different water depths. The preprocessed helium-oxygen speech is paired with manually annotated text to construct the first helium-oxygen speech dataset. S2. Helium-oxygen speech feature extraction and log-Mel spectrum construction: The preprocessed helium-oxygen speech waveforms in the helium-oxygen speech dataset are converted into fixed-dimensional log-Mel spectrum features through short-time Fourier transform and Mel filter bank. A second helium-oxygen speech dataset is constructed, consisting of the log-Mel spectrum features of the preprocessed helium-oxygen speech and the corresponding labeled text. The second helium-oxygen speech dataset is divided into a training set and an evaluation set. S3. Constructing the Whisper-LoRA model: Using the Whisper large model as the base model, while keeping the original parameters frozen, a low-rank incremental matrix is ​​introduced only on the query / value projection layer of multi-head attention to form the Whisper-LoRA model. The Whisper-LoRA model consists of an encoder and a decoder. S4. Whisper-LoRA Model Training and Evaluation: The training set is input into the Whisper-LoRA model for model training and fine-tuned using LoRA low-rank increments. The encoder encodes each logarithmic Mel spectral feature in the training set to obtain the corresponding speech feature sequence. The decoder decodes the obtained speech feature sequence and generates a Chinese text token sequence autoregressively. The loss value is calculated using a loss function. The evaluation set is input into the trained Whisper-LoRA model for model evaluation and CER calculation. When the loss value is lower than a set threshold and the CER meets the preset conditions, the Whisper-LoRA model converges. S5. Whisper-LoRA Model Prediction: The newly acquired helium-oxygen speech is preprocessed, converted into fixed-dimensional logarithmic Mel spectral features by short-time Fourier transform and Mel filter bank, and then input into the converged Whisper-LoRA model for speech recognition. During the recognition process, the hot word bias guides the decoding path to prioritize coverage of diving-related terms, enhances the multi-rate integration of TTA during testing to slightly scale the rate of the input helium-oxygen speech, uses candidate reordering technology of lightweight character bigram language model, and uses scrolling context prompts and long speech concatenation technology to perform feature recognition on the input helium-oxygen speech to generate high-precision text.

2. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 1, characterized in that, In S4, the encoder encodes each logarithmic Mel spectral feature in the training set to obtain the corresponding speech feature sequence: Official (1): ; in, is an encoder network consisting of multiple layers of self-attention and feedforward sub-layers stacked together; H is the latent acoustic representation of the encoder output, i.e., the speech feature sequence; The decoder decodes the acquired speech feature sequence and autoregressively generates a Chinese text token sequence: Given H, the decoder decodes the speech feature sequence and autoregressively generates the corresponding Chinese text token sequence y=(y1, ..., y...). N Its conditional probability decomposition is: Official (2): ; in, This is the token sequence after adding start and end markers; y N For the token x, the speech feature sequence represents helium-oxygen speech; The probability of the entire helium-oxygen speech being this token sequence y; θ represents all historical tokens generated up to step t; θ is the parameter set of the Whisper–LoRA model.

3. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 2, characterized in that, During the training phase, teacher-mandated instruction is used, and the decoder uses the true prefix at each step. As a condition, minimize the token-by-token cross-entropy loss: Official (3): ; in, Let L be the target token for step t; CE The log-likelihood loss at the character level is for the entire helium-oxygen speech.

4. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 1, characterized in that, In S4, the CER is guaranteed to meet the preset condition: the decrease in CER over multiple consecutive evaluations is less than 0.

001.

5. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 1, characterized in that, In S5, hot word bias guides the decoding path to prioritize covering terms in the diving domain: By organizing and statistically analyzing the data, key terms are extracted from the helium-oxygen diving mission scripts to form a set of hot words. By importing the set of hot words and applying a positive log-probability bias to the sub-word tokens that can be expanded into hot words during the decoding process, the decoding path is guided to prioritize the coverage of key terms in the diving field.

6. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 5, characterized in that, In the t-th decoding step, let the original logits be z. t ∈R |V| Where logits is the model's original score for each token, and V is the sub-vocabulary, which can be expanded to a subset of tokens containing hot words. Add a bias to it: Official (4): ; Where k is the index of the candidate token in the hot word set; z t,k This represents the raw log-odds of the model for token k without bias. The log-odds ratio after adding hot word bias; λ bias >0 represents the bias strength of hot words; II(·) is the indicator function, which takes the value 1 when the condition is true, and 0 otherwise; H t This is a set of tokens that can form hot words under the current text prefix; Then, the weighted probability distribution for the next step is obtained by applying softmax: Official (5): ; Where P(y) t = k | ·) represents the conditional probability that the decoder selects token k at step t under helium-oxygen conditions.

7. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 6, characterized in that, In S5, a technique was added to enhance the multi-rate integration of TTA during testing to slightly scale the rate of the input helium-oxygen speech: During the decoding process, a small speech rate scaling is applied to the speech feature sequence corresponding to the input helium-oxygen speech x to obtain a set of perturbed speech φ. s(x) Where s∈S is the set of velocity coefficients, the same Whisper–LoRA model is called for each velocity coefficient s to obtain the log-likelihood of the candidate transcriber y: Formula (6): log P θ TTA (y | x) = (1 / |S|) · Σ_{s∈S} log P θ (y | φ s(x) ) Where, φ s(x) This is the speech sequence obtained by playing the original helium-oxygen speech at different speeds according to coefficient s. Physically, it can be understood as a simulation of a diver speaking quickly and slowly; |S| represents the number of speed coefficients; log P θ TTA (·) represents the average log-likelihood of candidate transcribs after enhancing TTA integration during testing.

8. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 7, characterized in that, In S5, the candidate reordering technique for lightweight character bigram language models: Using a lightweight character bigram language model trained on reference transcription, each candidate transcription is weighted and sorted based on acoustic log-likelihood, hot word coverage, and language model score to output the optimal transcription. Specifically: We statistically analyze the unary and binary co-occurrence distributions of Chinese characters on reference transcribed texts, construct a lightweight character-binary grammar language model, and apply it to any transcribed text. The logarithmic probability is defined as: Official (7): Among them, c i For the i-th Chinese character; P(c i+1 | c i ) represents the character c in a statistical sense i Followed by the character c i+1 The conditional probability, after smoothing, gives a non-zero probability for unseen combinations; For the nth candidate transcription of helium-oxygen speech Its length is defined as L n Hot word coverage rate , representing the proportion of characters covered by hot words in the transcription, the comprehensive scoring function is: Formula (8): ； Where α is the weight of the acoustic log-likelihood score, reflecting the model's confidence in the speech-text mapping under helium-oxygen conditions; β is the weight of the hot word coverage, emphasizing the importance of correctly recognizing diving command words; γ is the weight of the language model score, measuring the naturalness of the transcribed sequence in terms of Chinese language statistical regularities; Cov_H(·) is the ratio of the number of matched characters to the total number of characters after matching the transcribed sequence with the hot word set. The final output is transcribed to the one with the highest overall score: Official (9): .

9. The Whisper-based speech recognition method for helium-oxygen environments as described in claim 8, characterized in that, In S5, scrolling contextual hints and long speech concatenation technology are used: For long helium-oxygen speech exceeding the single decoding length limit, during the decoding process, the encoded features of the entire long helium-oxygen speech segment are divided into C segments of fixed duration {x}. (1) , x (2) , ..., x (C) }, set a text hint p for the c-th segment. (c) Its decoding target is: Official (10): ; Where p (c) The prompt text at the end of the previous segment physically corresponds to the semantic continuation of the diver's unfinished sentence; After segment c is decoded, it is transcribed from its best value. The last L characters of the middle segment are used as a hint for the next segment: Official (11): ; The final transcription of the entire long helium-oxygen speech was obtained through simple concatenation: Official (12): .

10. A Whisper-based helium-oxygen environment speech recognition system, characterized in that, include: A helium-oxygen speech acquisition module is used to control the pressure inside the shipborne mobile saturation diving chamber to a corresponding preset pressure, fill it with helium-oxygen mixed gas, and collect the speech of multiple divers at different water depths. Each helium-oxygen speech is preprocessed to obtain the preprocessed helium-oxygen speech of each diver at different water depths. The preprocessed helium-oxygen speech is then paired with manually annotated text to construct the first helium-oxygen speech dataset. A feature extraction module is used to convert each preprocessed helium-oxygen speech waveform in the helium-oxygen speech dataset into fixed-dimensional log-Mel spectral features through short-time Fourier transform and Mel filter bank. The second helium-oxygen speech dataset is constructed by the log-Mel spectral features of the preprocessed helium-oxygen speech and the corresponding labeled text. The second helium-oxygen speech dataset is divided into a training set and an evaluation set. A model building module is used to use the Whisper large model as the base model. While keeping the original parameters frozen, a low-rank incremental matrix is ​​introduced only on the query / value projection layer of multi-head attention to form the Whisper-LoRA model. The Whisper-LoRA model consists of an encoder and a decoder. A model training and evaluation module is used to input the training set into the Whisper-LoRA model for model training and to fine-tune the model using LoRA low-rank increment. The encoder encodes each log Mel spectral feature in the training set to obtain the corresponding speech feature sequence. The decoder decodes the obtained speech feature sequence and generates a Chinese text token sequence autoregressively. The loss value is calculated using a loss function. The evaluation set is input into the trained Whisper-LoRA model for model evaluation and CER calculation. When the loss value is lower than a set threshold and the CER meets the preset conditions, the Whisper-LoRA model converges. A model prediction module is used to preprocess newly acquired helium-oxygen speech, convert it into fixed-dimensional logarithmic Mel spectral features through short-time Fourier transform and Mel filter bank, and then input it into a converged Whisper-LoRA model for speech recognition. During the recognition process, the module combines hot word bias to guide the decoding path to prioritize coverage of diving-related terms, multi-rate integration to enhance TTA during testing to slightly scale the speech rate of the input helium-oxygen speech, candidate reordering of a lightweight character binary grammar language model, and scrolling contextual prompts and long speech concatenation techniques to perform feature recognition on the input helium-oxygen speech to generate high-precision text.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in memory to execute the Whisper-based helium-oxygen environment speech recognition method according to any one of claims 1-9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the Whisper-based helium-oxygen environment speech recognition method according to any one of claims 1-9.

Images