Model training and speech recognition method and device, equipment and medium

Abstract

The invention discloses a training method of a voice recognition model based on noise deconstruction, a voice recognition method, a device, equipment and a medium. According to the method, a staged training strategy that a noise unwrapping module is firstly isolated and trained and then a Conformer-Transducer architecture is finely adjusted and trained is adopted, so that high calculation complexity and training difficulty caused by simultaneous training of a plurality of complex modules are avoided. In the isolation training stage, the performance of the noise unwrapping module can be quickly optimized; in the fine tuning training stage, the trained noise unwrapping module is utilized to concentrate on optimizing the Conformer-Transducer architecture, so that the training efficiency is improved, and the training time and the consumption of computing resources are reduced. In the isolation training and fine tuning training process, the parameters of part of modules are frozen, the number of parameters needing to be optimized is reduced, and therefore the calculation complexity is reduced. Noise and pure voice in a voice signal are deconstructed through the noise unwrapping module, and accurate semantic understanding is carried out in combination with a Conformer-Transducer architecture, so that the whole voice recognition model has higher robustness to noise.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and speech recognition, and in particular to a training method for a speech recognition model based on noise deconstruction, and a speech equipment method, apparatus, equipment and medium. Background Art

[0002] With the rapid development of technologies such as artificial intelligence and neural network architectures, the field of automatic speech recognition (ASR) has achieved remarkable results in recent years. Early traditional ASR systems mostly adopted a modular design, covering components such as feature extraction, acoustic modeling, and language models, and relied heavily on hand-designed features and statistical modeling techniques. However, the independent optimization of these modules has inherent limitations, making it difficult for the entire system to achieve optimal performance. With the rise of deep learning, end-to-end speech recognition methods based on neural networks have gradually become mainstream. With a unified network structure, this method achieves a direct mapping from audio input to text output, greatly simplifying the system architecture while significantly improving recognition accuracy. This has promoted the widespread application of speech recognition technology in many scenarios, such as smart homes, in-vehicle systems, and voice assistants.

[0003] End-to-end speech recognition technology further enhances the modeling capabilities of speech features and effectively improves system training efficiency by integrating advanced model structures such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transformer. However, in practical applications, this technology still faces severe challenges in complex noisy environments. Speech signals in real-world scenarios are often subject to various interferences such as environmental noise, reverberation, and overlapping speech from multiple speakers. These interferences seriously affect the acoustic modeling effect of the system, resulting in a significant decline in recognition accuracy. Therefore, improving the robustness of end-to-end speech recognition technology in noisy environments has become a key focus of current research.

[0004] To address this challenge, numerous studies have attempted to incorporate speech enhancement (SE) models as front-end preprocessing modules in automatic speech recognition systems. SE modules can be implemented in a variety of ways, including traditional methods and neural network-based approaches, and can be implemented in both the time and frequency domains. Furthermore, the training methods for SE and ASR modules are flexible, allowing for independent or joint training. However, it should be noted that SE module training objectives typically focus on improving speech quality, such as minimizing the mean squared error (MSE) or enhancing the signal-to-noise ratio (SNR), while ASR models primarily optimize the word error rate (WER). This mismatch in training objectives can easily lead to distortion in the speech processed by the SE module, negatively impacting the overall performance of the ASR system. Existing SE-based methods have significant limitations, and research on improving noise robustness by completely abandoning the SE module and relying solely on end-to-end models remains relatively unexplored and urgently requires further exploration and innovation. Summary of the Invention

[0005] The present application provides a training method for a speech recognition model based on noise deconstruction, as well as a speech recognition method, apparatus, device and medium, which are used to solve the problems that the existing end-to-end ASR technology is insufficiently robust in noisy environments and that the speech enhancement and ASR module objectives do not match.

[0006] In a first aspect, the present application provides a method for training a speech recognition model based on noise decomposition, the method comprising:

[0007] Based on a first mixed speech sample pair in a first mixed data set, a noise disentanglement module in a first speech recognition framework is isolatedly trained to obtain a trained noise disentanglement module and freeze the parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes first clean speech data and first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, wherein the Conformer-Transducer architecture includes a feature encoder, a Conformer block, and a Transducer decoder;

[0008] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0009] Acquire first clean speech data and first reference speech data in any first mixed speech sample pair;

[0010] Obtaining a first clean speech high-dimensional representation of the first clean speech data through a feature encoder and a Conformer block in the first speech recognition framework, and obtaining a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block;

[0011] Extracting a first pure clean representation from the first noisy speech high-dimensional representation by a first encoder in the noise unwrapping module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation by a second encoder in the noise unwrapping module;

[0012] reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module;

[0013] Adjusting parameters in the noise disentanglement module based on the loss between the first pure clean representation and the first high-dimensional representation of the clean speech, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech;

[0014] Based on the second mixed speech sample pairs and their semantic labels in the second mixed data set, fine-tuning the Conformer-Transducer architecture in the second speech recognition framework to obtain a trained target speech recognition framework; wherein the second mixed speech sample pairs include second clean speech data and second reference speech data related to the second clean speech data; the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters and the trained noise disentanglement module with frozen parameters;

[0015] During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed:

[0016] Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair;

[0017] Obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second clean speech high-dimensional representation of the second clean speech data; and obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second noisy speech high-dimensional representation of the second reference speech data;

[0018] Extracting a second pure clean representation from the second noisy speech high-dimensional representation by using the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation by using the second encoder in the trained noise disentanglement module;

[0019] reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through a disentanglement decoder in the trained noise disentanglement module;

[0020] Obtaining predicted semantic information based on the second pure clean identifier through a transducer decoder in the second speech recognition framework;

[0021] The parameters in the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

[0022] In a second aspect, the present application further provides a speech recognition method based on the above-mentioned model, the method comprising:

[0023] Obtaining voice data to be recognized;

[0024] The semantic information of the speech data to be recognized is obtained based on the speech data to be recognized through a pre-trained target speech recognition framework.

[0025] In a third aspect, the present application further provides a method and apparatus for training a speech recognition model based on noise deconstruction, the apparatus comprising:

[0026] a first training unit, configured to perform isolated training on a noise disentanglement module in a first speech recognition framework based on a first mixed speech sample pair in a first mixed data set, so as to obtain a trained noise disentanglement module and freeze parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes first clean speech data and first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, wherein the Conformer-Transducer architecture includes a feature encoder, a Conformer block, and a Transducer decoder;

[0027] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0028] Acquire first clean speech data and first reference speech data in any first mixed speech sample pair;

[0029] Obtaining a first clean speech high-dimensional representation of the first clean speech data through a feature encoder and a Conformer block in the first speech recognition framework, and obtaining a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block; extracting a first pure clean representation from the first noisy speech high-dimensional representation through a first encoder in the noise disentanglement module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation through a second encoder in the noise disentanglement module;

[0030] reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module;

[0031] Adjusting parameters in the noise disentanglement module based on the loss between the first pure clean representation and the first high-dimensional representation of the clean speech, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech;

[0032] A second training unit is configured to fine-tune the Conformer-Transducer architecture in a second speech recognition framework based on a second mixed speech sample pair and its semantic label in a second mixed data set, so as to obtain a trained target speech recognition framework; wherein the second mixed speech sample pair includes second clean speech data and second reference speech data related to the second clean speech data; and the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters and the trained noise disentanglement module with frozen parameters;

[0033] During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed:

[0034] Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair;

[0035] Obtaining a second clean speech high-dimensional representation of the second clean speech data through the feature encoder and Conformer block in the second speech recognition framework; and obtaining a second noisy speech high-dimensional representation of the second reference speech data through the feature encoder and Conformer block in the second speech recognition framework; extracting a second pure clean representation from the second noisy speech high-dimensional representation through the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation through the second encoder in the trained noise disentanglement module;

[0036] reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through a disentanglement decoder in the trained noise disentanglement module;

[0037] Obtaining predicted semantic information based on the second pure clean identifier through a transducer decoder in the second speech recognition framework;

[0038] The parameters in the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

[0039] In a fourth aspect, the present application further provides a speech recognition device based on the above-mentioned model, the device comprising:

[0040] An acquisition module, used to acquire the voice data to be recognized;

[0041] The processing module is used to obtain semantic information of the speech data to be recognized based on the speech data to be recognized through a pre-trained target speech recognition framework.

[0042] In a fifth aspect, the present application provides a computer device comprising a processor, wherein the processor is configured to implement the steps of the training method of the speech recognition model based on noise deconstruction as described above when executing a computer program stored in a memory, or to implement the steps of the speech recognition method as described above.

[0043] In a sixth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training method of the speech recognition model based on noise deconstruction as described above, or implements the steps of the speech recognition method as described above.

[0044] The beneficial effects of this application are as follows:

[0045] 1. When training the noise disentanglement module in isolation, the first encoder extracts a first pure clean representation from the high-dimensional representation of the first noisy speech, and the second encoder extracts the first pure noise representation. The first encoder focuses on capturing clean speech features, while the second encoder focuses on extracting noise features, enabling more detailed analysis of the components of the speech signal. By calculating the loss between the high-dimensional representation of the first clean speech and the first pure clean representation, the extracted clean speech is ensured to be as close as possible to the original clean speech. By calculating the loss between the first high-dimensional acoustic representation and the high-dimensional representation of the first noisy speech, the reconstructed acoustic representation is guaranteed to maintain overall consistency with the original noisy speech. Using these different losses to adjust the parameters of the noise disentanglement module, model learning is guided from different dimensions, effectively improving the noise disentanglement module's ability to separate noise from clean speech. By freezing the parameters of the Conformer-Transducer architecture and training only the noise disentanglement module in isolation, this avoids interference between different modules. When training the noise disentanglement module, there is no need to consider the complex structure and parameter adjustments of the Conformer-Transducer architecture, allowing resources to be focused on optimizing the performance of the noise disentanglement module. Isolation training makes the noise disentanglement module a relatively independent component, facilitating subsequent upgrades and optimizations.

[0046] 2. During fine-tuning of the Conformer-Transducer architecture, the trained noise disentanglement module is used to extract a second pure clean representation from the high-dimensional representation of the second noisy speech. This representation serves as the input to the Transducer decoder. Because the noise disentanglement module already possesses high noise separation capabilities, the extracted pure speech features are of high quality, providing a more accurate source of semantic information for the Transducer decoder. The Conformer-Transducer architecture parameters are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the high-dimensional representation of the second clean speech and the second pure clean representation, and the loss between the second high-dimensional acoustic representation and the high-dimensional representation of the second noisy speech. This multi-loss joint fine-tuning approach comprehensively optimizes model performance across three dimensions: semantic understanding, noise disentanglement, and acoustic representation reconstruction. The Conformer-Transducer architecture inherits the pre-trained parameters and is fine-tuned based on them, working in conjunction with the parameter-frozen noise disentanglement module. The pre-trained parameters provide a good initial state for the model, enabling faster convergence of fine-tuning training. The noise disentanglement module focuses on removing noise and providing clean input for the Conformer-Transducer architecture. The Conformer-Transducer architecture then performs accurate semantic recognition based on the clean input. The two work together to improve the performance of the entire speech recognition framework.

[0047] 3. A phased training strategy is adopted, first isolating and training the noise disentanglement module, then fine-tuning the Conformer-Transducer architecture. This avoids the high computational complexity and training difficulty associated with training multiple complex modules simultaneously. During the isolated training phase, the performance of the noise disentanglement module can be rapidly optimized. During the fine-tuning phase, the trained noise disentanglement module is leveraged to focus on optimizing the Conformer-Transducer architecture. Each phase has clear goals and tasks, which improves training efficiency and reduces training time and computing resources. During the isolated and fine-tuning training processes, the parameters of some modules are frozen, reducing the number of parameters that require optimization and thus computational complexity. For example, when isolating and training the noise disentanglement module, freezing the Conformer-Transducer architecture parameters avoids the need to update and calculate a large number of parameters during training, accelerating training. The noise disentanglement module deconstructs the noise and clean speech in the speech signal. Combined with the Conformer-Transducer architecture for accurate semantic understanding, the entire speech recognition model is more robust to noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram of the training process of a speech recognition model based on noise deconstruction provided in an embodiment of the present application;

[0050] Figure 2 The specific speech recognition framework diagram provided for this application;

[0051] Figure 3 Schematic diagram of three voice representations provided in the embodiment of the present application;

[0052] Figure 4 A schematic diagram of a speech recognition process provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a training device for a speech recognition model based on noise deconstruction provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0055] Figure 7It is a structural diagram of a computer device provided in an optional embodiment of the present application. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0057] In order to effectively solve the noise interference problem in an end-to-end ASR framework, avoid the target mismatch defect of relying on an independent speech enhancement module, and develop noise robustness enhancement technology that does not require additional preprocessing, the present application provides a training method for a speech recognition model based on noise deconstruction, as well as a speech recognition method, device, equipment and medium.

[0058] Example 1:

[0059] This application provides a training method for a speech recognition model based on noise decomposition. Figure 1 A schematic diagram of a training process for a speech recognition model based on noise decomposition provided in an embodiment of the present application, the process comprising:

[0060] S101: Based on a first mixed speech sample pair in a first mixed data set, the noise disentanglement module in the first speech recognition framework is isolated and trained to obtain a trained noise disentanglement module and freeze the parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes a first clean speech data and a first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, and the Conformer-Transducer architecture includes a feature encoder, a Conformer block and a Transducer decoder.

[0061] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0062] S101a: Acquire first clean speech data and first reference speech data in any first mixed speech sample pair.

[0063] S101b: Obtain a first clean speech high-dimensional representation of the first clean speech data through the feature encoder and the Conformer block in the first speech recognition framework, and obtain a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block.

[0064] S101c: extracting a first pure clean representation from the first noisy speech high-dimensional representation through a first encoder in the noise de-wrapping module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation through a second encoder in the noise de-wrapping module.

[0065] S101d: reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module;

[0066] S101e: Adjust parameters in the noise disentanglement module according to the loss between the first pure clean representation and the first clean speech high-dimensional representation, and the loss between the first high-dimensional acoustic representation and the first noisy speech high-dimensional representation.

[0067] In the present application, the training method of the speech recognition model based on noise deconstruction is applied to a computer device, which can be an intelligent terminal, such as a computer, a robot, etc., or a server, such as an application server, a business server, etc.

[0068] In real-world speech recognition scenarios, noise interference severely impacts recognition accuracy. To improve the performance of speech recognition systems in noisy environments, this training method isolates and trains the noise disentanglement module in the first speech recognition framework. The goal is to obtain a trained and parameter-frozen noise disentanglement module that effectively extracts clean speech information from noisy speech, while maintaining the performance advantage of the Conformer-Transducer architecture for clean speech recognition.

[0069] In order to obtain a speech recognition model with efficient noise disentanglement capability, the present application pre-collects a first mixed data set, and based on this, carries out isolated training of the noise disentanglement module in the first speech recognition framework, and finally obtains a trained noise disentanglement module. The first mixed data set serves as the basic data source for training the entire noise disentanglement module, and contains a large number of first mixed speech sample pairs, each of which consists of first clean speech data and first reference speech data related to the first clean speech data. Among them, the first clean speech data is pure and noise-free speech data, which provides an accurate speech information benchmark for training, and the first reference speech data has a certain correlation with the first clean speech data in terms of content, speaker, etc., and it can only be the noisy speech data corresponding to the first clean speech data. For example, by adding specific noise to the first clean speech data, the corresponding relationship between the two ensures that the noise and speech components can be accurately separated during training. Optionally, the type of the first reference speech data can also be clean speech data with the same or different pronunciation content as the first clean speech data. Incorporating this reference speech data during model training prevents the noise disentanglement module from overly suppressing clean speech input, thereby ensuring good ASR performance for both clean and noisy speech. This training design ensures that the entire system achieves a good balance between noise suppression, speech recognition performance, and model complexity.

[0070] The formats of the first clean voice data and the first reference voice data can be common audio formats, such as PCM, WAV, etc., to adapt to the input requirements of the model.

[0071] To ensure that the trained noise disentanglement module possesses good adaptability and generalization capabilities, and is capable of effectively handling speech noise disentanglement tasks in diverse scenarios, the first mixed speech sample pairs in the first mixed dataset should be as comprehensive and diverse as possible. From the perspective of scenario, this can include speech data from quiet environments, such as whispered conversations in a library, as well as speech data from complex noisy environments, such as speech against the background of humming machinery in a factory workshop or against the background of noisy sounds at a construction site. From the perspective of speech type, the first mixed speech sample pairs can include human speech of different genders, ages, and accents, and in a variety of forms, including everyday conversations, speeches, and singing. Furthermore, given the rich and diverse noise types found in real speech, the first reference speech data in the dataset can also be artificially enriched with different types of noise, such as pink noise, brown noise, and burst noise, to simulate realistic and complex speech environments. This allows the noise disentanglement module to learn a wider variety of noise characteristics and improve its disentanglement capabilities in diverse noise scenarios.

[0072] In a possible implementation, the first mixed speech sample pair and its corresponding first clean speech data and first reference speech data may be obtained from one or more of the following sources:

[0073] Real-world scene collection: Use microphones and recording equipment to collect voice data in real-life and work scenarios. For example, place microphones in public places such as shopping malls and train stations to record voice data containing various noise levels as the first reference voice data. Simultaneously, record corresponding pure voice data (which can be nearly pure using noise reduction equipment or post-processing) in the same environment as the first clean voice data. During the collection process, be sure to record relevant environmental and speaker information for subsequent data processing and model training.

[0074] Simulation generation: Utilizing speech synthesis technology and noise generation algorithms, simulated first mixed speech sample pairs are generated. For example, pure speech data can be generated using speech synthesis software as the first clean speech data. Then, noise of varying types and intensities is added to generate the first reference speech data. This method allows for flexible control of speech and noise characteristics, generating a large number of first mixed speech sample pairs that meet specific requirements and enriching the diversity of the dataset.

[0075] Public dataset acquisition: Acquire from public speech data repositories, such as the TIMIT dataset and the Aurora 4 dataset. These public datasets provide a rich variety of speech data, covering different speakers, different language scenarios, and different noise conditions. Select the first mixed speech sample pairs that meet the requirements from the public datasets, and organize and annotate them according to the requirements of this application. For example, for some datasets, the first reference speech data can be generated by adding specific noise to the original clean speech data, thereby constructing the first mixed speech sample pairs.

[0076] For example, the open-source LibriSpeech dataset is used as the source of a clean speech database. Specifically, the train-clean-100 subset of the LibriSpeech dataset is used to simulate the first clean speech data. During the validation phase, the dev-clean and dev-other datasets are used; during the testing phase, the test-clean and test-other datasets are used to accurately evaluate the training results. Noisy speech data can be simulated by adding noise data to the first clean speech data. The noise data can come from the CHiME-4 dataset. The CHiME-4 dataset contains real-world noise recordings captured using a six-channel long-range microphone array and close-range microphones in four different environments: a bus (BUS), a coffee shop (CAF), a pedestrian zone (PED), and a street intersection (STR). To generate simulated noisy speech, single-channel noise data is randomly selected from the CHiME-4 dataset and mixed with the clean LibriSpeech recordings (used as the simulated first clean speech data source). The signal-to-noise ratio (SNR) is randomly selected from a range of -5dB to 15dB. This process generates synthesized noisy speech data, which is paired one-to-one with the first clean speech data to form noise-clean training pairs (train-noisy-100 and train-clean-100), similar to the first mixed speech sample pairs. The generated noisy speech data is then combined and shuffled with the clean LibriSpeech data to form the synthesized data portion used to construct the first mixed dataset. Together with the first clean speech data actually collected or obtained through other means, it constitutes the first mixed dataset.

[0077] Through the above-mentioned data collection, acquisition and processing methods, a high-quality, compliant first mixed data set can be obtained, providing a reliable data basis for the isolation training of the noise disentanglement module, so that the noise disentanglement module can learn rich and diverse speech and noise features in the subsequent training process, thereby improving its noise disentanglement ability and generalization performance.

[0078] Taking into account the noisy environment in the real world, noise usually destroys the effective information carried by the speech, resulting in a significant decrease in ASR performance, especially under low signal-to-noise ratio (SNR) conditions. In order to solve this problem, in this application, the first speech recognition framework includes two key parts. One part is a parameter-adjustable noise disentanglement module, which is responsible for processing noisy speech and extracting clean speech representation and noise representation. It aims to obtain clean representations from noise-corrupted speech at the high-dimensional feature level. Compared with processing at the original speech feature level, it is more robust to noise interference. The other part is a pre-trained and parameter-frozen Conformer-Transducer architecture. The Conformer-Transducer architecture is an advanced speech recognition architecture, which includes a feature encoder, a Conformer block and a Transducer decoder. The feature encoder is used to convert the input speech data into a high-dimensional feature representation. The Conformer block can further extract deep features in the speech data, while the Transducer decoder is responsible for decoding the feature representation into the final speech recognition result. When training the noise disentanglement module, the parameters of the Conformer-Transducer architecture are frozen, that is, they are not adjusted during this training. This maintains its trained speech recognition performance while focusing on training the noise disentanglement module.

[0079] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0080] The first clean speech data and the first reference speech data in any first mixed speech sample pair are obtained from the first mixed dataset. These data serve as inputs for this training and provide a learning basis for the noise disentanglement module.

[0081] The feature encoder and Conformer blocks in the first speech recognition framework process the currently input first clean speech data and first reference speech data, respectively, to obtain a first clean speech high-dimensional representation of the first clean speech data and a first noisy speech high-dimensional representation of the first reference speech data. The feature encoder maps the speech data from the original audio signal space to a high-dimensional feature space, while the Conformer block further encodes the high-dimensional features to extract a more discriminative feature representation. This feature representation incorporates the rich acoustic and linguistic features of the clean speech, serving as an ideal reference for subsequent training. The first clean speech high-dimensional representation reflects the distribution of the first clean speech data in the feature space, while the first noisy speech high-dimensional representation incorporates the speech features of the first reference speech data as well as any noise features.

[0082] The first encoder in the noise disentanglement module extracts a first pure, clean representation from the first high-dimensional representation of the noisy speech. The first encoder separates the pure speech components from the noisy high-dimensional representation of the speech. By learning the inherent structure and patterns in the speech data, it removes noise interference and produces a pure representation of the speech features. Simultaneously, the second encoder in the noise disentanglement module extracts a first pure noise representation from the first high-dimensional representation of the noisy speech. The second encoder focuses on extracting the noise components from the speech data and effectively separates the noise from the speech signal by learning and modeling the noise features.

[0083] The de-entanglement decoder in the noise de-entanglement module reconstructs the first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation. The de-entanglement decoder recombines the separated pure speech representation and noise representation to restore the high-dimensional acoustic features of the original speech signal. This process not only verifies the noise de-entanglement module's effectiveness in separating speech and noise but also provides a reference for subsequent loss calculations.

[0084] It should be noted that the structure of the three key components contained in the noise disentanglement module can be any module or a combination of several modules, such as multi-layer LSTM (Long Short-Term Memory) layers, multiple FFN (Feed-Forward Network) modules, multiple convolution modules, etc., or a combination of these modules. This structural design allows the module to gradually learn complex features and extract the required information while ensuring minimal additional model parameter overhead.

[0085] The parameters in the noise disentanglement module are adjusted based on the loss between the first high-dimensional representation of the clean speech and the first pure clean representation, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech. The loss between the first high-dimensional representation of the clean speech and the first pure clean representation is used to measure the accuracy of the noise disentanglement module in extracting the first pure clean representation from the first high-dimensional representation of the noisy speech. This enforces alignment between the high-dimensional representation of the clean speech derived from the noisy speech and the representation obtained from the pure clean representation. This alignment enables the noise disentanglement module to suppress noise while preserving essential acoustic and linguistic information that is consistent in clean and noisy conditions. As a result, the entire model is guided to more effectively separate noise, resulting in a "cleaner" internal representation, which in turn enhances the performance of the downstream decoder. The closer the extracted first pure clean representation is to the first high-dimensional representation of the clean speech, the better the noise disentanglement module extracts the clean speech components. Common loss functions include mean squared error (MSE) and cross-entropy loss. The difference between the two is calculated to obtain a loss value, which reflects the performance of the noise disentanglement module in extracting the clean speech components. The loss between the first high-dimensional acoustic representation and the first high-dimensional representation of noisy speech is used to evaluate the quality of the noise disentanglement module in reconstructing the first high-dimensional acoustic representation. The more similar the reconstructed first high-dimensional acoustic representation is to the first high-dimensional representation of noisy speech, the better the noise disentanglement module can restore the characteristics of the original speech signal after separating speech and noise. The same loss function such as mean square error can be used for calculation. The loss value reflects the ability of the noise disentanglement module in reconstructing the speech signal. By minimizing the distance between the reconstructed acoustic representation and the extracted high-dimensional acoustic representation, the second encoder is encouraged to extract a pure noise representation that does not contain valid language and semantic content, which in turn indirectly enhances the noise suppression ability of the first encoder.

[0086] By comprehensively considering these two losses, the parameters in the noise disentanglement module are adjusted. For example, a gradient descent algorithm is used to update the parameters according to the loss value, so that the noise disentanglement module can better extract pure speech representation and noise representation in subsequent training, thereby improving the accuracy of noise disentanglement and the quality of reconstructed speech signals.

[0087] The above operation is performed for each first mixed speech sample pair in the first mixed data set. When a preset convergence condition is met, the noise disentanglement module training is completed. The preset convergence condition may be that the sum of the loss values corresponding to all first mixed speech sample pairs in the current iteration reaches a minimum or stabilizes, or that the number of iterations for training the noise disentanglement module reaches a set maximum number of iterations. In specific implementations, these conditions may be flexibly set based on actual circumstances.

[0088] After training is complete, the parameters of the trained noise disentanglement module are frozen, keeping them unchanged during subsequent model applications or other related training processes. This ensures that the noise disentanglement module operates stably in its trained state, preventing performance impacts caused by parameter changes. It also provides a solid foundation for subsequent optimization and application of the entire first speech recognition framework.

[0089] S102: Based on the second mixed speech sample pairs and their semantic labels in the second mixed data set, fine-tune the Conformer-Transducer architecture in the second speech recognition framework to obtain a trained target speech recognition framework; wherein the second mixed speech sample pairs include second clean speech data and second reference speech data related to the second clean speech data; the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters, and the trained noise disentanglement module with frozen parameters.

[0090] During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed:

[0091] S102a: Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair.

[0092] S102b: Obtain a second clean speech high-dimensional representation of the second clean speech data through the feature encoder and Conformer block in the second speech recognition framework; and obtain a second noisy speech high-dimensional representation of the second reference speech data through the feature encoder and Conformer block in the second speech recognition framework.

[0093] S102c: extracting a second pure clean representation from the second noisy speech high-dimensional representation through the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation through the second encoder in the trained noise disentanglement module.

[0094] S102d: Reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through the disentanglement decoder in the trained noise disentanglement module.

[0095] S102e: Obtain predicted semantic information based on the second pure clean identifier through the Transducer decoder in the second speech recognition framework.

[0096] S102f: Fine-tune the parameters in the Conformer-Transducer architecture based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

[0097] In actual speech recognition application scenarios, speech data in different fields and scenarios have their own unique characteristics and semantic patterns. Although the noise disentanglement module was isolated and trained in the early stage through the first mixed data set, so that it has a certain noise disentanglement ability, in order to make the entire speech recognition framework better adapt to new task requirements, especially in scenarios that require accurate understanding of speech semantics, the Conformer-Transducer architecture needs to be fine-tuned and trained. By introducing a second mixed data set containing semantic labels, the Conformer-Transducer architecture can further learn the mapping relationship between speech data and semantics on the basis of the good noise processing ability of the noise disentanglement module, thereby obtaining a trained target speech recognition framework and improving its speech recognition accuracy and semantic understanding ability under specific tasks.

[0098] In the present application, the second mixed data set is specially constructed for fine-tuning the Conformer-Transducer architecture. Among them, the second mixed speech sample pair serves as the core data unit, and each sample pair consists of second clean speech data and second reference speech data related thereto. The second clean speech data is pure and noise-free speech data, which serves as the benchmark speech for semantic annotation and model learning. The second reference speech data is associated with the second clean speech data in terms of content, speaker, etc., and it may only be the noisy speech data corresponding to the second clean speech data. For example, by adding specific noise generation to the second clean speech data, the correspondence between the two ensures that the noise and speech components can be accurately separated during training. Optionally, the type of the second reference speech data can also be clean speech data with the same pronunciation content as the second clean speech data.

[0099] The semantic labels of the second mixed speech sample pair can be obtained by manually annotating the second clean speech data or using automatic annotation tools combined with manual verification. The semantic labels can accurately reflect the semantic content expressed by the second clean speech data. For example, they can be text sentences, keywords, semantic categories, etc. The specific form depends on the actual task requirements. These semantic labels provide supervision information for fine-tuning the Conformer-Transducer architecture, enabling it to learn how to extract semantically relevant features from the speech data.

[0100] It should be noted that the specific method for obtaining the second mixed data set is similar to the method for obtaining the first mixed data set. For details, please refer to the above embodiment and will not be described in detail here.

[0101] In this application, the second speech recognition framework is the core architecture of the entire fine-tuning training, which integrates the parameter-adjustable Conformer-Transducer architecture and the parameter-frozen trained noise disentanglement module. During the fine-tuning training process, the parameters of the parameter-adjustable Conformer-Transducer architecture will be adjusted according to the training objectives. During the fine-tuning training process, the parameters of the noise disentanglement module remain frozen. This is because in the early isolation training, the noise disentanglement module has learned effective noise disentanglement capabilities and can separate pure speech components and noise components from the high-dimensional representation of speech containing noise. Freezing its parameters can avoid damage to its trained capabilities during the fine-tuning training process, while ensuring that the Conformer-Transducer architecture can focus on learning the relationship between speech and semantics, thereby improving the efficiency of fine-tuning training.

[0102] During any iteration of fine-tuning the Conformer-Transducer architecture, the following steps are performed:

[0103] Randomly select any second mixed speech sample pair from the second mixed dataset and obtain the second clean speech data and second reference speech data. Simultaneously, obtain the semantic label corresponding to this second mixed speech sample pair. This data will serve as the input for this iterative fine-tuning training, providing a foundation for model learning.

[0104] In one possible implementation, during the data acquisition process, in order to ensure the integrity and accuracy of the data, the acquired data will be preprocessed, such as unifying the sampling rate and normalizing the voice data, and formatting and cleaning the semantic tags.

[0105] The second clean speech data and the second reference speech data are processed by the feature encoder and the Conformer blocks in the second speech recognition framework. The feature encoder and the Conformer blocks encode and extract features from the second clean speech data to obtain a high-dimensional representation of the second clean speech data. This high-dimensional representation reflects the distribution of the second clean speech data in the feature space and includes characteristic information such as pitch, timbre, and intonation.

[0106] The second reference speech data is also processed through the feature encoder and Conformer blocks to obtain a high-dimensional representation of the second noisy speech. Unlike the high-dimensional representation of the second clean speech, this high-dimensional representation includes not only speech features but also possible noise features, providing more complex input conditions for subsequent noise disentanglement and semantic extraction.

[0107] In the present application, the noise disentanglement module performs a comprehensive disentanglement operation to separate the high-dimensional noise representation into different noise and clean speech components, effectively minimizing noise interference and retaining semantic information in the clean speech representation. The enhanced acoustic representation is then input into the Transducer decoder to generate a transcription. Exemplarily, the first encoder in the noise disentanglement module that has completed the training extracts a second pure clean representation from the second noisy speech high-dimensional representation. The first encoder uses its learned noise disentanglement ability to separate pure speech components from the high-dimensional representation of speech containing noise. This process removes noise interference by learning the intrinsic structure and pattern in the speech data, obtains a pure speech feature representation, and provides high-quality speech feature input for subsequent semantic extraction.

[0108] Simultaneously, the second encoder in the trained noise disentanglement module extracts a second pure noise representation from the second high-dimensional representation of the noisy speech. The second encoder focuses on extracting the noise component in the speech data and effectively separates the noise from the speech signal by learning and modeling the noise characteristics. The extracted second pure noise representation can be used for subsequent noise analysis and processing, and also helps further optimize the performance of the noise disentanglement module.

[0109] The disentanglement decoder in the noise disentanglement module that has completed the training reconstructs the second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation. The disentanglement decoder recombines the separated pure speech representation and noise representation to restore the high-dimensional acoustic features of the original speech signal. This process not only verifies the noise disentanglement module's effect on separating speech and noise, but also provides a reference for subsequent loss calculations. The reconstructed second high-dimensional acoustic representation should be as close as possible to the original second noisy speech high-dimensional representation, which reflects the overall performance of the noise disentanglement module.

[0110] The Transducer decoder in the second speech recognition framework obtains predicted semantic information based on the second pure clean representation. The Transducer decoder receives the second pure clean representation processed by the Conformer block and the noise disentanglement module and uses its decoding capabilities to convert these features into semantic information. The predicted semantic information is an estimate of the semantics expressed by the input speech data, in the form of semantic labels, such as textual sentences or keywords.

[0111] The parameters of the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second high-dimensional representation of clean speech, and the loss between the second high-dimensional acoustic representation and the second high-dimensional representation of noisy speech. The loss between the predicted semantic information and the semantic label is used to measure the difference between the predicted semantic information output by the Transducer decoder and the true semantic label. Common loss functions include Connectionist Temporal Classification (CTC) loss and RNN-T (Recurrent Neural Network-Transducer) loss. By calculating the difference between the two, a loss value is obtained, which reflects the model's performance in semantic understanding. The smaller the loss value, the more accurate the model's understanding of speech semantics. The loss between the second high-dimensional representation of clean speech and the second pure clean representation is used to evaluate the accuracy of the noise disentanglement module in extracting the second pure clean representation from the second high-dimensional representation of noisy speech. The closer the extracted second pure clean representation is to the second high-dimensional representation of clean speech, the better the noise disentanglement module is at extracting the clean speech components. During fine-tuning training, although the parameters of the noise disentanglement module are frozen, the loss still serves as an important reference indicator to help the Conformer-Transducer architecture better understand the process and results of noise disentanglement. Commonly used loss functions include mean square error (MSE), etc. By calculating the difference between the two, a loss value is obtained, which reflects the performance of the noise disentanglement module in extracting pure speech components. The loss between the second high-dimensional acoustic representation and the second high-dimensional representation of noisy speech is used to evaluate the quality of the noise disentanglement module in reconstructing the second high-dimensional acoustic representation. If the reconstructed second high-dimensional acoustic representation is more similar to the second high-dimensional representation of noisy speech, it means that the noise disentanglement module can better restore the characteristics of the original speech signal after separating speech and noise. Loss functions such as mean square error can also be used for calculation. The loss value reflects the ability of the noise disentanglement module in reconstructing speech signals.

[0112] In one possible implementation, fine-tuning parameters in the Conformer-Transducer architecture based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation includes:

[0113] Determine a joint loss based on the loss between the predicted semantic information and the semantic label and the weight value thereof, the loss between the second pure clean representation and the second clean speech high-dimensional representation and the weight value thereof, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation and the weight value thereof;

[0114] Based on the joint loss, the parameters in the Conformer-Transducer architecture are fine-tuned.

[0115] In actual fine-tuning training, different losses may contribute differently to the improvement of model performance. Therefore, a weight value is assigned to different losses, and the size of the weight value can be set according to factors such as actual task requirements, data characteristics, and the early performance of the model. For example, if semantic understanding occupies a more important position in this fine-tuning training task, then the weight value of the loss can be appropriately increased. If the model is expected to perform better in noise disentanglement to assist semantic understanding, then the weight of the loss can be appropriately increased. Of course, the weight values corresponding to different losses can also be used as a hyperparameter and adjusted along with the training process, which is not specifically limited here. According to the above three losses and their corresponding weight values, the joint loss is determined by weighted summation. Among them, the joint loss function can be expressed as:

[0116] L NoisyD-CT =γL Conformer-T +αL CON +βL R

[0117] Among them, L NoisyD-CT represents the joint loss function, L Conformer-T represents the loss between the predicted semantic information and the semantic label, and γ represents L Conformer-T Weight values, such as 1, 0.9, etc., L CON represents the loss between the second clean speech high-dimensional representation and the second pure clean representation, α represents L CON The weight value, such as 0.3, 0.5, etc., L R represents the loss between the second high-dimensional acoustic representation and the second high-dimensional representation of noisy speech, and β represents L R The weight value is 1, 1.2, etc.

[0118] By using weights, we can combine the losses from different aspects into a comprehensive measure of model performance. The calculated joint loss can then be used to update the parameters in the Conformer-Transducer architecture.

[0119] Taking these three losses into account, we employ optimization methods such as gradient descent to update the parameters of the Conformer-Transducer architecture based on the loss values. Through continuous iterative fine-tuning training, the Conformer-Transducer architecture is better adapted to the characteristics and semantic patterns of the second mixed dataset, improving its performance in speech recognition and semantic understanding.

[0120] The above operation is performed on each second mixed speech sample pair in the second mixed data set, and the fine-tuning training is completed when the preset convergence condition is met. Among them, satisfying the preset convergence condition can be that the sum of the loss values corresponding to all the second mixed speech sample pairs in the current iteration process reaches a minimum value or tends to be stable, or the number of iterations of fine-tuning training for the Conformer-Transducer architecture reaches the set maximum number of iterations, etc. In specific implementation, it can be flexibly set according to actual conditions. For example, a threshold can be set. When the change of the sum of the loss values for multiple consecutive iteration cycles is less than the threshold, it is considered that the training has reached a convergence state; or a maximum number of iterations can be set according to actual needs and computing resource limitations, and the training is stopped when the number is reached.

[0121] In one possible implementation, the trained target speech recognition framework primarily consists of a fine-tuned Conformer-Transducer architecture and the first encoder from the trained noise disentanglement module. The connections between these components are tight and organized, working together to achieve efficient and accurate speech recognition and semantic understanding. The feature encoder, as the front-end processing module of the entire framework, undertakes the important task of mapping the raw speech signal from the low-dimensional audio waveform space to the high-dimensional feature space. Through a series of carefully designed filter banks and feature extraction algorithms, it captures the fundamental acoustic features of the speech signal, such as frequency and energy distribution, providing a rich and representative feature representation for subsequent processing. The Conformer block follows closely behind, receiving the high-dimensional features output by the feature encoder and further mining the deep semantic and pattern information within these features. Combining the local feature extraction capabilities of a convolutional neural network (CNN) with the global context modeling capabilities of a Transformer, the Conformer block can effectively capture temporal dependencies and complex speech patterns in speech, such as the combination of different phonemes and changes in speech prosody. The two are connected via a direct data flow, with the feature vectors processed by the feature encoder being sequentially input into the Conformer block on a time-step basis. This seamless connection ensures the continuity and integrity of speech features during processing, enabling the Conformer block to fully leverage the basic features extracted by the feature encoder for deeper feature analysis and processing, providing high-quality feature input for subsequent noise disentanglement and semantic decoding. The high-dimensional speech features processed by the Conformer block contain both pure speech components and possible noise components in the speech signal. As the core component of the noise disentanglement module, the first encoder's primary function is to accurately separate pure speech features from these mixed features. By connecting to the Conformer block, the first encoder directly accesses speech representations obtained through preliminary feature extraction and pattern analysis. These representations already contain rich speech information, helping the first encoder to more accurately identify and separate noise from pure speech. The output features of the Conformer block serve as the input to the first encoder, and the connection between the two ensures efficient transfer of feature information. During this feature transfer process, some dimensionality adjustments or preprocessing may be performed to meet the input requirements of the first encoder. This connection enables the first encoder to perform noise disentanglement based on the high-level features extracted by the Conformer block, improving the accuracy and efficiency of noise disentanglement. The high-dimensional representation of the clean speech output by the first encoder after noise disentanglement is a high-quality speech feature after removing noise interference.As the final output module of the framework, the Transducer decoder is responsible for decoding these pure speech features into semantic information that can be understood by humans, such as text-based sentences and keywords. By connecting to the first encoder, the Transducer decoder can directly obtain pure speech features, avoiding the interference of noise on the semantic decoding process, thereby improving the accuracy of semantic decoding. The output features of the first encoder are input into the Transducer decoder in a specific format and timing. The Transducer decoder uses its internal self-attention mechanism and feedforward neural network structures to gradually decode the input features and gradually generate the final semantic output. This connection method ensures a smooth conversion from pure speech features to semantic information, enabling the entire target speech recognition framework to efficiently complete the speech recognition task.

[0122] The trained target speech recognition framework can be applied to a variety of scenarios requiring speech recognition and semantic understanding, such as intelligent voice assistants, voice conference recording, and voice navigation. In practical applications, the target speech recognition framework can receive noisy speech input, remove noise interference using the trained noise disentanglement module, and then convert the clean speech signal into accurate semantic information through the Conformer-Transducer architecture, providing users with high-quality voice services. Furthermore, because the target speech recognition framework fully considers different scenarios and semantic patterns during fine-tuning training, it has excellent generalization and adaptability, capable of handling various complex speech environments.

[0123] The beneficial effects of this application are as follows:

[0124] 1. When training the noise disentanglement module in isolation, the first encoder extracts a first pure clean representation from the high-dimensional representation of the first noisy speech, and the second encoder extracts the first pure noise representation. The first encoder focuses on capturing clean speech features, while the second encoder focuses on extracting noise features, enabling more detailed analysis of the components of the speech signal. By calculating the loss between the high-dimensional representation of the first clean speech and the first pure clean representation, the extracted clean speech is ensured to be as close as possible to the original clean speech. By calculating the loss between the first high-dimensional acoustic representation and the high-dimensional representation of the first noisy speech, the reconstructed acoustic representation is guaranteed to maintain overall consistency with the original noisy speech. Using these different losses to adjust the parameters of the noise disentanglement module, model learning is guided from different dimensions, effectively improving the noise disentanglement module's ability to separate noise from clean speech. By freezing the parameters of the Conformer-Transducer architecture and training only the noise disentanglement module in isolation, this avoids interference between different modules. When training the noise disentanglement module, there is no need to consider the complex structure and parameter adjustments of the Conformer-Transducer architecture, allowing resources to be focused on optimizing the performance of the noise disentanglement module. Isolation training makes the noise disentanglement module a relatively independent component, facilitating subsequent upgrades and optimizations.

[0125] 2. During fine-tuning of the Conformer-Transducer architecture, the trained noise disentanglement module is used to extract a second pure clean representation from the high-dimensional representation of the second noisy speech. This representation serves as the input to the Transducer decoder. Because the noise disentanglement module already possesses high noise separation capabilities, the extracted pure speech features are of high quality, providing a more accurate source of semantic information for the Transducer decoder. The Conformer-Transducer architecture parameters are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the high-dimensional representation of the second clean speech and the second pure clean representation, and the loss between the second high-dimensional acoustic representation and the high-dimensional representation of the second noisy speech. This multi-loss joint fine-tuning approach comprehensively optimizes model performance across three dimensions: semantic understanding, noise disentanglement, and acoustic representation reconstruction. The Conformer-Transducer architecture inherits the pre-trained parameters and is fine-tuned based on them, working in conjunction with the parameter-frozen noise disentanglement module. The pre-trained parameters provide a good initial state for the model, enabling faster convergence of fine-tuning training. The noise disentanglement module focuses on removing noise and providing clean input for the Conformer-Transducer architecture. The Conformer-Transducer architecture then performs accurate semantic recognition based on the clean input. The two work together to improve the performance of the entire speech recognition framework.

[0126] 3. A phased training strategy is adopted, first isolating and training the noise disentanglement module, then fine-tuning the Conformer-Transducer architecture. This avoids the high computational complexity and training difficulty associated with training multiple complex modules simultaneously. During the isolated training phase, the performance of the noise disentanglement module can be rapidly optimized. During the fine-tuning phase, the trained noise disentanglement module is leveraged to focus on optimizing the Conformer-Transducer architecture. Each phase has clear goals and tasks, which improves training efficiency and reduces training time and computing resources. During the isolated and fine-tuning training processes, the parameters of some modules are frozen, reducing the number of parameters that require optimization and thus computational complexity. For example, when isolating and training the noise disentanglement module, freezing the Conformer-Transducer architecture parameters avoids the need to update and calculate a large number of parameters during training, accelerating training. The noise disentanglement module deconstructs the noise and clean speech in the speech signal. Combined with the Conformer-Transducer architecture for accurate semantic understanding, the entire speech recognition model is more robust to noise.

[0127] Example 2:

[0128] In order to obtain a Conformer-Transducer architecture that can accurately recognize semantics, based on the above embodiment, in this application, the Conformer-Transducer architecture with frozen parameters after pre-training is obtained by the following method:

[0129] Based on the third clean speech data and its semantic labels in the third dataset, the original Conformer-Transducer architecture is iteratively trained to obtain a Conformer-Transducer architecture with frozen parameters after pre-training.

[0130] In the process of building high-performance speech recognition systems, the Conformer-Transducer architecture, as an advanced end-to-end speech recognition model, has demonstrated excellent performance. However, directly using the untrained original Conformer-Transducer architecture often fails to achieve ideal recognition results in practical applications. Therefore, it is necessary to pre-train it so that it can learn the common features and semantic patterns in the speech data, thereby laying a solid foundation for the subsequent fine-tuning training combined with the noise disentanglement module. For example, the original Conformer-Transducer architecture is iteratively trained based on the third clean speech data and its semantic labels in the third dataset.

[0131] In this application, the third dataset is specially constructed for pre-training the original Conformer-Transducer architecture. Among them, the third clean speech data is the core component of the third dataset. These speech data are recorded in a relatively quiet environment without noise interference, ensuring the purity and high quality of the speech signal. The third clean speech data comes from a wide range of sources, covering a variety of speech types and scenarios, such as human speech of different age groups, different genders, different accents, and speech with different speaking speeds and tones. This diversity helps the model learn a wider range of speech features and improve its generalization ability. The semantic labels of the third clean speech data are obtained by combining manual annotation and automatic annotation. Manual annotation is carried out by professional annotators to perform semantic understanding of the speech data sentence by sentence or word by word, and mark the accurate text content. Automatic annotation uses some advanced speech recognition algorithms and tools to perform preliminary semantic conversion on the speech data, which is then manually verified and corrected. In this way, the accuracy and reliability of the semantic labels are ensured, providing high-quality supervision information for model training.

[0132] For example, a complete vocabulary in a semantic tag can consist of 5,000 Byte Pair Encoding (BPE) units. The BPE encoding method can effectively handle out-of-vocabulary words, breaking them down into smaller sub-word units, improving the model's ability to handle unseen words.

[0133] The original Conformer-Transducer architecture forms the foundation of models that require pre-training. It consists of a feature encoder, a Conformer block, and a Transducer decoder. At the beginning of each training iteration, a batch of third-party clean speech data and its corresponding semantic labels are randomly loaded from the third dataset. This random loading ensures that the model is exposed to a diverse set of samples during training, preventing overfitting.

[0134] In one example, during training, the speech data input to the speech recognition framework is acoustic features, such as 80-dimensional log-mel fbank features. This feature effectively captures the spectral information in the speech signal and provides a rich acoustic representation for the model.

[0135] In one possible implementation, during training, to enhance the model's adaptability to different speech variants, the SpecAugment method can be used to enhance the input acoustic features. SpecAugment simulates the various variations that may occur in real speech by randomly masking or distorting the acoustic features in both the time and frequency domains, enabling the model to learn more robust feature representations.

[0136] In one example, the loaded speech data can be preprocessed. First, the speech signal sampling rate is unified to ensure that all speech data has a consistent sampling rate, facilitating subsequent feature extraction and processing. Next, normalization is performed to adjust the amplitude of the speech signal to an appropriate range, improving model training stability. Furthermore, the speech data can be framed and windowed as needed to further extract time-domain and frequency-domain features.

[0137] The acquired third clean speech data is input into the feature encoder of the original Conformer-Transducer architecture. The feature encoder processes the input speech signal, extracts the basic acoustic features of the speech, and maps them into a high-dimensional feature space to obtain an initial high-dimensional feature representation. For example, a stack of 15 Conformer encoder layers can be used for the feature encoder in the Conformer-Transducer architecture. Each Conformer layer consists of a 1024-dimensional feedforward module and a 256-dimensional attention module with four self-attention heads. This architecture combines the local feature extraction capabilities of a convolutional neural network (CNN) with the global context modeling capabilities of a Transformer. The feedforward module performs nonlinear transformations on features, enhancing their expressiveness; the attention module uses a multi-head self-attention mechanism to capture long-range dependencies between features, enabling the model to better understand the semantic information in the speech. The initial high-dimensional feature representation is then input into the Conformer block, which further encodes and extracts the features to obtain a more semantically representative high-dimensional feature representation that captures the temporal dependencies and complex speech patterns in speech. For example, the Conformer block can learn the combination patterns between different phonemes and changes in speech prosody, thereby extracting more semantically representative features. The high-dimensional feature representation output by the Conformer block is input into the Transducer decoder. Based on these features, the Transducer decoder gradually generates predicted semantic information, such as textual sentences and keywords. For example, the Transducer decoder can use an RNN-T framework composed of a single-layer 256-dimensional LSTM, with a total model size of 30.53MB. The LSTM structure can effectively handle temporal dependencies in sequence data. By controlling the flow of information through a gating mechanism, the Transducer decoder can dynamically adjust the generation of subsequent semantics based on the currently generated partial semantic information and input features, thereby improving the accuracy of semantic prediction.

[0138] The loss between the predicted semantic information and the true semantic label is calculated. For example, the Connectionist Temporal Classification (CTC) loss and the RNN-T (Recurrent Neural Network-Transducer) loss are used. Based on the calculated loss value, the backpropagation algorithm and optimizer (such as the Stochastic Gradient Descent (SGD) optimizer and the Adam optimizer) are used to update the parameters in the original Conformer-Transducer architecture. By continuously iteratively updating the parameters, the model can gradually learn the characteristics and semantic patterns in the speech data, improving its speech recognition and semantic understanding capabilities.

[0139] After each training iteration, the model is judged to see if it meets the preset convergence criteria. Convergence criteria can be set as the loss value reaching a minimum or stabilizing during the current iteration, for example, the change in the loss value for multiple consecutive iterations is less than a set threshold; or the number of iterations reaches a preset maximum number of iterations.

[0140] When the model meets the convergence criteria, the original Conformer-Transducer architecture is considered to have completed pre-training. At this point, the parameters of the Conformer-Transducer architecture are frozen so that they do not change during subsequent training and are used to generate high-dimensional clean representations for each corresponding noisy speech representation target. This high-level representation alignment between noisy and clean speech provides powerful noise suppression guidance for the proposed noise disentanglement module (NoisyD). The purpose of parameter freezing is to preserve the general features and semantic patterns learned by the model during pre-training, avoiding over-adjustment or corruption in subsequent task-specific training, thereby ensuring stable performance of the model in specific tasks.

[0141] After parameter freezing, the Conformer-Transducer architecture retains a large number of common speech features and semantic patterns learned during pre-training. These common features provide strong support for subsequent task-specific training, enabling the model to adapt and learn more quickly when faced with new tasks. Furthermore, since the parameters are frozen, they do not need to be updated during subsequent task-specific training, reducing the computational and time costs of training. This also avoids over-tuning of parameters, helping to improve model training efficiency and stability, preventing any noise-related gradient updates from propagating to the pre-trained Conformer-Transducer architecture. This phase preserves the learned acoustic and linguistic representations. The large-scale, third-clean speech data used during pre-training provides the model with strong generalization capabilities. This generalization capability is preserved after parameter freezing, enabling the model to perform well on speech data from diverse sources and styles.

[0142] Example 3:

[0143] Figure 2 This is a specific speech recognition framework diagram provided for this application. Figure 2As shown, the original Conformer-Transducer architecture is iteratively trained based on the third clean speech data and its semantic labels from the third dataset to obtain a Conformer-Transducer architecture with fixed parameters after training. This Conformer-Transducer architecture comprises a feature encoder, a Conformer block, and a Transducer decoder. For example, a stack of 15 Conformer encoder layers can be used for the feature encoder in the Conformer-Transducer architecture. Each Conformer layer consists of a 1024-dimensional feedforward module and a 256-dimensional attention module with four self-attention heads. This architecture combines the local feature extraction capabilities of a convolutional neural network (CNN) with the global context modeling capabilities of a Transformer. The feedforward module performs nonlinear transformations on features, enhancing their expressiveness; the attention module uses a multi-head self-attention mechanism to capture long-range dependencies between features, enabling the model to better understand the semantic information in speech. The initial high-dimensional feature representation is then input into the Conformer block, which further encodes and extracts the features to obtain a more semantically representative high-dimensional feature representation that captures the temporal dependencies and complex speech patterns in speech. For example, the Conformer block can learn the combination patterns between different phonemes and changes in speech prosody, thereby extracting more semantically representative features. The high-dimensional feature representation output by the Conformer block is input into the Transducer decoder. Based on these features, the Transducer decoder gradually generates predicted semantic information, such as textual sentences and keywords. For example, the Transducer decoder can use an RNN-T framework composed of a single-layer 256-dimensional LSTM, with a total model size of 30.53MB. The LSTM structure can effectively handle temporal dependencies in sequence data. By controlling the flow of information through a gating mechanism, the Transducer decoder can dynamically adjust the generation of subsequent semantics based on the currently generated partial semantic information and input features, thereby improving the accuracy of semantic prediction.

[0144] After obtaining a trained Conformer-Transducer architecture, its parameters are frozen so that they remain unchanged during subsequent training. These parameters are then used to generate high-dimensional clean representations for each corresponding noisy speech representation target. To improve the performance of speech recognition systems in noisy environments, the noise disentanglement module in the first speech recognition framework is trained in isolation. The goal is to obtain a trained, parameter-frozen noise disentanglement module that effectively extracts clean speech information from noisy speech while maintaining the performance advantage of the Conformer-Transducer architecture for clean speech recognition. The first speech recognition framework consists of two key components. One is a parameter-adjustable noise disentanglement module that processes noisy speech to extract clean and noisy representations. This module aims to obtain clean representations from noise-corrupted speech at the high-dimensional feature level, making it more robust to noise interference than processing at the raw speech feature level. The other is a pre-trained, parameter-frozen Conformer-Transducer architecture.

[0145] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0146] The first clean speech data and first reference speech data from any first mixed speech sample pair are obtained from the first mixed dataset. These data serve as input for this training and provide a learning foundation for the noise disentanglement module. The feature encoder and Conformer blocks in the first speech recognition framework process the first clean speech data and the first reference speech data, respectively, to obtain a first clean speech high-dimensional representation of the first clean speech data and a first noisy speech high-dimensional representation of the first reference speech data. The feature encoder maps the speech data from the original audio signal space to a high-dimensional feature space, while the Conformer block further encodes the high-dimensional features to extract a more discriminative feature representation. This feature representation incorporates the rich acoustic and linguistic features of the clean speech, serving as an ideal reference for subsequent training. The first clean speech high-dimensional representation reflects the distribution of the first clean speech data in the feature space, while the first noisy speech high-dimensional representation incorporates the speech features of the first reference speech data as well as any noise features. The first encoder in the noise disentanglement module extracts the first pure clean representation from the first noisy speech high-dimensional representation. The first encoder separates the pure speech component from the noisy high-dimensional representation of speech. It learns the intrinsic structure and patterns in the speech data, removes noise interference, and obtains a pure speech feature representation. Simultaneously, the second encoder in the noise disentanglement module extracts a first pure noise representation from the first noisy high-dimensional representation of speech. The second encoder focuses on extracting the noise component from the speech data and effectively separates the noise from the speech signal by learning and modeling the noise features. The disentanglement decoder in the noise disentanglement module reconstructs the first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation. The disentanglement decoder recombines the separated pure speech representation and noise representation to restore the high-dimensional acoustic features of the original speech signal. This process not only verifies the effectiveness of the noise disentanglement module in separating speech and noise but also provides a reference for subsequent loss calculations.

[0147] It should be noted that the structure of the three key components contained in the noise disentanglement module can be any module or a combination of several modules, such as multi-layer LSTM layers, multiple FFN modules, multiple convolution modules, etc., or a combination of these modules. This structural design allows the module to gradually learn complex features and extract the required information while ensuring minimal additional model parameter overhead.

[0148] According to the loss between the first clean speech high-dimensional representation and the first pure clean representation (L CON ), and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of noisy speech (L R), adjust the parameters in the noise disentanglement module. Among them, the loss between the first clean speech high-dimensional representation and the first pure clean representation is used to measure the accuracy of the noise disentanglement module in extracting the first pure clean representation from the first noisy speech high-dimensional representation, forcing the pure clean representation derived from the noisy speech to be aligned with the clean speech high-dimensional representation. This alignment enables the noise disentanglement module to suppress noise while retaining the basic acoustic and language information that is consistent under clean and noisy conditions. Therefore, the entire model is guided to separate noise more effectively, thereby producing a "cleaner" internal representation, thereby enhancing the performance of the downstream sensor decoder. The loss between the first high-dimensional acoustic representation and the first noisy speech high-dimensional representation is used to evaluate the quality of the first high-dimensional acoustic representation reconstructed by the noise disentanglement module. By minimizing the distance between the reconstructed acoustic representation and the extracted high-dimensional acoustic representation, the second encoder is promoted to extract a pure noise representation that does not contain effective language and semantic content, which in turn indirectly enhances the noise suppression ability of the first encoder.

[0149] After training is complete, the parameters of the trained noise disentanglement module are frozen, keeping them unchanged during subsequent model applications or other related training processes. This ensures that the noise disentanglement module operates stably in its trained state, preventing performance impacts caused by parameter changes. It also provides a solid foundation for subsequent optimization and application of the entire first speech recognition framework.

[0150] To better adapt the entire speech recognition framework to new task requirements, particularly in scenarios requiring accurate semantic understanding of speech, the Conformer-Transducer architecture requires fine-tuning training. By introducing a second mixed dataset containing semantic labels, the Conformer-Transducer architecture, building on the noise disentanglement module's already robust noise processing capabilities, further learns the mapping between speech data and semantics. This results in a trained target speech recognition framework, improving both speech recognition accuracy and semantic understanding for specific tasks. This second speech recognition framework, the core architecture for fine-tuning training, integrates the parameter-adjustable Conformer-Transducer architecture and the parameter-frozen trained noise disentanglement module. During fine-tuning training, the parameters of the parameter-adjustable Conformer-Transducer architecture are adjusted based on the training objectives. The parameters of the noise disentanglement module remain frozen during fine-tuning because, during initial isolation training, the noise disentanglement module has already learned effective noise disentanglement capabilities, enabling it to separate pure speech and noise components from the noisy high-dimensional representation of speech. Freezing its parameters can avoid damaging its trained capabilities during fine-tuning training, while ensuring that the Conformer-Transducer architecture can focus on learning the relationship between speech and semantics, improving the efficiency of fine-tuning training.

[0151] During any iteration of fine-tuning the Conformer-Transducer architecture, the following steps are performed:

[0152] Randomly select any second mixed speech sample pair from the second mixed dataset and obtain the second clean speech data and second reference speech data. Simultaneously, obtain the semantic label corresponding to this second mixed speech sample pair. This data will serve as the input for this iterative fine-tuning training, providing a foundation for model learning.

[0153] In one possible implementation, during the data acquisition process, in order to ensure the integrity and accuracy of the data, the acquired data will be preprocessed, such as unifying the sampling rate and normalizing the voice data, and formatting and cleaning the semantic tags.

[0154] The second clean speech data and the second reference speech data are processed by the feature encoder and the Conformer blocks in the second speech recognition framework. The feature encoder and the Conformer blocks encode and extract features from the second clean speech data to obtain a high-dimensional representation of the second clean speech data. This high-dimensional representation reflects the distribution of the second clean speech data in the feature space and includes characteristic information such as pitch, timbre, and intonation.

[0155] The second reference speech data is also processed through the feature encoder and Conformer blocks to obtain a high-dimensional representation of the second noisy speech. Unlike the high-dimensional representation of the second clean speech, this high-dimensional representation includes not only speech features but also possible noise features, providing more complex input conditions for subsequent noise disentanglement and semantic extraction.

[0156] In the present application, the noise disentanglement module performs a comprehensive disentanglement operation to separate the high-dimensional noise representation into different noise and clean speech components, effectively minimizing noise interference and retaining semantic information in the clean speech representation. The enhanced acoustic representation is then input into the Transducer decoder to generate a transcription. Exemplarily, the first encoder in the noise disentanglement module that has completed the training extracts a second pure clean representation from the second noisy speech high-dimensional representation. The first encoder uses its learned noise disentanglement ability to separate pure speech components from the high-dimensional representation of speech containing noise. This process removes noise interference by learning the intrinsic structure and pattern in the speech data, obtains a pure speech feature representation, and provides high-quality speech feature input for subsequent semantic extraction.

[0157] Simultaneously, the second encoder in the trained noise disentanglement module extracts a second pure noise representation from the second high-dimensional representation of the noisy speech. The second encoder focuses on extracting the noise component in the speech data and effectively separates the noise from the speech signal by learning and modeling the noise characteristics. The extracted second pure noise representation can be used for subsequent noise analysis and processing, and also helps further optimize the performance of the noise disentanglement module.

[0158] The disentanglement decoder in the noise disentanglement module that has completed the training reconstructs the second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation. The disentanglement decoder recombines the separated pure speech representation and noise representation to restore the high-dimensional acoustic features of the original speech signal. This process not only verifies the noise disentanglement module's effect on separating speech and noise, but also provides a reference for subsequent loss calculations. The reconstructed second high-dimensional acoustic representation should be as close as possible to the original second noisy speech high-dimensional representation, which reflects the overall performance of the noise disentanglement module.

[0159] The Transducer decoder in the second speech recognition framework obtains predicted semantic information based on the second pure clean representation. The Transducer decoder receives the second pure clean representation processed by the Conformer block and the noise disentanglement module and uses its decoding capabilities to convert these features into semantic information. The predicted semantic information is an estimate of the semantics expressed by the input speech data, in the form of semantic labels, such as textual sentences or keywords.

[0160] According to the loss between the predicted semantic information and the semantic label (including L RNN-T and L CTC ), the loss between the second pure clean representation and the second clean speech high-dimensional representation (L CON ), and the loss between the second high-dimensional acoustic representation and the second high-dimensional representation of noisy speech (L R ), fine-tune the parameters in the Conformer-Transducer architecture.

[0161] The training method of the noise deconstruction-based speech recognition model provided in this application is superior to the traditional Conformer-Transducer architecture that does not use this method in terms of the word error rate (WER) for evaluating speech recognition performance. Compared with the traditional Conformer-Transducer architecture, the speech recognition framework and training method proposed in this application have significantly improved both the simulated noise test set and the real noise test set while ensuring performance on the clean test set. The word error rate results of the target speech recognition framework proposed in this application (denoted as NoisyD-CT in the table below) and the traditional Conformer-Transducer (denoted as Conformer-T in the table below) architecture on the LibriSpeech test set and the noise test set under different SNR conditions are shown in the table below:

[0162]

[0163] The table above uses the open-source LibriSpeech dataset as the source of the clean speech database. During the validation (Dev) phase, the Dev-Clean and Dev-Other datasets were obtained from the LibriSpeech test set. During the testing phase, the Test-Clean and Test-Other datasets were obtained from the LibriSpeech test set. To generate simulated noisy speech, single-channel noise data was randomly selected from the CHiME-4 dataset as the noisy test set and mixed with clean speech data from LibriSpeech. The signal-to-noise ratio (SNR) was randomly selected from a range of -5dB to 15dB. This table uses SNRs of -5, 0, 5, 10, and 15 as examples. By analyzing the evaluation results of the NoisyD-CT framework proposed in this application and the traditional Conformer-T architecture on the LibriSpeech test set and noisy test sets with different SNR conditions, it can be seen that whether on the validation set and test set of LibriSpeech or on the noisy test sets with different SNR levels, NoisyD-CT outperforms the traditional Conformer-T architecture under all test conditions, confirming the advantages of combining noise disentanglement methods and training strategies. In particular, the NoisyD-CT framework achieves greater improvements under higher SNR and relatively clean conditions. For example, the word error rate at low SNR levels is reduced from 43.5% to 41.8%, while under cleaner conditions, the word error rate is reduced from 10.9% to 9.0%. These results emphasize that the NoisyD-CT framework proposed in this application outperforms the mainstream traditional Conformer-T architecture in enhancing noise robustness and overall ASR performance.

[0164] In addition, in order to illustrate the ability of the NoisyD-CT framework proposed in this application to decompose noisy speech data into clean representations, the simulated noisy speech with ID 26-496-0021.wav is selected as an example. The SNR of this example is 7 and the noise type is STR noise. Figure 3 The three sub-figures shown are: (a) high-dimensional representation of noisy speech h t-noisy , which is output by the Conformer block when processing noisy input; (b) high-dimensional representation of noisy speech h t-noisy Pure clean representation after noise deconstruction Generated by the first encoder in the noise disentanglement module; (c) When the corresponding clean speech is provided as input, the clean speech high-dimensional representation h is obtained from the pre-trained feature encoder and Conformer block t As can be seen from the figure, the pure clean representation of the noisy speech in (b) after noise deconstruction The high-dimensional representation of clean speech obtained from the corresponding clean speech data in (c) t The results are very close, indicating that noise is effectively suppressed while preserving the acoustic properties of clean speech. This strong similarity highlights the effectiveness of the noise deconstruction mechanism of the speech recognition framework proposed in this application and provides additional evidence for its ability to improve ASR performance under noisy conditions. The experimental results show that the noise-robust speech recognition technology based on noise deconstruction proposed in this application has a significant effect on improving the noise robustness of the speech recognition system, meeting the expected results.

[0165] Example 4:

[0166] This application also provides a speech recognition method based on a model trained in any of the above embodiments, Figure 4 A schematic diagram of a speech recognition process provided in an embodiment of the present application, the process includes:

[0167] S401: Acquire speech data to be recognized.

[0168] S402: Acquire semantic information of the speech data to be recognized based on the speech data to be recognized through a pre-trained target speech recognition framework.

[0169] The speech recognition method provided in this application is applied to a computer device, which can be a smart device or a server. The computer device performing speech recognition in this application can be the same as or different from the computer device performing the noise decomposition-based speech recognition model training described above.

[0170] In a possible implementation, the training method of the speech recognition model based on noise decomposition is generally performed in an offline manner. After obtaining the target speech recognition model, the target speech recognition model can be saved in the computer device for speech recognition.

[0171] It should be noted that the training method process of the target speech recognition model has been described in the above embodiments 1-3 and will not be detailed here.

[0172] In this application, the methods for acquiring the speech data to be recognized are broad and diverse, designed to meet the needs of different application scenarios. For example, direct user input: In many interactive application scenarios, such as intelligent voice assistants and voice interactive games, users can directly speak voice commands or express content through an audio input device such as a microphone, and the system will collect these voice signals in real time as the speech data to be recognized. This acquisition method is convenient and direct, providing an instant and natural interactive experience. For example, when using an intelligent voice assistant to check the weather, a user only needs to say "What's the weather like today?" and the voice assistant will collect the voice data in real time as the speech data to be recognized. For example, reading from a storage device: In many data processing and analysis scenarios, the speech data to be recognized may already be stored on a local storage device (such as a hard drive, USB flash drive, etc.) or in a cloud storage system. The system can read pre-recorded voice files from these storage devices based on specific needs. These voice files may come from historical voice records, voice databases, etc. For example, in the field of speech research, researchers can read a large number of voice files with different speakers and content from professional voice databases as the speech data to be recognized for algorithm evaluation and model training. Another example is network transmission and reception: With the rapid development of internet technology, acquiring voice data to be recognized via network transmission is becoming increasingly common. For example, in scenarios such as remote voice conferencing and online voice teaching, voice data is transmitted to the receiving end in real time via the network. The receiving system receives this voice data stream through a network interface and processes it as voice data to be recognized. This approach breaks geographical restrictions and enables efficient remote voice interaction.

[0173] In one possible implementation, to ensure the quality and consistency of the speech data to be recognized and improve the accuracy of subsequent speech recognition, a series of preprocessing operations are typically performed after the speech data is acquired. For example, speech data from different sources may have different sampling rates. Inconsistent sampling rates can affect the feature representation and subsequent processing of the speech signal. Therefore, the sampling rate of all speech data to be recognized can be unified to a standard value, such as the common 16kHz or 44.1kHz. Using a sampling rate conversion algorithm, high-sampling-rate speech data can be downsampled, or low-sampling-rate speech data can be upsampled, ensuring consistent data accuracy in the time dimension. For another example, actual collected speech data is often subject to various noise interferences, such as environmental noise and equipment noise. To improve the signal-to-noise ratio (SNR) of the speech signal, a noise reduction algorithm is used to process the speech data. Common noise reduction algorithms include spectral subtraction, Wiener filtering, and deep learning-based noise reduction models. These algorithms can effectively suppress noise components, highlight the characteristics of the speech signal, and provide purer input for subsequent speech recognition. For another example, speech endpoint detection aims to accurately identify the start and end points of a speech signal and remove silent and noisy segments. By analyzing characteristics of speech signals such as energy and zero-crossing rate, algorithms such as threshold judgment and dynamic programming are used to determine the endpoints of speech. This reduces the amount of data required for subsequent processing, improves processing efficiency, and prevents the impact of silence and noise on speech recognition results.

[0174] In this application, the pre-trained target speech recognition framework is the core of the entire speech recognition process. The framework consists of a fine-tuned Conformer-Transducer architecture and a trained first encoder in the noise disentanglement module. The pre-trained feature encoder in the fine-tuned Conformer-Transducer architecture is connected to the Conformer block, the Conformer block is connected to the first encoder, and the first encoder is connected to the Transducer decoder in the fine-tuned Conformer-Transducer architecture. This architectural design enables the framework to fully utilize the advantages of different modules to achieve efficient speech feature extraction, noise disentanglement, and semantic decoding.

[0175] After acquiring the speech data to be recognized, it can be input into the target speech recognition framework. First, the feature encoder extracts features from the speech data and combines these features into high-dimensional feature vectors. These feature vectors contain the essential acoustic information of the speech signal, providing a foundation for subsequent feature processing and semantic recognition. The extracted feature vectors are then input into the Conformer block. The Conformer block processes the input feature vectors to uncover deeper semantic and pattern information in the speech signal, such as the combination patterns of different phonemes and the prosodic characteristics of the speech. The high-dimensional representation of the noisy speech obtained after processing by the Conformer block contains pure speech components and possible noise components. At this point, the high-dimensional representation of the noisy speech is input into the first encoder. The first encoder leverages its capabilities learned through noise disentanglement training to separate pure speech features from the high-dimensional representation of the noisy speech. By analyzing the distribution and pattern of the features, the first encoder accurately identifies and removes noise components, outputting a pure, clean representation. This step effectively eliminates the interference of noise on subsequent semantic decoding, improving the accuracy of semantic recognition. The pure, clean representation output by the first encoder is input into the Transducer decoder. The Transducer decoder progressively decodes the input features. During the decoding process, the Transducer decoder uses learned knowledge to convert the clean representation into human-understandable semantic information, such as textual sentences and keywords. The Transducer decoder gradually constructs a complete semantic output by continuously predicting the next possible semantic symbol.

[0176] The semantic information of the speech data to be recognized can be output in a variety of formats to meet the needs of different application scenarios. Common output formats include text strings and structured data (such as JSON format). Text string output is intuitive and easy for users to read and understand directly; structured data output facilitates subsequent data processing and analysis. For example, in a voice customer service system, semantic information can be stored as structured data in a database for classification, statistics, and analysis of customer issues.

[0177] Example 5:

[0178] Based on the same inventive concept, the present application also provides a training device for a speech recognition model based on noise deconstruction. Figure 5 A schematic diagram of a training device for a speech recognition model based on noise decomposition provided in an embodiment of the present application, the device comprising:

[0179] A first training unit 51 is configured to perform isolated training on a noise disentanglement module in a first speech recognition framework based on a first mixed speech sample pair in a first mixed data set to obtain a trained noise disentanglement module and freeze parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes first clean speech data and first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, wherein the Conformer-Transducer architecture includes a feature encoder, a Conformer block, and a Transducer decoder;

[0180] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0181] Acquire first clean speech data and first reference speech data in any first mixed speech sample pair;

[0182] Obtaining a first clean speech high-dimensional representation of the first clean speech data through a feature encoder and a Conformer block in the first speech recognition framework, and obtaining a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block;

[0183] Extracting a first pure clean representation from the first noisy speech high-dimensional representation by a first encoder in the noise unwrapping module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation by a second encoder in the noise unwrapping module;

[0184] reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module;

[0185] Adjusting parameters in the noise disentanglement module based on the loss between the first pure clean representation and the first high-dimensional representation of the clean speech, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech;

[0186] A second training unit 52 is configured to fine-tune the Conformer-Transducer architecture in a second speech recognition framework based on a second mixed speech sample pair and its semantic label in a second mixed data set, so as to obtain a trained target speech recognition framework; wherein the second mixed speech sample pair includes second clean speech data and second reference speech data related to the second clean speech data; and the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters and the trained noise disentanglement module with frozen parameters;

[0187] During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed:

[0188] Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair;

[0189] Obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second clean speech high-dimensional representation of the second clean speech data; and obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second noisy speech high-dimensional representation of the second reference speech data;

[0190] Extracting a second pure clean representation from the second noisy speech high-dimensional representation by using the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation by using the second encoder in the trained noise disentanglement module;

[0191] reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through a disentanglement decoder in the trained noise disentanglement module;

[0192] Obtaining predicted semantic information based on the second pure clean identifier through a transducer decoder in the second speech recognition framework;

[0193] The parameters in the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

[0194] The training method and device of the noise deconstruction-based speech recognition model in this embodiment is presented in the form of a functional module, where the module refers to an application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0195] The further functional description of each of the above modules and units is the same as that of the corresponding embodiments 1-3 above and will not be repeated here.

[0196] Example 6:

[0197] Based on the same inventive concept, the present application also provides a speech recognition device, Figure 6 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application, the device comprising:

[0198] An acquisition module 61 is used to acquire speech data to be recognized;

[0199] The processing module 62 is configured to obtain semantic information of the speech data to be recognized based on the speech data to be recognized by using a pre-trained target speech recognition framework.

[0200] The training method and device of the noise deconstruction-based speech recognition model in this embodiment is presented in the form of a functional module, where the module refers to an application-specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0201] The further functional description of each of the above modules and units is the same as that of the corresponding embodiment 4 above, and will not be repeated here.

[0202] Example 7:

[0203] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present application. Figure 7As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. 6 components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of a GUI on an external input / output device (such as, a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0204] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0205] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0206] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of a computer device for displaying a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0207] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0208] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0209] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0210] Example 8:

[0211] Based on the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the following steps:

[0212] Based on a first mixed speech sample pair in a first mixed data set, a noise disentanglement module in a first speech recognition framework is isolatedly trained to obtain a trained noise disentanglement module and freeze the parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes first clean speech data and first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, wherein the Conformer-Transducer architecture includes a feature encoder, a Conformer block, and a Transducer decoder;

[0213] During any iterative isolation training of the noise disentanglement module, the following steps are performed:

[0214] Acquire first clean speech data and first reference speech data in any first mixed speech sample pair;

[0215] Obtaining a first clean speech high-dimensional representation of the first clean speech data through a feature encoder and a Conformer block in the first speech recognition framework, and obtaining a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block;

[0216] Extracting a first pure clean representation from the first noisy speech high-dimensional representation by a first encoder in the noise unwrapping module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation by a second encoder in the noise unwrapping module;

[0217] reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module;

[0218] Adjusting parameters in the noise disentanglement module based on the loss between the first pure clean representation and the first high-dimensional representation of the clean speech, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech;

[0219] Based on the second mixed speech sample pairs and their semantic labels in the second mixed data set, fine-tuning the Conformer-Transducer architecture in the second speech recognition framework to obtain a trained target speech recognition framework; wherein the second mixed speech sample pairs include second clean speech data and second reference speech data related to the second clean speech data; the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters and the trained noise disentanglement module with frozen parameters;

[0220] During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed:

[0221] Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair;

[0222] Obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second clean speech high-dimensional representation of the second clean speech data; and obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second noisy speech high-dimensional representation of the second reference speech data;

[0223] Extracting a second pure clean representation from the second noisy speech high-dimensional representation by using the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation by using the second encoder in the trained noise disentanglement module;

[0224] reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through a disentanglement decoder in the trained noise disentanglement module;

[0225] Obtaining predicted semantic information based on the second pure clean identifier through a transducer decoder in the second speech recognition framework;

[0226] The parameters in the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

[0227] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the training method of the speech recognition model based on noise deconstruction, the implementation of the above-mentioned computer-readable storage medium can refer to Examples 1-3 of the method, and the repeated parts will not be repeated.

[0228] Example 9:

[0229] Based on the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the following steps:

[0230] Obtaining voice data to be recognized;

[0231] The semantic information of the speech data to be recognized is obtained based on the speech data to be recognized through a pre-trained target speech recognition framework.

[0232] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the speech recognition method, the implementation of the above-mentioned computer-readable storage medium can refer to Example 4 of the method, and the repeated parts are not repeated here.

[0233] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for training a speech recognition model based on noise decomposition, characterized in that: The method comprises: Based on a first mixed speech sample pair in a first mixed data set, a noise disentanglement module in a first speech recognition framework is isolatedly trained to obtain a trained noise disentanglement module and freeze the parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes first clean speech data and first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, wherein the Conformer-Transducer architecture includes a feature encoder, a Conformer block, and a Transducer decoder; During any iterative isolation training of the noise disentanglement module, the following steps are performed: Acquire first clean speech data and first reference speech data in any first mixed speech sample pair; Obtaining a first clean speech high-dimensional representation of the first clean speech data through a feature encoder and a Conformer block in the first speech recognition framework, and obtaining a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block; Extracting a first pure clean representation from the first noisy speech high-dimensional representation by a first encoder in the noise unwrapping module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation by a second encoder in the noise unwrapping module; reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module; Adjusting parameters in the noise disentanglement module based on the loss between the first pure clean representation and the first high-dimensional representation of the clean speech, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech; Based on the second mixed speech sample pairs and their semantic labels in the second mixed data set, fine-tuning the Conformer-Transducer architecture in the second speech recognition framework to obtain a trained target speech recognition framework; wherein the second mixed speech sample pairs include second clean speech data and second reference speech data related to the second clean speech data; the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters and the trained noise disentanglement module with frozen parameters; During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed: Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair; Obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second clean speech high-dimensional representation of the second clean speech data; and obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second noisy speech high-dimensional representation of the second reference speech data; Extracting a second pure clean representation from the second noisy speech high-dimensional representation by using the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation by using the second encoder in the trained noise disentanglement module; reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through a disentanglement decoder in the trained noise disentanglement module; Obtaining predicted semantic information based on the second pure clean identifier through a transducer decoder in the second speech recognition framework; The parameters in the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

2. The method according to claim 1, wherein The Conformer-Transducer architecture with frozen parameters after pre-training is obtained as follows: Based on the third clean speech data and its semantic labels in the third dataset, the original Conformer-Transducer architecture is iteratively trained to obtain a Conformer-Transducer architecture with frozen parameters after pre-training.

3. The method according to claim 1, wherein The types of the reference speech data include noisy speech data and clean speech data, and the reference speech data is the first reference speech data and / or the second reference speech data.

4. The method according to claim 1, wherein Fine-tuning parameters in the Conformer-Transducer architecture according to the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation, comprises: Determine a joint loss based on the loss between the predicted semantic information and the semantic label and the weight value thereof, the loss between the second pure clean representation and the second clean speech high-dimensional representation and the weight value thereof, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation and the weight value thereof; Based on the joint loss, the parameters in the Conformer-Transducer architecture are fine-tuned.

5. The method according to claim 1, wherein The trained target speech recognition framework includes: a fine-tuned Conformer-Transducer architecture and a first encoder in the trained noise disentanglement module; wherein the feature encoder in the fine-tuned Conformer-Transducer architecture is connected to a Conformer block, the Conformer block is connected to the first encoder, and the first encoder is connected to a Transducer decoder in the fine-tuned Conformer-Transducer architecture.

6. A speech recognition method based on a model trained by the method according to any one of claims 1 to 5, characterized in that: The method comprises: Obtaining voice data to be recognized; The semantic information of the speech data to be recognized is obtained based on the speech data to be recognized through a pre-trained target speech recognition framework.

7. A training device for a speech recognition model based on noise decomposition, characterized in that: The device comprises: a first training unit, configured to perform isolated training on a noise disentanglement module in a first speech recognition framework based on a first mixed speech sample pair in a first mixed data set, so as to obtain a trained noise disentanglement module and freeze parameters of the trained noise disentanglement module; wherein the first mixed speech sample pair includes first clean speech data and first reference speech data related to the first clean speech data; the first speech recognition framework includes the noise disentanglement module with adjustable parameters, and a Conformer-Transducer architecture with frozen parameters after pre-training, wherein the Conformer-Transducer architecture includes a feature encoder, a Conformer block, and a Transducer decoder; During any iterative isolation training of the noise disentanglement module, the following steps are performed: Acquire first clean speech data and first reference speech data in any first mixed speech sample pair; Obtaining a first clean speech high-dimensional representation of the first clean speech data through a feature encoder and a Conformer block in the first speech recognition framework, and obtaining a first noisy speech high-dimensional representation of the first reference speech data through the feature encoder and the Conformer block; Extracting a first pure clean representation from the first noisy speech high-dimensional representation by a first encoder in the noise unwrapping module; and extracting a first pure noise representation from the first noisy speech high-dimensional representation by a second encoder in the noise unwrapping module; reconstructing a first high-dimensional acoustic representation based on the first pure noise representation and the first pure clean representation through a disentanglement decoder in the noise disentanglement module; Adjusting parameters in the noise disentanglement module based on the loss between the first pure clean representation and the first high-dimensional representation of the clean speech, and the loss between the first high-dimensional acoustic representation and the first high-dimensional representation of the noisy speech; A second training unit is configured to fine-tune the Conformer-Transducer architecture in a second speech recognition framework based on a second mixed speech sample pair and its semantic label in a second mixed data set, so as to obtain a trained target speech recognition framework; wherein the second mixed speech sample pair includes second clean speech data and second reference speech data related to the second clean speech data; and the second speech recognition framework includes the Conformer-Transducer architecture with adjustable parameters and the trained noise disentanglement module with frozen parameters; During any iteration of fine-tuning training of the Conformer-Transducer architecture, the following steps are performed: Acquire second clean speech data and second reference speech data in any second mixed speech sample pair, as well as a semantic label of the second mixed speech sample pair; Obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second clean speech high-dimensional representation of the second clean speech data; and obtaining, through the feature encoder and Conformer block in the second speech recognition framework, a second noisy speech high-dimensional representation of the second reference speech data; Extracting a second pure clean representation from the second noisy speech high-dimensional representation by using the first encoder in the trained noise disentanglement module; and extracting a second pure noise representation from the second noisy speech high-dimensional representation by using the second encoder in the trained noise disentanglement module; reconstructing a second high-dimensional acoustic representation based on the second pure noise representation and the second pure clean representation through a disentanglement decoder in the trained noise disentanglement module; Obtaining predicted semantic information based on the second pure clean identifier through a transducer decoder in the second speech recognition framework; The parameters in the Conformer-Transducer architecture are fine-tuned based on the loss between the predicted semantic information and the semantic label, the loss between the second pure clean representation and the second clean speech high-dimensional representation, and the loss between the second high-dimensional acoustic representation and the second noisy speech high-dimensional representation.

8. A speech recognition device based on a model trained by the method according to any one of claims 1 to 5, characterized in that: The device comprises: An acquisition module, used to acquire the voice data to be recognized; The processing module is used to obtain semantic information of the speech data to be recognized based on the speech data to be recognized through a pre-trained target speech recognition framework.

9. A computer device, characterized in that: The computer device includes a processor, which is used to implement the steps of the training method of the speech recognition model based on noise deconstruction as described in any one of claims 1 to 5 above when executing the computer program stored in the memory, or to implement the steps of the speech recognition method as described in claim 6 above.

10. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the steps of the training method of the speech recognition model based on noise deconstruction as described in any one of claims 1 to 5 above, or implements the steps of the speech recognition method as described in claim 6 above.

Images