Model training method, speech recognition method, device, medium, and program product

Abstract

This invention discloses a model training method, a speech recognition method, a device, a medium, and a program product. The method includes: decoupling sample speech information by an acoustic feature encoder to determine sample speech content features and sample speech acoustic style features; performing acoustic feature discrimination on the sample speech content features using an acoustic feature discriminator to determine adversarial loss; fusing the sample speech content features and sample speech acoustic style features using a feature fusion module to determine fused sample speech features; extracting text information from the fused sample speech features using an automatic speech recognition decoder, and determining the automatic speech recognition loss based on the extracted text and speech recognition training samples; and substituting the adversarial loss and the automatic speech recognition loss into a pre-constructed total loss function to complete the training of the speech recognition model. This enables the speech recognition model to recognize speech data acquired with different accents and in different environments.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a model training method, a speech recognition method, a device, a medium, and a program product. Background Technology

[0002] With the rapid development of artificial intelligence and machine learning technologies, speech recognition technology has made significant progress and is gradually integrating into people's daily lives. From intelligent voice assistants to voice-driven interactive systems, the application of these technologies has greatly improved the naturalness and convenience of human-computer interaction. Against this backdrop, Automatic Speech Recognition (ASR), as a technology that converts human speech signals into text, aims to achieve accurate mapping from speech to text. Its robustness heavily relies on large-scale, highly diverse speech datasets.

[0003] However, acquiring large-scale speech data is currently quite difficult, and conventional methods such as adding noise and speed perturbation only superficially enrich the acquired speech data. At the same time, ASR model training often only optimizes the word error rate, treating the input information as a whole to consider the accuracy of the text obtained from speech recognition, failing to fully utilize the rich acoustic information in the speech signal, and making it difficult to meet the requirements of high-precision and high-robust speech-to-text conversion. Summary of the Invention

[0004] This invention provides a model training method, a speech recognition method, a device, a medium, and a program product. By decoupling the pronunciation-related features and text content-related features in speech information, it assists in training a speech recognition model. The trained speech recognition model can then recognize speech data acquired with different accents and in different environments. By utilizing the rich acoustic information in the speech data, the robustness of the speech recognition process and the accuracy of the recognition results are improved.

[0005] In a first aspect, embodiments of the present invention provide a model training method, wherein the speech recognition model includes at least an acoustic feature encoder, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder, the method comprising:

[0006] The sample speech information from the speech recognition training samples is input into the untrained speech recognition model. The acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features.

[0007] The acoustic feature discriminator is used to perform acoustic feature discrimination on the speech content features of the sample to determine the adversarial loss;

[0008] The feature fusion module is used to fuse the content features and acoustic style features of the sample speech to determine the fused sample speech features.

[0009] Text information is extracted from the speech features of the fused samples using an automatic speech recognition decoder. The extracted text results are then determined, and the automatic speech recognition loss is determined based on the extracted text results and the speech recognition training samples.

[0010] The adversarial loss and automatic speech recognition loss are substituted into the pre-built total loss function, and the speech recognition model is trained based on the total loss function.

[0011] Secondly, embodiments of the present invention also provide a speech recognition method, including:

[0012] Acquire the voice information to be processed;

[0013] The speech information to be processed is input into a pre-trained speech recognition model, and the speech recognition result is determined based on the output of the speech recognition model.

[0014] The speech recognition model is trained using the model training method provided in any embodiment of the present invention.

[0015] Thirdly, embodiments of the present invention also provide a voice recognition device, comprising:

[0016] At least one processor; and a memory communicatively connected to the at least one processor;

[0017] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to implement the speech recognition method of any embodiment of the present invention.

[0018] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the speech recognition method of any embodiment of the present invention.

[0019] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program, which, when executed by a processor, is used to perform the speech recognition method of any embodiment of the present invention.

[0020] This invention provides a model training method, speech recognition method, device, medium, and program product. The method involves inputting sample speech information from speech recognition training samples into an untrained speech recognition model. An acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features. An acoustic feature discriminator performs acoustic feature discrimination on the sample speech content features to determine the adversarial loss. A feature fusion module fuses the sample speech content features and sample speech acoustic style features to determine the fused sample speech features. An automatic speech recognition decoder extracts text information from the fused sample speech features to determine the sample text extraction result. Based on the sample text extraction result and the speech recognition training samples, an automatic speech recognition loss is determined. The adversarial loss and the automatic speech recognition loss are then substituted into a pre-constructed total loss function, and the speech recognition model is trained based on the total loss function. By adopting the above technical solution, a speech recognition model used to identify and extract content-related information in natural language is enhanced by adding a speech information decoder that decouples the input speech information into speech content features and acoustic style features, and an acoustic feature discriminator for training the feature separation capability of the speech information decoder. This allows the speech recognition model to extract features that are sufficiently separated from content-related and non-content-related information during training, rather than extracting and recognizing spectral features as a whole. This enables more accurate extraction of text-related speech content features, which are then accurately determined by the automatic speech recognition decoder. Because the acoustic feature encoder can decouple acoustic style features unrelated to speech content, the recognition accuracy of the speech recognition model is less affected by different accents and environmental information in the input speech information. This allows the speech recognition model to utilize the rich acoustic information in the speech data, improving the robustness and accuracy of the trained speech recognition model in the speech recognition process.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a flowchart of a model training method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a model training method provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a flowchart example of another model training method provided in Embodiment 2 of the present invention;

[0026] Figure 4 This is a flowchart example of another model training method provided in Embodiment 2 of the present invention;

[0027] Figure 5 This is a flowchart of a speech recognition method provided in Embodiment 3 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of a speech recognition model training device provided in Embodiment 4 of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of a speech recognition method device provided in Embodiment 5 of the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of a voice recognition device provided in Embodiment Six of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a model training method provided in Embodiment 1 of the present invention. This embodiment is applicable to training a speech recognition model so that the model can accurately and fully recognize and extract content from speech information expressed in natural language, acquired in different environments and with different accents. This method can be executed by a speech recognition model training device, which can be implemented in software and / or hardware and can be configured in a speech recognition model training equipment. Optionally, the speech recognition model training equipment can be a laptop, desktop computer, or smart tablet, etc., and this embodiment of the present invention does not impose any limitations on this.

[0035] To more clearly illustrate the speech recognition model proposed in this embodiment of the invention, the modules included in the speech recognition model are first described here. The speech recognition model includes at least an acoustic feature encoder, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder.

[0036] In this embodiment, the acoustic feature encoder can be specifically understood as an encoder used to convert the input raw speech signal into a high-level representation vector with semantic relevance, lower dimensionality, and stronger anti-interference capability. In this embodiment, it is used to decouple the speech information input to the speech recognition model, separating it into speech content features and acoustic style features. Specifically, speech content features can be understood as features in the speech information that are related to the textual content actually intended to be expressed by the speaker, i.e., features indicating "what was said"; acoustic style features can be understood as non-semantic features in the speech information that are not related to the semantic information of the textual content actually intended to be expressed by the speaker, and may affect the accuracy of speech recognition, i.e., features indicating "who spoke, how they spoke, and where they spoke".

[0037] Optional acoustic style features include at least one of the following:

[0038] Human-related vocal characteristics;

[0039] Environmentally relevant sound characteristics.

[0040] For example, human-related features in acoustic style characteristics may include the following:

[0041] 1) Speaker identity; due to the unique characteristics of each person's personal timbre and vocal tract shape, different speakers will have different acoustic style characteristics.

[0042] 2) Speech rhythm; the acoustic features that constitute intonation, such as pitch, speech rate, duration of each phoneme, and loudness, due to different expressive needs and different people's expressive habits.

[0043] 3) Age and gender; due to differences in physiological structure, people of different ages and genders have different acoustic characteristics.

[0044] 4) Accent; the acoustic style characteristics formed by the pronunciation habits of specific regions or social groups, such as the pronunciation of vowels and the clarity of consonants.

[0045] 5) Emotion; Different emotional states during speech result in different acoustic style characteristics.

[0046] For example, environment-related features in acoustic style characteristics may include the following:

[0047] 1) Acoustic pattern features introduced by interference or recording characteristics generated during the recording process of the recording equipment.

[0048] 2) Acoustic pattern features introduced during voice information transmission due to the stability of the transmission channel.

[0049] 3) Acoustic pattern features introduced by background noise and other factors in the acquisition environment.

[0050] In this embodiment, the feature fusion module can be specifically understood as a module used to integrate features from different sources, dimensions, and types into a single, comprehensive feature based on expressive power using a specific algorithm. In this embodiment of the invention, the feature fusion module can be understood as a module used to fuse separate speech content features and acoustic style features according to recognition requirements.

[0051] In this embodiment, the acoustic feature discriminator can be specifically understood as a discriminator used to determine whether the input sample speech content features also have acoustic style features.

[0052] In this embodiment, the automatic speech recognition decoder can be specifically understood as a decoder used to map high-level acoustic representations into human-understandable text sequences.

[0053] like Figure 1 As shown in the figure, the model training method provided by this embodiment of the invention specifically includes the following steps:

[0054] S101. Input the sample speech information from the speech recognition training samples into the untrained speech recognition model. The acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features.

[0055] In this embodiment, the speech recognition training samples can be specifically understood as sample data pre-constructed based on actual conditions for training the speech recognition model. Optionally, the speech recognition training samples may include sample speech information and sample speech content information contained within the sample speech information. Specifically, the sample speech information can be understood as natural language speech information directly given by a person; the sample speech content information can be understood as information presented in text form within the sample speech information that can actually represent its semantic content.

[0056] It is understandable that the sample speech content features and sample speech acoustic style features have the same meaning as the speech content features and speech acoustic style features explained above. The only difference is that these features are extracted from the sample speech data.

[0057] Specifically, the sample speech information from the speech recognition training samples is used as input to the untrained speech recognition model. The acoustic feature encoder in the speech recognition model decouples this input, yielding the corresponding sample speech content features and sample speech acoustic style features. The obtained sample speech content features can then be used as input to the acoustic feature discriminator, while both the sample speech content features and the sample speech acoustic style features are used together as input to the feature fusion module for subsequent training.

[0058] S102. The acoustic feature discriminator is used to perform acoustic feature discrimination on the speech content features of the sample to determine the adversarial loss.

[0059] In this embodiment, the adversarial loss can be specifically understood as the loss of the generator and discriminator in the adversarial network playing against each other. It can be used to drive the generated samples to approximate the real data distribution. In this embodiment of the invention, the adversarial loss is used to characterize the ability of the acoustic feature encoder to decouple speech content features and speech acoustic style features clearly, so that the acoustic feature encoder can separate speech content features and speech acoustic style features as clearly as possible.

[0060] Specifically, an acoustic feature discriminator is used to perform acoustic feature discrimination on the sample speech content features, and the adversarial loss is determined based on the amount of acoustic features contained in the sample speech content features.

[0061] S103. The feature fusion module is used to fuse the content features and acoustic style features of the sample speech to determine the fused sample speech features.

[0062] Specifically, after the acoustic feature encoder decouples the sample speech information into sample speech content features and sample speech acoustic style features, to ensure the integrity of the semantic information input into the automatic speech recognition decoder, the two need to be fused again. At this point, a feature fusion module will fuse the sample speech content features and sample speech acoustic style features, and the fused features will be determined as the fused sample speech features. Optionally, in this embodiment of the invention, the fusion of sample speech content features and sample speech acoustic style features can be achieved by concatenation followed by projection. Representing the acoustic style features of sample speech, in The fused sample speech features represent the content features of the sample speech. It can be represented as:

[0063] ;

[0064] in, Indicates will Vector sum Vectors are concatenated along their dimensions, such as... , The concatenated vector .in, and A learnable weight matrix and bias vector together form a fully connected layer, whose function is to process the concatenated high-dimensional vector. Projected onto a dimension suitable for the input of an automatic speech recognition decoder. .

[0065] It is understandable that, since the prosody and vocal characteristics of different phonemes can affect semantic understanding, in order to provide complete contextual understanding and improve the accuracy of text information extraction by the automatic speech recognition decoder, the feature fusion module still needs to fuse the decoupled sample speech content features and sample speech acoustic style features. However, in this embodiment of the invention, the sample speech content features are first extracted by filtering out acoustic feature interference through an acoustic feature encoder, which makes the text information extracted from the sample speech content features more accurate. Moreover, the fusion and splicing method is more controllable than the feature extraction of the entire sample speech information. Therefore, the automatic speech recognition decoder trained in this way can extract text information from the fused sample speech features more accurately.

[0066] S104. Extract text information from the speech features of the fused samples using an automatic speech recognition decoder, determine the sample text extraction results, and determine the automatic speech recognition loss based on the sample text extraction results and the speech recognition training samples.

[0067] Specifically, the automatic speech recognition decoder, which acts as an end-to-end decoder, completes the mapping of acoustic features to text information. Based on the acoustic features and context-related information in the fused sample speech features, the semantic content information contained in the sample speech information is extracted in text form. The extracted text sequence is determined as the sample text extraction result. The sample extraction result is compared with the sample speech content information in the speech recognition training samples to determine the automatic speech recognition loss.

[0068] S105. Substitute the adversarial loss and automatic speech recognition loss into the pre-constructed total loss function, and complete the training of the speech recognition model based on the total loss function.

[0069] In this embodiment, the total loss function can be understood as a loss function pre-constructed based on the modules in the speech recognition model, used for the overall training of the speech recognition model. For example, when the speech recognition model includes an acoustic feature encoder, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder, the total loss function... Possible forms:

[0070] ;

[0071] in, For the loss of automatic speech recognition, To combat the losses, These are the weighting coefficients.

[0072] Specifically, the determined adversarial loss and automatic speech recognition loss are substituted into the pre-constructed total loss function. Based on the total loss function and the separately existing adversarial loss and automatic speech recognition loss, the parameters of each component module in the speech recognition model are trained and adjusted to obtain the trained speech recognition model.

[0073] The technical solution of this invention involves inputting sample speech information from speech recognition training samples into an untrained speech recognition model. An acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features. An acoustic feature discriminator performs acoustic feature discrimination on the sample speech content features to determine the adversarial loss. A feature fusion module fuses the sample speech content features and sample speech acoustic style features to determine the fused sample speech features. An automatic speech recognition decoder extracts text information from the fused sample speech features to determine the sample text extraction result. Based on the sample text extraction result and the speech recognition training samples, an automatic speech recognition loss is determined. The adversarial loss and the automatic speech recognition loss are then substituted into a pre-constructed total loss function, and the speech recognition model is trained based on the total loss function. By adopting the above technical solution, a speech recognition model used to identify and extract content-related information in natural language is enhanced by adding a speech information decoder that decouples the input speech information into speech content features and acoustic style features, and an acoustic feature discriminator for training the feature separation capability of the speech information decoder. This allows the speech recognition model to extract features that are sufficiently separated from content-related and non-content-related information during training, rather than extracting and recognizing spectral features as a whole. This enables more accurate extraction of text-related speech content features, which are then accurately determined by the automatic speech recognition decoder. Because the acoustic feature encoder can decouple acoustic style features unrelated to speech content, the recognition accuracy of the speech recognition model is less affected by different accents and environmental information in the input speech information. This allows the speech recognition model to utilize the rich acoustic information in the speech data, improving the robustness and accuracy of the trained speech recognition model in the speech recognition process.

[0074] Example 2

[0075] Figure 2This is a flowchart of a model training method provided in Embodiment 2 of the present invention. This embodiment further optimizes the above-mentioned optional implementation methods, clarifying the training method of the speech recognition model after adding a new feature enhancement module and other modules. During model training, the feature enhancement module enriches and enhances the acoustic features within the model, enabling the trained automatic speech recognition decoder to better adapt to speech recognition in different language expression states. Simultaneously, during the training process, by fixing the acoustic feature discriminator and other modules in the speech recognition model respectively, adversarial training with opposing goals is performed in stages. This allows the speech recognition model to better learn acoustic invariance, improving the robustness of the trained speech recognition model and further enhancing the accuracy of subsequent speech recognition using the model.

[0076] like Figure 2 As shown in the figure, the model training method provided by this embodiment of the invention specifically includes the following steps:

[0077] S201. Input the sample speech information from the speech recognition training samples into the untrained speech recognition model. The acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features.

[0078] The speech recognition model also includes a feature enhancement module.

[0079] In this embodiment, the feature enhancement module can be specifically understood as a module used to systematically and structurally enhance the acoustic pattern features of sample speech, so that the enhanced acoustic features of sample speech can fill the blank areas that the acoustic pattern features in the speech recognition training samples cannot cover. When the number of speech recognition training samples is limited, this module provides richer input to the automatic speech recognition decoder during the training process.

[0080] S202. Use an acoustic feature discriminator to perform acoustic feature discrimination on the speech content features of the sample and determine the adversarial loss.

[0081] It is understandable that the method for determining the countermeasure losses here is the same as in S102, and will not be repeated here.

[0082] S203. The acoustic style features of the sample speech are enhanced by the feature enhancement module, and the enhanced style features are fused with the sample speech content features by the pre-built feature fusion module to determine the fused sample speech features.

[0083] Feature enhancement is used to expand the acoustic coverage of acoustic style features.

[0084] Specifically, the acoustic style features of the sample speech are used as input to the feature enhancement module. After feature enhancement, multiple enhanced style features are obtained. This can be understood as providing a richer expression of the sample speech content features in the speech recognition training samples under acoustic styles. The enhanced style features and sample speech content features are used together as input to the feature fusion module for feature fusion. Compared to the original situation where only one fused sample speech feature can be obtained from a single speech recognition training sample, this method can obtain multiple different fused sample speech features containing the same sample speech content, thus enriching the training input of the automatic speech recognition decoder.

[0085] Optionally, feature enhancement methods include at least one of the following: linear interpolation and extrapolation, random noise addition, and cluster-based pattern reorganization.

[0086] For example, linear interpolation can be understood as creating a new point as an enhanced style feature on the line segment connecting two sample speech acoustic style features. For instance, it can mix a "male voice" feature and a "female voice" feature to produce a "neutral" or mixed virtual acoustic feature as an enhanced style feature.

[0087] For example, extrapolation can be understood as extending the acoustic pattern features of a sample speech along the direction (or the opposite direction) from one sample speech acoustic pattern feature to another, and using the extended features as enhanced pattern features. For instance, it can extrapolate an enhanced pattern feature with a "heavier accent" from the acoustic pattern features of a sample speech in a "slightly accented" sample.

[0088] For example, adding random noise can be understood as superimposing a small, random noise vector onto the acoustic pattern features of the sample speech to enhance the pattern features. This method is equivalent to performing a "local exploration" in the acoustic pattern space, simulating subtle or unpredictable fluctuations in the acoustic pattern features, preventing the speech recognition model from overfitting to the acoustic pattern features during training, and enhancing the robustness of the trained speech recognition model.

[0089] For example, cluster-based pattern reorganization can be understood as performing unsupervised clustering of the acoustic pattern features of each sample in multiple speech recognition training samples used to train the speech recognition model. Each cluster represents a typical acoustic pattern (such as "crisp female voice", "deep male voice" and "a certain accent"). When performing pattern reorganization, a sample acoustic pattern feature is randomly sampled from near the center point of one cluster and mixed or extrapolated with a sample acoustic pattern feature from another cluster. Based on this, guided acoustic pattern feature enhancement can be performed, and mixed patterns from different acoustic patterns can be actively created.

[0090] It is important to clarify that, for speech recognition models, the traditional encoder's role is to map the extracted features, not to decouple the speech information. However, the acoustic feature encoder proposed in this embodiment of the invention is used to decouple the acoustic features from the non-acoustic features of the input speech information. Furthermore, at the model architecture level, this embodiment of the invention, based on the existing speech recognition model architecture which includes an encoder and a decoder, adds a feature fusion module to adapt to the decoupling capability of the acoustic feature encoder. Additionally, although existing speech recognition models may contain feature enhancement modules, these are not feature enhancement modules for enhancing the decoupled acoustic pattern features, but rather for enhancing the overall speech information. Therefore, they cannot generate more samples at a lower cost to complete sufficient training for the speech recognition model.

[0091] S204. Extract text information from the speech features of the fused samples using an automatic speech recognition decoder, determine the sample text extraction results, and determine the automatic speech recognition loss based on the sample text extraction results and the speech recognition training samples.

[0092] It is understood that S202 and S203-S204 can be executed simultaneously or in any order, and this embodiment of the invention does not impose any restrictions on this.

[0093] S205. In the fixed speech recognition model, except for the acoustic feature discriminator, the acoustic feature discriminator is trained with the goal of minimizing adversarial loss.

[0094] It is understandable that minimizing the adversarial loss can be interpreted as enabling the acoustic feature discriminator to better distinguish between speech content features and acoustic style features, making the determined adversarial loss more accurate, and thus enabling the total loss function based on the adversarial loss to better train the decoupling ability of the acoustic feature encoder.

[0095] S206. A fixed acoustic feature discriminator is used to train the automatic speech recognition decoder and feature fusion module with the goal of minimizing the automatic speech recognition loss, and to train each module except the automatic speech recognition decoder and feature fusion module based on the total loss function.

[0096] Understandably, minimizing the loss in automatic speech recognition can be understood as enabling the automatic speech recognition decoder to better recognize the content information contained in the speech, and enabling the feature fusion module to more accurately fuse speech content features with acoustic pattern features for the automatic speech recognition decoder to recognize, which can be used to improve the accuracy of speech recognition.

[0097] It is understood that the speech recognition model in this application includes an acoustic feature encoder, a feature enhancement module, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder. The training of each module except the automatic speech recognition decoder and the feature fusion module based on the total loss function may include: training the acoustic feature encoder with the objective of minimizing the total loss function.

[0098] Optionally, the speech recognition model also includes a reconstruction module, accordingly. Figure 3 This is a flowchart example of another model training method provided in Embodiment 2 of the present invention, as shown below. Figure 3 As shown, the specific steps include the following:

[0099] S301. Input the sample speech information from the speech recognition training samples into the untrained speech recognition model. The acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features.

[0100] It is understandable that the method of processing sample speech information through acoustic feature encoder is the same as that in S101 above, and will not be repeated here.

[0101] S302. Use an acoustic feature discriminator to perform acoustic feature discrimination on the speech content features of the sample and determine the adversarial loss.

[0102] It is understandable that the method for determining the countermeasure losses here is the same as in S102, and will not be repeated here.

[0103] S303. The reconstruction module reconstructs the speech information of the sample speech content features and the sample speech acoustic style features to determine the reconstructed sample speech information, and determines the reconstruction loss based on the reconstructed sample speech information and the sample speech information.

[0104] In this embodiment, the reconstruction module can be specifically understood as a module used to reconstruct the content features and acoustic style features obtained by decoupling the acoustic feature encoder into speech information before decoupling of the acoustic feature encoder. This module can determine the correctness of the decoupling of the acoustic feature encoder by reverse reconstruction of the features obtained by decoupling the acoustic feature encoder.

[0105] Specifically, the sample speech content features and sample speech acoustic style features are used as inputs to the reconstruction module. The reconstruction module completes the reconstruction from features to speech information to obtain reconstructed sample speech information. The reconstructed sample speech information is compared with the original sample speech information to determine whether the original sample speech information can be obtained by reverse engineering based on the decoupled sample speech content features and sample speech acoustic style features. The reconstruction loss is determined based on the comparison results.

[0106] S304. The acoustic style features of the sample speech are enhanced by the feature enhancement module, and the enhanced style features are fused with the sample speech content features by the pre-built feature fusion module to determine the fused sample speech features.

[0107] It is understandable that the method for determining the speech features of the enhanced and fused samples is the same as in S203, and will not be repeated here.

[0108] S305. Extract text information from the speech features of the fused samples using an automatic speech recognition decoder, determine the sample text extraction results, and determine the automatic speech recognition loss based on the sample text extraction results and the speech recognition training samples.

[0109] It is understandable that the method for determining the loss of automatic speech recognition is the same as that in S104, and will not be repeated here.

[0110] It is understood that S302, S303 and S304-S305 can be executed simultaneously or in any order, and the embodiments of the present invention do not impose any restrictions on this.

[0111] S306. Substitute the adversarial loss, the automatic speech recognition loss, and the reconstruction loss into the pre-constructed total loss function, and complete the training of the speech recognition model based on the total loss function.

[0112] For example, in a speech recognition model that includes a reconstruction module, the total loss function Possible forms:

[0113] ;

[0114] in, For the loss of automatic speech recognition, To combat the losses, To rebuild the losses, and These are the weighting coefficients.

[0115] Accordingly, training a speech recognition model based on the total loss function can be achieved in the following way:

[0116] In a fixed speech recognition model, all modules except the acoustic feature discriminator are trained with the goal of minimizing adversarial loss.

[0117] A fixed acoustic feature discriminator is used to train the automatic speech recognition decoder and feature fusion module with the goal of minimizing the automatic speech recognition loss. The modules other than the automatic speech recognition decoder and feature fusion module are trained based on the total loss function.

[0118] It is understood that the speech recognition model in this application includes an acoustic feature encoder, a reconstruction module, a feature enhancement module, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder. The training of each module except the automatic speech recognition decoder and the feature fusion module based on the total loss function may include training the acoustic feature encoder with the objective of minimizing the total loss function.

[0119] Optionally, the speech recognition model also includes a mutual information estimator, accordingly. Figure 4 This is a flowchart example of another model training method provided in Embodiment 2 of the present invention, as shown below. Figure 4 As shown, the specific steps include the following:

[0120] S401. Input the sample speech information from the speech recognition training samples into the untrained speech recognition model. The acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features.

[0121] It is understandable that the method of processing sample speech information through acoustic feature encoder is the same as that in S101 above, and will not be repeated here.

[0122] S402. Use an acoustic feature discriminator to perform acoustic feature discrimination on the speech content features of the sample and determine the adversarial loss.

[0123] It is understandable that the method for determining the countermeasure losses here is the same as in S102, and will not be repeated here.

[0124] S403. Calculate the cross-entropy of the sample speech content features and the sample speech acoustic style features using a mutual information estimator to determine the mutual information loss.

[0125] In this embodiment, the mutual information estimator can be specifically understood as an algorithmic model used to determine the dependency between content features and acoustic style features obtained by decoupling from the acoustic feature encoder. Optionally, the mutual information estimator can determine the relative independence between different features by calculating the cross-entropy of two of the input feature information.

[0126] Specifically, the sample speech content features and sample speech acoustic style features are used as inputs to the mutual information estimator. The mutual information estimator calculates the cross-entropy between the sample speech content features and sample speech acoustic style features, and the calculated cross-entropy is determined as the mutual information loss.

[0127] S404. The acoustic style features of the sample speech are enhanced by the feature enhancement module, and the enhanced style features are fused with the sample speech content features by the pre-built feature fusion module to determine the fused sample speech features.

[0128] It is understandable that the method for determining the speech features of the enhanced and fused samples is the same as in S203, and will not be repeated here.

[0129] S405. Extract text information from the speech features of the fused samples using an automatic speech recognition decoder, determine the sample text extraction results, and determine the automatic speech recognition loss based on the sample text extraction results and the speech recognition training samples.

[0130] It is understandable that the method for determining the loss of automatic speech recognition is the same as that in S104, and will not be repeated here.

[0131] It is understood that S402, S403 and S404-S405 can be executed simultaneously or in any order, and this embodiment of the invention does not impose any restrictions on this.

[0132] S406. Substitute the adversarial loss, automatic speech recognition loss, and mutual information loss into the pre-constructed total loss function, and complete the training of the speech recognition model based on the total loss function.

[0133] For example, in a speech recognition model that includes mutual information loss, the total loss function Possible forms:

[0134] ;

[0135] in, For the loss of automatic speech recognition, To combat the losses, For mutual information loss, and These are the weighting coefficients.

[0136] Accordingly, training a speech recognition model based on the total loss function can be achieved in the following way:

[0137] In a fixed speech recognition model, all modules except the acoustic feature discriminator are trained with the goal of minimizing adversarial loss.

[0138] A fixed acoustic feature discriminator is used to train the automatic speech recognition decoder and feature fusion module with the goal of minimizing the automatic speech recognition loss. The modules other than the automatic speech recognition decoder and feature fusion module are trained based on the total loss function.

[0139] It is understood that the speech recognition model in this application includes an acoustic feature encoder, a mutual information estimator, a feature enhancement module, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder. The training of each module except the automatic speech recognition decoder and the feature fusion module based on the total loss function can include: training the acoustic feature encoder with the goal of minimizing the total loss function; and training the mutual information estimator with the goal of minimizing the mutual information loss.

[0140] Understandably, a speech recognition model can also include both a reconstruction module and a mutual information estimator. In this case, the total loss function... Possible forms:

[0141] ;

[0142] in, For the loss of automatic speech recognition, To combat the losses, To rebuild the losses, For mutual information loss, , and These are the weighting coefficients.

[0143] The training method for each module in the speech recognition model based on the total loss function is consistent with that described above, and will not be elaborated further in this embodiment of the invention.

[0144] The technical solution of this embodiment clarifies the training method of a speech recognition model after adding a new feature enhancement module and other modules. During model training, the feature enhancement module enriches and enhances the acoustic features within the model, enabling the trained automatic speech recognition decoder to better adapt to speech recognition in different language expressions. Simultaneously, during the training process, by fixing the acoustic feature discriminator and all other modules in the speech recognition model, a phased and opposing adversarial training approach is adopted. This allows the speech recognition model to better learn acoustic invariance, improving the robustness of the trained model and further enhancing the accuracy of subsequent speech recognition using the model.

[0145] In another embodiment of this application, the "acoustic feature discriminator" and the "acoustic feature encoder and automatic speech recognition decoder" are trained alternately in stages. That is, when training with the same batch of sample speech data, the "acoustic feature discriminator" is first trained a preset number of times (the preset number is set according to the actual application scenario and is an integer greater than 1), and then the "acoustic feature encoder and automatic speech recognition decoder" is trained once using the same batch of sample speech data. Then, the above process of first training the "acoustic feature discriminator" a preset number of times and then training the "acoustic feature encoder and automatic speech recognition decoder" once using a new batch of sample speech data is repeated until the training number of the "acoustic feature encoder and automatic speech recognition decoder" reaches a preset maximum number of times. The preset maximum number of times is preset based on the training effect. When training the preset maximum number of times, the above model can achieve the optimal effect. The above-mentioned alternating cyclic training method can increase the acoustic feature discriminator's ability to discriminate acoustic features. The following example illustrates this: Suppose there are currently 100 sample speech data, which are divided into 10 batches of sample speech data (each batch contains 10 sample speech data). The maximum preset number of iterations is set to 10, and the preset number of iterations for the acoustic feature discriminator is 2. First, the acoustic feature discriminator is trained twice using the first batch of 10 speech sample data. Then, the second output of the acoustic feature discriminator relative to the first batch of 10 speech sample data is used to train the acoustic feature encoder and automatic speech recognition decoder once simultaneously. Then, the above training process is repeated using the second batch of 10 training data until the acoustic feature encoder and automatic speech recognition decoder have been trained 10 times, at which point the training ends.

[0146] Example 3

[0147] Figure 5 This is a flowchart of a speech recognition method provided in Embodiment 3 of the present invention. This embodiment of the invention is applicable to situations where content recognition and extraction are performed on speech information expressed in natural language form acquired under different environments and accent scenarios. The method can be executed by a speech recognition device, which can be implemented by software and / or hardware, and can be configured in a speech recognition equipment. Optionally, the speech recognition device can be a laptop, desktop computer, or smart tablet, etc., and this embodiment of the invention does not impose any limitations on this.

[0148] like Figure 5 As shown in the figure, the speech recognition method provided by this embodiment of the invention specifically includes the following steps:

[0149] S501, Obtain the voice information to be processed.

[0150] In this embodiment, the speech information to be processed can be specifically understood as speech acquired externally, expressed in natural language, and whose content information needs to be extracted. It is understood that this embodiment of the invention does not limit the acquisition environment of the speech information to be processed, the accent of the person providing the speech information, or whether the person's expression is affected by pathological reasons.

[0151] Specifically, when a user needs to perform voice recognition, the voice acquisition device in the voice recognition device can be triggered to acquire the given voice information and determine the acquired voice information as the voice information to be processed. Optionally, the voice acquisition device can be a microphone or other devices with voice acquisition function, and this embodiment of the invention does not limit this.

[0152] S502. Input the speech information to be processed into the pre-trained speech recognition model, and determine the speech recognition result based on the model output of the speech recognition model.

[0153] The speech recognition model is trained using the model training method proposed in any of the above embodiments.

[0154] Specifically, the speech information to be processed is input into a pre-trained speech recognition model. Through the acoustic feature encoder, feature fusion module, and automatic speech recognition decoder in the speech recognition model, the speech information to be processed is decoupled into speech content features and acoustic style features in sequence. The decoupled features are then re-fused according to the recognition requirements of the automatic speech recognition decoder to obtain fused features. The automatic speech recognition decoder then performs text sequence mapping on the fused features to obtain the model output result with text information. The model output result can then be determined as the speech recognition result for semantic recognition of the speech information to be processed.

[0155] The technical solution of this embodiment adds a speech information decoder to the speech recognition model used to identify and extract content-related information in natural language. This decoder decouples the input speech information into speech content features and acoustic style features. This allows the speech recognition model to extract content-related and non-content-related features separately during feature extraction, rather than treating them as a whole for spectral feature extraction and recognition. This enables more accurate extraction of text-related speech content features, which are then accurately determined by the automatic speech recognition decoder. Because the acoustic feature encoder decouples acoustic style features unrelated to speech content, the recognition accuracy of the speech recognition model is less affected by different accents and environmental information in the input speech information. This allows the speech recognition model to improve the robustness of the speech recognition process and the accuracy of the recognition results by utilizing the rich acoustic information in the speech data.

[0156] Example 4

[0157] Figure 6 This is a schematic diagram of the structure of a speech recognition model training device provided in Embodiment 4 of the present invention, as shown below. Figure 6 As shown, there are information decoupling module 61, adversarial loss determination module 62, fusion feature determination module 63, speech loss determination module 64, and model training module 65.

[0158] The system includes the following modules: Information decoupling module 61, which inputs sample speech information from the speech recognition training samples into the untrained speech recognition model, decoupling the sample speech information using an acoustic feature encoder to determine the sample speech content features and sample speech acoustic style features; Adversarial loss determination module 62, which uses an acoustic feature discriminator to perform acoustic feature discrimination on the sample speech content features to determine the adversarial loss; Fusion feature determination module 63, which uses a feature fusion module to perform feature fusion on the sample speech content features and sample speech acoustic style features to determine the fused sample speech features; Speech loss determination module 64, which uses an automatic speech recognition decoder to extract text information from the fused sample speech features, determines the sample text extraction result, and determines the automatic speech recognition loss based on the sample text extraction result and the speech recognition training samples; and Model training module 65, which substitutes the adversarial loss and the automatic speech recognition loss into a pre-constructed total loss function and completes the training of the speech recognition model based on the total loss function.

[0159] The technical solution of this invention adds a speech information decoder to the speech recognition model used to identify and extract content-related information in natural language. This decoder decouples the input speech information into speech content features and acoustic pattern features. An acoustic feature discriminator is also added to train the feature separation capability of the speech information decoder. This allows the speech recognition model to extract features that are sufficiently separated from content-related and non-content-related information during training, rather than extracting and recognizing spectral features as a whole. This enables more accurate extraction of text-related speech content features, which are then accurately determined by the automatic speech recognition decoder. Because the acoustic feature encoder decouples acoustic pattern features unrelated to speech content, the recognition accuracy of the speech recognition model is less affected by different accents and environmental information in the input speech information. This allows the speech recognition model to utilize the rich acoustic information in the speech data, improving the robustness and accuracy of the trained speech recognition model during the speech recognition process.

[0160] Optionally, the speech recognition model also includes a feature enhancement module. Correspondingly, the fusion feature determination module 63 is further configured to: enhance the acoustic style features of the sample speech through the feature enhancement module, and fuse the generated enhanced style features with the sample speech content features through a pre-built feature fusion module to determine the fused sample speech features; the feature enhancement is used to expand the acoustic coverage of the acoustic style features.

[0161] Optionally, the speech recognition model may also include a reconstruction module.

[0162] Correspondingly, the speech recognition model training device also includes: a reconstruction loss determination module, which is used to reconstruct speech information by using the reconstruction module to reconstruct the content features and acoustic style features of the sample speech, determine the reconstructed sample speech information, and determine the reconstruction loss based on the reconstructed sample speech information and the sample speech information.

[0163] Correspondingly, model training module 65 is also used for:

[0164] The adversarial loss, automatic speech recognition loss, and reconstruction loss are substituted into the pre-constructed total loss function, and the speech recognition model is trained based on the total loss function.

[0165] Optionally, the speech recognition model may also include a mutual information estimator.

[0166] Correspondingly, the speech recognition model training device also includes: a mutual information loss determination module, which is used to calculate the cross-entropy of the sample speech content features and the sample speech acoustic style features through a mutual information estimator to determine the mutual information loss.

[0167] Correspondingly, model training module 65 is also used for:

[0168] The adversarial loss, automatic speech recognition loss, and mutual information loss are substituted into the pre-constructed total loss function, and the speech recognition model is trained based on the total loss function.

[0169] Optionally, the speech recognition model can be trained based on the total loss function, including:

[0170] In a fixed speech recognition model, all modules except the acoustic feature discriminator are trained with the goal of minimizing adversarial loss.

[0171] A fixed acoustic feature discriminator is used to train the automatic speech recognition decoder and feature fusion module with the goal of minimizing the automatic speech recognition loss. The modules other than the automatic speech recognition decoder and feature fusion module are trained based on the total loss function.

[0172] Optionally, the modules other than the automatic speech recognition decoder and feature fusion module are trained based on the total loss function, including:

[0173] The acoustic feature encoder is trained with the goal of minimizing the total loss function;

[0174] The mutual information estimator is trained with the goal of minimizing the mutual information loss.

[0175] The speech recognition model training device provided in this embodiment of the invention can execute the model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0176] Example 5

[0177] Figure 7 This is a schematic diagram of the structure of a speech recognition method device provided in Embodiment 5 of the present invention, as shown below. Figure 7 As shown, the speech recognition device includes an information acquisition module 71 and a recognition result determination module 72.

[0178] The information acquisition module 71 is used to acquire the speech information to be processed; the recognition result determination module 72 is used to input the speech information to be processed into a pre-trained speech recognition model and determine the speech recognition result based on the model output of the speech recognition model; wherein the speech recognition model is trained by the model training method provided in any of the above embodiments.

[0179] The speech recognition device provided in the embodiments of the present invention can execute the speech recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0180] Example 6

[0181] Figure 8 This is a schematic diagram of a voice recognition device according to Embodiment Six of the present invention. The voice recognition device 80 can represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, communication base stations, and other suitable computers. The voice recognition device 80 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0182] like Figure 8As shown, the voice recognition device 80 includes at least one processor 81 and a memory, such as a read-only memory (ROM) 82 and a random access memory (RAM) 83, communicatively connected to the at least one processor 81. The memory stores computer programs executable by the at least one processor. The processor 81 can perform various appropriate actions and processes based on the computer program stored in the ROM 82 or loaded from storage unit 88 into the RAM 83. The RAM 83 can also store various programs and data required for the operation of the voice recognition device 80. The processor 81, ROM 82, and RAM 83 are interconnected via a bus 84. An input / output (I / O) interface 85 is also connected to the bus 84.

[0183] Multiple components in the voice recognition device 80 are connected to the I / O interface 85, including: an input unit 86, such as a keyboard, mouse, etc.; an output unit 87, such as various types of displays, speakers, etc.; a storage unit 88, such as a disk, optical disk, etc.; and a communication unit 89, such as a network card, modem, wireless transceiver, etc. The communication unit 89 allows the voice recognition device 80 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0184] Processor 81 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 81 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 81 performs the various methods and processes described above, such as speech recognition methods.

[0185] In some embodiments, the speech recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 88. In some embodiments, part or all of the computer program may be loaded and / or installed on the speech recognition device 80 via ROM 82 and / or communication unit 89. When the computer program is loaded into RAM 83 and executed by processor 81, one or more steps of the speech recognition method described above may be performed. Alternatively, in other embodiments, processor 81 may be configured to perform the speech recognition method by any other suitable means (e.g., by means of firmware).

[0186] Optionally, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as provided in any embodiment of the present invention.

[0187] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0189] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a speech recognition device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the speech recognition device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0191] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0192] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0193] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0194] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A model training method, characterized in that, The speech recognition model includes at least an acoustic feature encoder, a feature fusion module, an acoustic feature discriminator, and an automatic speech recognition decoder, and the method includes: The sample speech information from the speech recognition training samples is input into the untrained speech recognition model, and the acoustic feature encoder decouples the sample speech information to determine the sample speech content features and sample speech acoustic style features. The acoustic feature discriminator is used to perform acoustic feature discrimination on the sample speech content features to determine the adversarial loss; The feature fusion module performs feature fusion on the sample speech content features and the sample speech acoustic style features to determine the fused sample speech features. The automatic speech recognition decoder extracts text information from the fused sample speech features, determines the sample text extraction result, and determines the automatic speech recognition loss based on the sample text extraction result and the speech recognition training samples. The adversarial loss and the automatic speech recognition loss are substituted into the pre-constructed total loss function, and the speech recognition model is trained based on the total loss function.

2. The model training method according to claim 1, characterized in that, The speech recognition model further includes: a feature enhancement module; Accordingly, after determining the sample speech content features and sample speech acoustic style features, the method further includes: The feature enhancement module enhances the acoustic style features of the sample speech, and the pre-built feature fusion module fuses the generated enhanced style features with the sample speech content features to determine the fused sample speech features; the feature enhancement is used to expand the acoustic coverage of the acoustic style features.

3. The model training method according to claim 2, characterized in that, The speech recognition model further includes: a reconstruction module; Accordingly, after determining the sample speech content features and sample speech acoustic style features, the method further includes: The reconstruction module reconstructs speech information from the sample speech content features and the sample speech acoustic style features, determines the reconstructed sample speech information, and determines the reconstruction loss based on the reconstructed sample speech information and the sample speech information. Accordingly, substituting the adversarial loss and the automatic speech recognition loss into the pre-constructed total loss function, and training the speech recognition model based on the total loss function, further includes: The adversarial loss, the automatic speech recognition loss, and the reconstruction loss are substituted into the pre-constructed total loss function, and the speech recognition model is trained based on the total loss function.

4. The model training method according to claim 2, characterized in that, The speech recognition model further includes: a mutual information estimator; Accordingly, after determining the sample speech content features and sample speech acoustic style features, the method further includes: The mutual information loss is determined by calculating the cross-entropy of the sample speech content features and the sample speech acoustic style features using the mutual information estimator. Accordingly, substituting the adversarial loss and the automatic speech recognition loss into the pre-constructed total loss function, and training the speech recognition model based on the total loss function, further includes: The adversarial loss, the automatic speech recognition loss, and the mutual information loss are substituted into the pre-constructed total loss function, and the speech recognition model is trained based on the total loss function.

5. The model training method according to any one of claims 1-4, characterized in that, The training of the speech recognition model based on the total loss function includes: Fix all modules in the speech recognition model except for the acoustic feature discriminator, and train the acoustic feature discriminator with the goal of minimizing the adversarial loss; The acoustic feature discriminator is fixed, and the automatic speech recognition decoder and the feature fusion module are trained with the goal of minimizing the automatic speech recognition loss. The modules other than the automatic speech recognition decoder and the feature fusion module are trained based on the total loss function.

6. The model training method according to claim 5, characterized in that, The training of all modules except the automatic speech recognition decoder and the feature fusion module based on the total loss function includes: The acoustic feature encoder is trained with the objective of minimizing the total loss function; The mutual information estimator is trained with the goal of minimizing the mutual information loss.

7. A speech recognition method, characterized in that, include: Acquire the voice information to be processed; The speech information to be processed is input into a pre-trained speech recognition model, and the speech recognition result is determined based on the model output of the speech recognition model. The speech recognition model is trained using the model training method described in any one of claims 1-6.

8. A voice recognition device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the speech recognition method of claim 7.

9. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the speech recognition method as described in claim 7.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the speech recognition method as described in claim 7.

Images