A speech recognition method and apparatus, and a storage medium

Abstract

The embodiment of the application provides a speech recognition method, which comprises: obtaining to-be-recognized speech data; wherein the to-be-recognized speech data is speech data comprising at least one dialect; determining speech acoustic features and dialect embedding features corresponding to the to-be-recognized speech data; inputting the speech acoustic features and the dialect embedding features into an encoding network to obtain a feature sequence corresponding to the to-be-recognized speech data; wherein the encoding network comprises at least one layer of encoder, each of the at least one layer of encoder comprises a gating network, a shared expert network and a plurality of private expert networks; generating a recognition result corresponding to the to-be-recognized speech data according to the feature sequence, dynamically selecting a corresponding private expert network to process acoustic encoding features corresponding to the to-be-recognized speech data through a weight value output by the gating network in the encoding network, and modeling common features among different dialects through the shared expert network, so as to improve the accuracy of speech recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus, and storage medium. Background Technology

[0002] Currently, most speech recognition research focuses on Mandarin recognition, which has abundant data resources, and has achieved good recognition performance. Research on dialects mostly concentrates on single dialects, typically improving speech recognition performance by incorporating dialect speech data from the target region.

[0003] However, when recognizing speech data containing multiple dialects, common recognition schemes fail to adequately model the acoustic and linguistic differences between the dialects, leading to a decrease in recognition rate. To improve speech recognition accuracy, a separate speech recognition system needs to be built for each dialect, which is cumbersome and reduces terminal performance. Summary of the Invention

[0004] This application provides a speech recognition method, apparatus, and storage medium that can effectively improve the accuracy of speech recognition while ensuring terminal performance.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a speech recognition method, the speech recognition method comprising:

[0007] Acquire speech data to be recognized; wherein the speech data to be recognized includes at least one dialect;

[0008] Determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified;

[0009] The speech acoustic features and the dialect embedding features are input into the coding network, and the feature sequence corresponding to the speech data to be recognized is output; wherein, the coding network includes at least one encoder layer, and each of the at least one encoder layer includes a gating network, a shared expert network and multiple private expert networks;

[0010] The recognition result corresponding to the speech data to be recognized is generated based on the feature sequence.

[0011] Secondly, embodiments of this application provide a speech recognition device, which includes: an acquisition unit, a determination unit, an output unit, and a generation unit;

[0012] The acquisition unit is used to acquire speech data to be recognized; wherein, the speech data to be recognized includes at least one dialect;

[0013] The determining unit is used to determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified;

[0014] The output unit is used to input the speech acoustic features and the dialect embedding features into the coding network and output the feature sequence corresponding to the speech data to be recognized; wherein, the coding network includes at least one encoder layer, and each of the at least one encoder layer includes a gating network, a shared expert network and multiple private expert networks;

[0015] The generation unit is used to generate a recognition result corresponding to the speech data to be recognized based on the feature sequence.

[0016] Thirdly, embodiments of this application provide a voice recognition device, which includes: a processor and a memory; wherein,

[0017] The memory is used to store computer programs that can run on the processor;

[0018] The processor is configured to execute the speech recognition method described above when running the computer program.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium, characterized in that the storage medium stores computer program code, which, when executed by a computer, implements the speech recognition method as described above.

[0020] This application provides a speech recognition method, apparatus, and storage medium. The method includes: the speech recognition apparatus acquiring speech data to be recognized; wherein the speech data to be recognized includes speech data comprising at least one dialect; determining the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized; inputting the speech acoustic features and dialect embedding features into an encoding network, and outputting a feature sequence corresponding to the speech data to be recognized; wherein the encoding network includes at least one encoder layer, each encoder layer including a gating network, a shared expert network, and multiple private expert networks; and generating a recognition result corresponding to the speech data to be recognized based on the feature sequence. Therefore, a speech recognition device can first acquire the speech data to be recognized and determine the corresponding speech acoustic features and dialect embedding features. Then, the speech acoustic features and dialect embedding features are input into the coding network to obtain the feature sequence corresponding to the speech data to be recognized. Among these features, the speech recognition device can process speech data including different types of dialects through a specific private expert network, thereby improving the accuracy of speech recognition while ensuring terminal performance. At the same time, a shared expert network can also process the acoustic coding features corresponding to the speech acoustic features, and can model the common features between different dialects, thereby further improving the accuracy of speech recognition. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the speech recognition method proposed in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the encoder structure proposed in an embodiment of this application. Figure 1 ;

[0023] Figure 3 This is a schematic diagram of the encoder structure proposed in an embodiment of this application. Figure 2 ;

[0024] Figure 4 This is a schematic diagram of the architecture of the speech recognition method proposed in the embodiments of this application;

[0025] Figure 5 This is a schematic diagram of the composition structure of the voice recognition device proposed in the embodiments of this application. Figure 1 ;

[0026] Figure 6 This is a schematic diagram of the composition structure of the voice recognition device proposed in the embodiments of this application. Figure 2 . Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0028] Currently, most speech recognition research focuses on Mandarin recognition, which has abundant data resources and has achieved good recognition performance, widely used in people's daily lives. However, dialects are widely distributed, and many people still habitually use dialects in their daily speech. Speech recognition systems built using standard Mandarin show a significant drop in recognition rate when encountering dialects. Furthermore, dialects vary greatly between different regions, with differences manifesting in phonetics, vocabulary, and grammar, particularly in phonetics. According to modern popular classification, modern Chinese dialects can be divided into seven major dialect areas: Northern Mandarin, Wu Mandarin, Xiang Mandarin, Hakka Mandarin, Min Mandarin, Yue Mandarin, Gan Mandarin, and Jin Mandarin. Within complex dialect areas, some can be further subdivided into several sub-dialects. Current research on dialects mostly focuses on individual dialects, typically improving speech recognition performance by incorporating dialectal data from the target region. This requires building a separate speech recognition system for each dialect, which is not only cumbersome to implement but also degrades terminal performance. Common recognition schemes cannot effectively model the acoustic and linguistic differences of various dialects, resulting in a decrease in recognition rate.

[0029] To address the issue that common speech recognition technologies cannot simultaneously balance terminal performance and speech recognition accuracy, this application provides a speech recognition method, apparatus, and storage medium. The method includes: a speech recognition apparatus acquiring speech data to be recognized; wherein the speech data to be recognized includes speech data comprising at least one dialect; determining the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized; inputting the speech acoustic features and dialect embedding features into an encoding network, and outputting a feature sequence corresponding to the speech data to be recognized; wherein the encoding network includes at least one encoder layer, each encoder layer including a gating network, a shared expert network, and multiple private expert networks; and generating a recognition result corresponding to the speech data to be recognized based on the feature sequence. Therefore, a speech recognition device can first acquire the speech data to be recognized and determine the corresponding speech acoustic features and dialect embedding features. Then, the speech acoustic features and dialect embedding features are input into the coding network to obtain the feature sequence corresponding to the speech data to be recognized. Among these features, the speech recognition device can process speech data including different types of dialects through a specific private expert network, thereby improving the accuracy of speech recognition while ensuring terminal performance. At the same time, a shared expert network can also process the acoustic coding features corresponding to the speech acoustic features, and can model the common features between different dialects, thereby further improving the accuracy of speech recognition.

[0030] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0031] Example 1

[0032] This application provides a speech recognition method. Figure 1 This is a schematic diagram of the speech recognition method proposed in the embodiments of this application, such as... Figure 1 As shown, the method for terminal to perform speech recognition may include the following steps:

[0033] Step 101: Obtain the speech data to be recognized; wherein, the speech data to be recognized is speech data including at least one dialect.

[0034] In the embodiments of this application, the speech recognition device can first acquire speech data to be recognized. The speech data to be recognized can be speech data including at least one dialect, for example, speech data including Northern dialect, Wu dialect, Xiang dialect, Hakka dialect, and Min dialect. This application does not specifically limit the specific dialect categories included in the speech data to be recognized.

[0035] It should be noted that, in the embodiments of this application, the voice recognition device can be any device with storage function, including but not limited to mobile phones, personal computers (PCs), tablet computers, wearable smart devices, smart cameras, etc.

[0036] It should be noted that, in the embodiments of this application, there are multiple ways to acquire the voice data to be recognized. For example, the voice data can be acquired in real time through the voice acquisition module in the voice recognition device, or it can be acquired from the voice data pre-stored in the voice recognition device, or it can be acquired from the application in the voice recognition device. This application does not specifically limit the way to acquire the voice data to be recognized.

[0037] Step 102: Determine the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized.

[0038] In the embodiments of this application, after acquiring the speech data to be recognized, the speech recognition device can determine the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized.

[0039] Furthermore, in the embodiments of this application, when the speech recognition device determines the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized, it can first obtain the speech acoustic features corresponding to the speech data to be recognized; then, it can extract features from the speech acoustic features to obtain the dialect embedding features corresponding to the speech data to be recognized.

[0040] Furthermore, in the embodiments of this application, the speech recognition device can extract the speech acoustic features corresponding to the speech data to be recognized. For example, these features can be directly generated through operations such as spectrum framing, time-frequency conversion, and filtering. Commonly used features include Mel-scale Frequency Cepstral Coefficients (MFCC) and filter bank-based features (Fbank). This application does not specifically limit the extraction method.

[0041] Furthermore, in the embodiments of this application, after the speech recognition device obtains the speech acoustic features corresponding to the speech data to be recognized, it can also perform feature extraction on the speech acoustic features to obtain the dialect embedding features corresponding to the speech data to be recognized.

[0042] Step 103: Input the speech acoustic features and dialect embedding features into the coding network to obtain the feature sequence corresponding to the speech data to be recognized; wherein, the coding network includes at least one encoder layer, and each encoder layer includes a gating network, a shared expert network and multiple private expert networks.

[0043] It should be noted that, in the embodiments of this application, the speech recognition device can dynamically select the corresponding private expert network to process the acoustic coding features corresponding to the speech data to be recognized by the weight value output by the gating network. In other words, speech data including different types of dialects can be processed by a specific private expert network.

[0044] It should be noted that, in the embodiments of this application, the speech recognition device can improve the accuracy of speech recognition by modeling the common features between different dialects through a shared expert network.

[0045] In the embodiments of this application, after determining the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized, the speech recognition device can input the speech acoustic features and dialect embedding features into the encoding network to obtain the feature sequence corresponding to the speech data to be recognized; wherein, the encoding network includes at least one encoder layer, and each encoder layer includes a gating network, a shared expert network, and multiple private expert networks.

[0046] It should be noted that, in the embodiments of this application, Figure 2 This is a schematic diagram of the encoder structure proposed in an embodiment of this application. Figure 1 ,like Figure 2 As shown, the encoder includes a gating network, a shared expert network, and multiple private expert networks. The number of private expert networks can be n, and this application does not specify a particular number of private expert networks.

[0047] It should be noted that, in the embodiments of this application, the shared expert network and the private expert network can be the same neural network or different neural networks. The neural network can refer to a two-layer feedforward neural network (FNN) or other neural networks, such as a deep neural network (DNN). This application does not specifically limit the type of neural network.

[0048] It should be noted that, in the embodiments of this application, Figure 3 This is a schematic diagram of the encoder structure proposed in an embodiment of this application. Figure 2 ,like Figure 3 As shown, the encoder includes a gating network, a shared expert network, and multiple private expert networks, as well as a non-expert network layer.

[0049] It should be noted that, in the embodiments of this application, the non-expert network layer can be composed of a self-attention mechanism, normalization with residuals, or other neural networks, such as Long Short-Term Memory (LSTM). This application does not impose specific limitations on the structure of the non-expert network layer.

[0050] Furthermore, in the embodiments of this application, when the speech recognition device inputs the speech acoustic features and dialect embedding features into the coding network and outputs the feature sequence corresponding to the speech data to be recognized, it can first input the speech acoustic features and dialect embedding features into the first layer encoder to determine the feature output sequence of the first layer encoder; then input the feature output sequence of the (N-1)th layer encoder and the dialect embedding features into the Nth layer encoder to determine the feature output sequence of the Nth layer encoder; where N is an integer greater than or equal to 2; and the feature output sequence of the last layer encoder is determined as the feature sequence.

[0051] It should be noted that in the embodiments of this application, the structure of each layer encoder is the same.

[0052] Furthermore, in the embodiments of this application, when the speech recognition device inputs the speech acoustic features and dialect embedding features into the first layer encoder and determines the output sequence of the first layer encoder, it can first determine the acoustic coding features corresponding to the speech acoustic features through a non-expert network layer; input the acoustic coding features and dialect embedding features into a gating network to determine multiple weight values ​​corresponding to multiple private expert networks, and generate the output sequence of the private expert networks based on the multiple weight values ​​and acoustic coding features; then input the acoustic coding features into a shared expert network to determine the output sequence of the shared expert network; and determine the feature output sequence based on the output sequence of the private expert networks and the output sequence of the shared expert networks.

[0053] It should be noted that, in the embodiments of this application, the speech recognition device determines the acoustic coding features corresponding to the speech acoustic features through a non-expert network layer. This can refer to obtaining the acoustic coding features corresponding to the speech acoustic features by performing up-dimensional encoding on the speech acoustic features through a non-expert network layer.

[0054] Furthermore, in the embodiments of this application, when inputting acoustic coding features and dialect embedding features into a gating network to determine multiple weight values ​​corresponding to multiple private expert networks, the acoustic coding features and dialect embedding features can be concatenated first to obtain concatenated features; then the concatenated features are input into the gating network to obtain multiple weight values ​​corresponding to multiple private expert networks.

[0055] For example, in an embodiment of this application, the input to the gating network is a vector concatenated from acoustic coding features and dialect embedding features. When the number of private expert networks in each encoder layer is n, the output of the gating unit can be a vector containing n probability values, where the n probability values ​​correspond to the weight values ​​of the private expert networks. The input-output relationship of the gating unit can be expressed by the following formula:

[0056]

[0057] Where e represents the dialect embedding feature, o l-1 Let l represent the acoustic coding features output by the previous non-expert network layer, and l denote the l-th layer encoder. For the learnable parameters of the l-th layer gated network, This is the output of the gating network.

[0058] For example, suppose the first-layer encoder has four private expert networks, namely private expert network 1, private expert network 2, private expert network 3, and private expert network 4; if the output vector of the gating network is {0.6, 0.2, 0.1, 0.1}, where each probability value corresponds to the weight value of the selected private expert network, that is, the weight value of private expert network 1 is 0.6, the weight value of private expert network 2 is 0.2, the weight value of private expert network 3 is 0.1, and the weight value of private expert network 4 is 0.1. This application does not specifically limit the number of private expert networks.

[0059] It should be noted that, in the embodiments of this application, the speech recognition device uses the vector concatenated with acoustic coding features and dialect embedding features as the input of the gating network. This can guide the gating network to select the corresponding private expert network more accurately. At the same time, the dialect embedding features can provide the gating network with a discriminative feature representation, making the weight values ​​of the private expert network output by the gating network more differentiated. This enables specific private experts to process the acoustic coding features corresponding to specific types of speech data.

[0060] It should be noted that, in the embodiments of this application, after determining multiple weight values ​​corresponding to multiple private expert networks, the speech recognition device can generate the output sequence of the private expert network based on the multiple weight values ​​and acoustic coding features.

[0061] It should be noted that, in the embodiments of this application, when generating the output sequence of a private expert network based on multiple weight values ​​and acoustic coding features, the target private expert network can be determined from multiple private expert networks based on multiple weight values ​​first; then the acoustic feature encoding is input into the target private expert network to obtain the original feature sequence; and then the original feature sequence is weighted according to the weights corresponding to the target private expert network to determine the output sequence of the private expert network.

[0062] For example, suppose the first-layer encoder has three private expert networks, namely private expert 1, private expert 2, and private expert 3; if the output vector of the gating network is {0.6, 0.2, 0.1}, that is, the weight value corresponding to private expert network 1 is 0.6, the weight value corresponding to private expert network 2 is 0.2, and the weight value corresponding to private expert network 3 is 0.1, when the speech recognition device determines the target private expert network from multiple private expert networks based on multiple weight values, it can choose the private expert network with the highest weight value as the target private expert network, that is, private expert network 1 with the corresponding weight value of 0.6 as the target private expert network, or it can choose the two private expert networks with the highest weight values ​​as the target private expert network, or it can choose all private expert networks as the target private expert network. The number of private expert networks selected is not specifically limited in this application.

[0063] Furthermore, in the embodiments of this application, the speech recognition device can input acoustic feature encoding into a target private expert network to obtain an original feature sequence. The original feature sequence can then be obtained by processing the acoustic encoded features by the target private expert.

[0064] Furthermore, in the embodiments of this application, after the speech recognition device obtains the original feature sequence, it can perform weighted processing on the original feature sequence according to the weights corresponding to the target private expert network in the first layer encoder to determine the output sequence of the private expert network.

[0065] For example, the original feature sequence can be multiplied by the weights corresponding to the target private expert network in the first-layer encoder to determine the output sequence of the private expert network. The output sequence of the private expert network can be represented by the following formula:

[0066]

[0067] in, Let represent the output (original feature sequence) of the l-th layer target private expert network, where i represents the ID of the target private expert network. This represents the weight value corresponding to the l-th layer target private expert network. This represents the output sequence of the l-th layer private expert network.

[0068] It should be noted that, in the embodiments of this application, the speech recognition device selects the corresponding private expert network according to the weight value of the private expert network output by the gating network, and then processes the acoustic coding features corresponding to the speech data of different dialects, so as to realize that a specific private expert network processes the acoustic coding features corresponding to specific types of speech data, thereby improving the accuracy of speech recognition.

[0069] Furthermore, in the embodiments of this application, after the speech recognition device determines the output sequence of the private expert network, it can determine the feature output sequence based on the output sequence of the private expert network and the output sequence of the shared expert network, including: determining the feature output sequence based on the output sequence of the private expert network and the output sequence of the shared expert network.

[0070] Specifically, the speech recognition device determines the feature output sequence based on the output sequence of the private expert network and the output sequence of the shared expert network, which can be represented by the following formula:

[0071]

[0072] in, This represents the output sequence of the l-th layer shared expert network. Let y represent the output sequence of the l-th layer private expert network, λ represent the weighting coefficients of the shared expert network output, which can be experimental parameters. l This represents the mixed output sequence (feature output sequence) of the l-th layer.

[0073] It should be noted that, in the embodiments of this application, after obtaining the feature output sequence of the first layer, the speech recognition device can send the feature output sequence of the first layer to the next layer encoder.

[0074] It should be noted that, in the embodiments of this application, the speech recognition device determines the feature output sequence through the output sequence of a private expert network and the output sequence of a shared expert network. The input of the shared expert network is the acoustic coding feature corresponding to the speech data to be recognized. Assuming that the speech data to be recognized includes speech data of Guizhou dialect and Sichuan dialect, which have certain similarities in pronunciation, the shared expert network can be used to model the common features between Guizhou dialect and Sichuan dialect, which can further improve the accuracy of speech recognition, especially the recognition accuracy of low-resource dialects.

[0075] Furthermore, in the embodiments of this application, the speech recognition device inputs the output sequence of the (N-1)th layer encoder and the dialect embedding features into the Nth layer encoder. When determining the output sequence of the Nth layer encoder, the acoustic coding features corresponding to the output sequence of the (N-1)th layer encoder can be determined first through a non-expert network layer. The acoustic coding features and the dialect embedding features are then input into a gating network to determine multiple weight values ​​corresponding to multiple private expert networks. Based on the multiple weight values ​​and the acoustic coding features, the output sequence of the private expert networks is generated. Simultaneously, the acoustic coding features can be input into a shared expert network to determine the output sequence of the shared expert network. The feature output sequence is determined based on the output sequences of the private expert networks and the shared expert network.

[0076] For example, when N equals 2, the speech recognition device can input the output sequence of the first layer encoder and the dialect embedding feature into the second layer encoder, and then determine the output sequence of the second layer encoder. Specifically, firstly, the acoustic coding feature corresponding to the output sequence of the first layer encoder is determined through the non-expert network layer; then, the acoustic coding feature and the dialect embedding feature are input into the gating network to determine multiple weight values ​​corresponding to multiple private expert networks. The method by which the speech recognition device determines multiple weight values ​​corresponding to multiple private expert networks in the second layer encoder is the same as the method by which it determines the weight values ​​corresponding to multiple private expert networks in the first layer encoder, and can also be expressed by formula (1); the speech recognition device generates the output sequence of the private expert network based on the multiple weight values ​​and the acoustic coding feature. The output sequence of the private expert network can also be expressed by formula (2); the acoustic coding feature is input into the shared expert network to determine the output sequence of the shared expert network; the feature output sequence is determined based on the output sequence of the private expert network and the output sequence of the shared expert network, which can also be expressed by formula (3).

[0077] Further, in the embodiments of this application, the speech recognition device inputs the output sequence of the (N-1)th layer encoder and the dialect embedding feature into the Nth layer encoder to determine the output sequence of the Nth layer encoder, including: determining the acoustic coding feature corresponding to the output sequence of the (N-1)th layer encoder through a non-expert network layer; firstly, the acoustic coding feature and the dialect embedding feature can be input into a gating network to determine multiple weight values ​​corresponding to multiple private expert networks. The method for determining multiple weight values ​​corresponding to multiple private expert networks in the Nth layer encoder is the same as the method for determining the weight values ​​corresponding to multiple private expert networks in the first layer encoder, which can be represented by formula (1). Based on the multiple weight values ​​and the acoustic coding feature, the output sequence of the private expert network is generated. The method for generating the output sequence of the private expert network in the Nth layer encoder is the same as that of the first layer encoder, which can be represented by formula (2). The acoustic coding feature is input into a shared expert network to determine the output sequence of the shared expert network. Then, the feature output sequence is determined based on the output sequence of the private expert network and the output sequence of the shared expert network, which is the same as the method for determining the feature output sequence of the first layer encoder, which can be represented by formula (3).

[0078] Step 104: Generate the recognition result corresponding to the speech data to be recognized based on the feature sequence.

[0079] In the embodiments of this application, after obtaining the feature output sequence, i.e. the feature sequence, of the last encoder layer, the speech recognition device can generate the recognition result corresponding to the speech data to be recognized based on the feature sequence.

[0080] It should be noted that, in the embodiments of this application, the speech recognition device can input the feature sequence to the decoder to obtain the recognition result corresponding to the speech data to be recognized; that is, to obtain the text information corresponding to the speech data including at least one dialect.

[0081] Therefore, it can be seen that speech recognition devices can dynamically select specific private expert networks to process speech data containing multiple dialects by using the weight values ​​output by the gating network in the coding network, without having to build a corresponding speech recognition system for each dialect. This improves the accuracy of speech recognition while ensuring terminal performance. At the same time, the shared expert network can also process the acoustic coding features corresponding to the speech data to be recognized, and can model the common features between different dialects, thereby further improving the accuracy of speech recognition.

[0082] This application provides a speech recognition method, which includes: a speech recognition device acquiring speech data to be recognized; wherein the speech data to be recognized includes speech data containing at least one dialect; determining the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized; inputting the speech acoustic features and dialect embedding features into an encoding network, and outputting a feature sequence corresponding to the speech data to be recognized; wherein the encoding network includes at least one encoder layer, each encoder layer including a gating network, a shared expert network, and multiple private expert networks; and generating a recognition result corresponding to the speech data to be recognized based on the feature sequence. Therefore, the speech recognition device can first acquire the speech data to be recognized and determine the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized, and then input the speech acoustic features and dialect embedding features into the encoding network to obtain the feature sequence corresponding to the speech data to be recognized. The speech recognition device can process speech data including different types of dialects through a specific private expert network, thereby improving the accuracy of speech recognition while ensuring terminal performance. Simultaneously, the shared expert network can also process the acoustic coding features corresponding to the speech acoustic features, modeling the common features between different dialects, thereby further improving the accuracy of speech recognition.

[0083] Example 2

[0084] Based on the above embodiments, another embodiment of this application provides a speech recognition method that can be applied to a system (speech recognition model) with multi-dialect speech recognition performance. Based on the idea of ​​hybrid experts (hybrid expert networks), different expert networks (private expert networks) are introduced, where each expert (private expert network) can be understood as handling a specific task, with different tasks corresponding to different dialect categories. The speech recognition method proposed in this embodiment will be further described below.

[0085] It should be noted that, in the embodiments of this application, Figure 4 This is a schematic diagram of the architecture of the speech recognition method proposed in the embodiments of this application, as shown below. Figure 4 As shown in the diagram, the architecture of the speech recognition method includes an acoustic feature calculation module, a dialect recognizer, an encoding network, and a decoder. The encoding network can include an N-layer encoder, which includes a gating network, a shared expert network, and multiple private expert networks, as well as a non-expert network layer. The number of private experts is n.

[0086] It should be noted that, in the embodiments of this application, the system (speech recognition model) for multi-dialect speech recognition performance mainly includes the following modules: (1) Acoustic feature calculation module: This module is used to extract the acoustic features of the target speech data (speech data to be recognized), mainly generated directly through operations such as spectrum framing, time-frequency conversion, and filtering. Commonly used features include Mel-frequency cepstral coefficients and Fbank. (2) Dialect recognizer module: This module is mainly used to extract the embedding features of dialects. This module consists of a pre-trained dialect recognizer. The speech acoustic features are processed by the dialect recognizer to obtain high-dimensional embedding features of the dialect. These features (dialect embedding features) are concatenated with the acoustic coding features and then sent to the gate control network. (3) Speech recognition network: This network is mainly used to output the text prediction results of the target speech (speech data to be recognized); This application adopts a sequence-based end-to-end network structure. A complete speech recognition network generally includes two parts: (1) Encoding module (encoding network), which encodes the speech acoustic features of each frame of speech data into higher-dimensional features to realize the modeling of acoustic information. In this application, this module is composed of multi-layer encoders with the same structure; (2) Decoding module, which mainly models the language information and is responsible for decoding the output of the encoding module into the corresponding text. This application does not make specific limitations on the network structure of the decoding module.

[0087] Furthermore, in the embodiments of this application, each layer of encoder has the same network structure, mainly including the following two major modules: (1) Non-hybrid expert network (MoE) module (non-expert network layer). This module is responsible for encoding the calculated speech acoustic feature sequence into higher-dimensional features to perform high-dimensional modeling of acoustic information. This application does not limit the network structure. For example, self-attention mechanism and normalization with residuals can be used to form the non-expert layer (non-expert network layer). (2) Improved hybrid expert network (MoE) module. This module consists of a gating network, multiple private experts (private expert networks), and a shared expert (shared expert network). The basic idea of ​​MoE is to select which experts (private expert networks) need to be activated by the current input through the gating network during the training and inference process, and determine the weight of each expert (private expert network). Generally speaking, the expert network (private expert network) here can be a conventional neural network. This application does not specifically limit the type of private expert network.

[0088] It should be noted that in the embodiments of this application, the idea of ​​MoE (Hybrid Expert Network Module) is applied to the multi-dialect recognition task and targeted improvements are made. First, there are acoustic and linguistic differences between different dialects. Feeding different types of dialects into a single model cannot effectively model the differences in the task. Therefore, for different types of dialects, this application uses different expert networks to model them, referred to here as private expert networks. On the other hand, knowledge between different dialects is correlated. Such individual modeling methods ignore the correlation between dialect knowledge, making it difficult to perform knowledge transfer between languages, especially for some dialect data with low resources, which is difficult to model effectively. This application introduces a shared expert network to model the common features of different dialects. For different types of dialect input data, the gating network dynamically selects the corresponding private expert model (private expert network) according to the different types of input. In order to guide the gating network to select the corresponding private expert (private expert network) more accurately, this application feeds the dialect embedding features as input into the gating network.

[0089] Furthermore, in the embodiments of this application, the target speech (speech data to be recognized) is data that mixes multiple dialects. First, acoustic feature calculation is performed on the target speech (speech data to be recognized) to obtain the corresponding acoustic features. Then, these features are fed into a non-hybrid expert network layer (non-expert network layer) for high-dimensional encoding to obtain the acoustic coding features of that layer. These acoustic coding features are fed into a series of private expert sub-models (private expert networks) and shared expert sub-models (shared expert networks). This application sets up an L-layer encoder, with each encoder layer having n private experts (private expert networks), one shared expert (shared expert network), and one gating network. The gating network is used to determine the output weights of the corresponding private experts (private expert networks), and the dimension of the output vector is equal to the number of private expert networks set. In this application, the input vector of the gating unit is the vector concatenated from the dialect embedding features and the acoustic coding features. The input-output relationship of the gating unit can be represented by Equation 1.

[0090] Further, in the embodiments of this application, the output value of the gating network is a vector consisting of n probability values, where n is equal to the number of private expert networks, and the value represents the probability of the corresponding private expert network being selected. Assume that the encoder layer has four private expert networks: private expert 1, private expert 2, private expert 3, and private expert 4. The output vector of the gating network is {0.6, 0.2, 0.1, 0.1}. Each probability value corresponds to the weight of the selected private expert network. This application does not limit the number of private experts selected. For example, the expert model (private expert network) with the highest probability among the four private expert models can be selected as the target sub-model (target private expert network) for acoustic coding feature processing; the two expert models (private expert networks) with the highest probability can also be selected as target sub-models (target private expert networks) for acoustic coding feature processing; or all private expert models (private expert networks) can be selected as target sub-models (target private expert networks) for acoustic coding feature processing.

[0091] Furthermore, in the embodiments of this application, the speech recognition device selects the private expert network corresponding to the highest output probability value of the gating network. After selecting the private expert model (private expert network), the acoustic coding features are fed into the selected private expert sub-model (private expert network). The output sequence of the private expert sub-model (private expert network) is multiplied by the corresponding weight value to obtain the output of the hybrid private expert model (private expert network output sequence), as shown in Formula 2.

[0092] It should be noted that, in the embodiments of this application, the speech recognition device selects the corresponding private expert network according to the weight value of the private expert network output by the gating network, and then processes the acoustic coding features corresponding to the speech data of different dialects, so as to realize that the specific private expert network processes the acoustic coding features corresponding to specific types of speech data, thereby improving the accuracy of speech recognition.

[0093] Furthermore, in the embodiments of this application, the speech recognition device can introduce a shared expert sub-model (shared expert network) in each encoder layer to model the common features between different dialects. For certain types of dialect data, such as Guizhou dialect and Sichuan dialect, there are certain similarities in pronunciation. Even dialects with large differences in pronunciation, such as Cantonese and Mandarin, still have some linguistic similarities. Therefore, using a shared expert network to uniformly model the input dialect can effectively improve the modeling capability of multi-dialect networks. In addition, some dialect data is scarce and the annotation cost is expensive, making it difficult for the model to effectively model a small amount of data. It is necessary to learn relevant knowledge from resource-rich speech data. Introducing a shared expert network can effectively improve the recognition performance of some low-resource dialects, thereby improving the accuracy of speech recognition.

[0094] It should be noted that, in the embodiments of this application, the output of the shared expert sub-model (shared expert network) and the output of the private expert network are weighted to obtain a mixed output sequence (feature output sequence) and sent to the next layer encoder, as shown in Formula 3.

[0095] It should be noted that, in the embodiments of this application, the speech recognition device determines the feature output sequence through the output sequence of a private expert network and the output sequence of a shared expert network. The input of the shared expert network is the acoustic coding feature corresponding to the speech data to be recognized. Assuming that the speech data to be recognized includes speech data of Yunnan dialect and Sichuan dialect, which have certain similarities in pronunciation, the shared expert network can be used to model the common features between Yunnan dialect and Sichuan dialect, which can further improve the accuracy of speech recognition, especially the recognition accuracy of low-resource dialects.

[0096] It should be noted that, in the embodiments of this application, after the speech recognition device obtains the feature output sequence, i.e., the feature sequence, of the last encoder layer, it can generate the recognition result corresponding to the speech data to be recognized based on the feature sequence.

[0097] It should be noted that, in the embodiments of this application, the speech recognition device can input the feature sequence to the decoder to obtain the recognition result corresponding to the speech data to be recognized; that is, to obtain the text information corresponding to the speech data including at least one dialect.

[0098] It should be noted that in the embodiments of this application, the speech recognition device concatenates the dialect embedding features and acoustic coding features before sending them to the gate control unit. The reasons for this design are mainly twofold: (1) Since the acoustic coding features output by the non-hybrid expert network (non-expert network layer) do not have dialect category information, the output probability of the gate control network will not be discriminative. Furthermore, since the acoustic coding features of each encoder layer are different, this will also lead to a large difference in the output values ​​of each gate control network. That is, for the same input data, each encoder layer selects a different private expert network. (2) For different types of dialect input data, the dialect embedding features can provide a discriminative representation, thereby guiding the output weight values ​​of the gate control network to be more differentiated. Moreover, the dialect embedding features input to the gate control networks of different coding layers are the same, which makes the output weight values ​​of the gate control networks of different coding layers similar. By introducing dialect embedding features to guide the gate control network, a specific private expert network can be used to process specific types of input data.

[0099] This application provides a speech recognition method, which includes: a speech recognition device acquiring speech data to be recognized; wherein the speech data to be recognized includes speech data containing at least one dialect; determining the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized; inputting the speech acoustic features and dialect embedding features into an encoding network, and outputting a feature sequence corresponding to the speech data to be recognized; wherein the encoding network includes at least one encoder layer, each encoder layer including a gating network, a shared expert network, and multiple private expert networks; and generating a recognition result corresponding to the speech data to be recognized based on the feature sequence. Therefore, the speech recognition device can first acquire the speech data to be recognized and determine the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized, and then input the speech acoustic features and dialect embedding features into the encoding network to obtain the feature sequence corresponding to the speech data to be recognized. The speech recognition device can process speech data including different types of dialects through a specific private expert network, thereby improving the accuracy of speech recognition while ensuring terminal performance. Simultaneously, the shared expert network can also process the acoustic coding features corresponding to the speech acoustic features, modeling the common features between different dialects, thereby further improving the accuracy of speech recognition.

[0100] Example 3

[0101] Based on the above embodiments, this application provides a speech recognition device. Figure 5 Schematic diagram of the composition structure of a voice recognition device Figure 1 ,like Figure 5 As shown, the speech recognition device 10 includes: an acquisition unit 11, a determination unit 12, and a generation unit 13;

[0102] The acquisition unit 11 is used to acquire speech data to be recognized; wherein, the speech data to be recognized includes at least one dialect;

[0103] The determining unit 12 is used to determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified;

[0104] The acquisition unit 11 is further configured to input the speech acoustic features and the dialect embedding features into the coding network to acquire the feature sequence corresponding to the speech data to be recognized; wherein, the coding network includes at least one encoder layer, and each of the at least one encoder layer includes a gating network, a shared expert network, and multiple private expert networks;

[0105] The generation unit 13 is used to generate the recognition result corresponding to the speech data to be recognized based on the feature sequence.

[0106] In the embodiments of this application, further, Figure 6 Schematic diagram of the composition structure of a voice recognition device Figure 2 ,like Figure 6 As shown, the voice recognition device 10 proposed in this application embodiment may further include a processor 14, a memory 15 storing executable instructions of the processor 14, and further, the voice recognition device 10 may also include a communication interface 16 and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0107] In the embodiments of this application, the processor 14 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this application embodiment does not specifically limit this. The voice recognition device 10 may also include a memory 15, which can be connected to the processor 14. The memory 15 is used to store executable program code, which includes computer operation instructions. The memory 15 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0108] In embodiments of this application, bus 17 is used to connect communication interface 16, processor 14, and memory 15, as well as the mutual communication between these devices.

[0109] In embodiments of this application, memory 15 is used to store instructions and data.

[0110] Furthermore, in an embodiment of this application, the processor 14 is used by the speech recognition device to acquire speech data to be recognized; wherein the speech data to be recognized is speech data including at least one dialect;

[0111] Determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified;

[0112] The speech acoustic features and the dialect embedding features are input into the coding network to obtain the feature sequence corresponding to the speech data to be recognized; wherein, the coding network includes at least one encoder layer, and each of the at least one encoder layer includes a gating network, a shared expert network and multiple private expert networks;

[0113] The recognition result corresponding to the speech data to be recognized is generated based on the feature sequence.

[0114] In practical applications, the aforementioned memory 15 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 14.

[0115] This application provides a speech recognition device that acquires speech data to be recognized, wherein the speech data to be recognized includes at least one dialect; determines the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized; inputs the speech acoustic features and dialect embedding features into an encoding network, and outputs a feature sequence corresponding to the speech data to be recognized; wherein the encoding network includes at least one encoder layer, each encoder layer including a gating network, a shared expert network, and multiple private expert networks; and generates a recognition result corresponding to the speech data to be recognized based on the feature sequence. Thus, the speech recognition device first acquires the speech data to be recognized and determines the speech acoustic features and dialect embedding features corresponding to the speech data to be recognized, and then inputs the speech acoustic features and dialect embedding features into the encoding network to acquire the feature sequence corresponding to the speech data to be recognized. The speech recognition device can process speech data including different types of dialects through a specific private expert network, thereby improving the accuracy of speech recognition while ensuring terminal performance. Simultaneously, the shared expert network can also process the acoustic coding features corresponding to the speech acoustic features, and can model the common features between different dialects, thereby further improving the accuracy of speech recognition.

[0116] This application provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the speech recognition method described above.

[0117] Specifically, the program instructions corresponding to a speech recognition method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a speech recognition method in the storage media are read or executed by an electronic device, the following steps are included:

[0118] Acquire speech data to be recognized; wherein, the speech data to be recognized includes speech data of at least one dialect;

[0119] Determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified;

[0120] The speech acoustic features and the dialect embedding features are input into the coding network to obtain the feature sequence corresponding to the speech data to be recognized; wherein, the coding network includes at least one encoder layer, and each of the at least one encoder layer includes a gating network, a shared expert network and multiple private expert networks;

[0121] The recognition result corresponding to the speech data to be recognized is generated based on the feature sequence.

[0122] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0123] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A speech recognition method, characterized in that, The method includes: Acquire speech data to be recognized; wherein, the speech data to be recognized is speech data including at least one dialect; Determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified; The speech acoustic features and the dialect embedding features are input into the encoding network to obtain the feature sequence corresponding to the speech data to be recognized; wherein, the encoding network includes at least one encoder layer, each encoder layer including a gated network, a shared expert network and multiple private expert networks; each encoder layer includes a non-expert network layer. Specifically, the speech acoustic features and the dialect embedding features are input into the coding network to obtain the feature sequence corresponding to the speech data to be recognized, including: The speech acoustic features and the dialect embedding features are input into the first layer encoder to determine the feature output sequence of the first layer encoder. The feature output sequence of the (N-1)th layer encoder and the dialect embedding features are input into the Nth layer encoder to determine the feature output sequence of the Nth layer encoder; where N is an integer greater than or equal to 2. The feature output sequence of the last encoder layer is determined as the feature sequence; The step of inputting the speech acoustic features and the dialect embedding features into the first layer encoder to determine the feature output sequence of the first layer encoder includes: The acoustic coding features corresponding to the speech acoustic features are determined through the non-expert network layer; The acoustic coding features and the dialect embedding features are input into the gated network to determine multiple weight values ​​corresponding to the multiple private expert networks, and the output sequence of the private expert network is generated based on the multiple weight values ​​and the acoustic coding features. The acoustically coded features are input into the shared expert network to determine the output sequence of the shared expert network; the feature output sequence is then determined based on the output sequence of the private expert network and the output sequence of the shared expert network. The recognition result corresponding to the speech data to be recognized is generated based on the feature sequence.

2. The method according to claim 1, characterized in that, The step of inputting the output sequence of the (N-1)th layer encoder and the dialect embedding features into the Nth layer encoder to determine the output sequence of the Nth layer encoder includes: The acoustic coding features corresponding to the output sequence of the (N-1)th layer encoder are determined by the non-expert network layer. The acoustic coding features and the dialect embedding features are input into the gated network to determine multiple weight values ​​corresponding to the multiple private expert networks, and the output sequence of the private expert network is generated based on the multiple weight values ​​and the acoustic coding features. The acoustic coding features are input into the shared expert network to determine the output sequence of the shared expert network; The feature output sequence is determined based on the output sequence of the private expert network and the output sequence of the shared expert network.

3. The method according to claim 1 or 2, characterized in that, The step of inputting the acoustic coding features and the dialect embedding features into the gating network to determine multiple weight values ​​corresponding to the multiple private expert networks includes: The acoustic coding features and the dialect embedding features are concatenated to obtain concatenated features; The concatenated features are input into the gating network to obtain multiple weight values ​​corresponding to the multiple private expert networks.

4. The method according to claim 1 or 2, characterized in that, The step of generating the output sequence of the private expert network based on the plurality of weight values ​​and the acoustic coding features includes: The target private expert network is determined from the plurality of private expert networks based on the plurality of weight values; The acoustic feature encoding is input into the target private expert network to obtain the original feature sequence; The original feature sequence is weighted according to the weights corresponding to the target private expert network to determine the output sequence of the private expert network.

5. The method according to claim 1 or 2, characterized in that, Determining the feature output sequence based on the output sequence of the private expert network and the output sequence of the shared expert network includes: The feature output sequence is determined based on the output sequence of the private expert network and the output sequence of the shared expert network.

6. The method according to claim 1, characterized in that, Determining the speech acoustic features and dialect embedding features corresponding to the speech data to be identified includes: The speech data to be recognized is input into the acoustic feature calculation module to obtain the speech acoustic features corresponding to the speech data to be recognized. The dialect embedding features corresponding to the speech data to be recognized are obtained by extracting features from the acoustic features of the speech using a dialect recognizer.

7. The method according to claim 1, characterized in that, The step of generating the recognition result corresponding to the speech data to be recognized based on the feature sequence includes: The feature sequence is input into the decoder to obtain the recognition result corresponding to the speech data to be recognized.

8. A voice recognition device, characterized in that, The speech recognition device includes: an acquisition unit, a determination unit, and a generation unit; The acquisition unit is used to acquire speech data to be recognized; wherein, the speech data to be recognized includes at least one dialect; The determining unit is used to determine the speech acoustic features and dialect embedding features corresponding to the speech data to be identified; The acquisition unit is further configured to input the speech acoustic features and the dialect embedding features into an encoding network to acquire the feature sequence corresponding to the speech data to be recognized; wherein, the encoding network includes at least one encoder layer, each of the at least one encoder layer including a gated network, a shared expert network, and multiple private expert networks; each of the at least one encoder layer includes a non-expert network layer; wherein, inputting the speech acoustic features and the dialect embedding features into the encoding network to acquire the feature sequence corresponding to the speech data to be recognized includes: inputting the speech acoustic features and the dialect embedding features into a first encoder layer to determine the feature output sequence of the first encoder layer; inputting the feature output sequence of the (N-1)th encoder layer and the dialect embedding features into the Nth encoder layer to determine the feature output sequence of the Nth encoder layer; wherein, N is an integer greater than or equal to 2; and determining the feature output sequence of the last encoder layer as the feature sequence; wherein, The step of inputting the speech acoustic features and the dialect embedding features into a first-layer encoder to determine the feature output sequence of the first-layer encoder includes: determining the acoustic coding features corresponding to the speech acoustic features through the non-expert network layer; inputting the acoustic coding features and the dialect embedding features into the gating network to determine multiple weight values ​​corresponding to the multiple private expert networks, and generating the output sequence of the private expert networks based on the multiple weight values ​​and the acoustic coding features; inputting the acoustic coding features into the shared expert network to determine the output sequence of the shared expert network; and determining the feature output sequence based on the output sequence of the private expert networks and the output sequence of the shared expert networks. The generation unit is used to generate a recognition result corresponding to the speech data to be recognized based on the feature sequence.

9. A voice recognition device, characterized in that, The voice recognition device comprises a processor and a memory; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1-7 when running the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer program code, which, when executed by a computer, performs the method described in any one of claims 1-7.

Images