Speech recognition methods, speech recognition model training methods, equipment and media

Abstract

This application discloses a speech recognition method, a speech recognition model training method, an apparatus, and a medium. The method includes: acquiring speech to be recognized and acquiring a trained speech recognition model; the speech recognition model includes an encoding network and a decoding network; in each stage of encoding the speech to be recognized using the encoding network, firstly, the predicted attribute category to which the speech to be recognized belongs is obtained under the target speech attribute, and then encoding is performed based on the predicted attribute category under the target speech attribute to obtain a first encoded feature; the first encoded feature is decoded based on the decoding network to obtain the recognized text of the speech to be recognized; wherein, the speech recognition model is adjusted at least based on a first loss, the first loss representing the difference between the preset attribute category of the sample speech annotation under the target speech attribute and the sample attribute category obtained by the speech recognition model. Through the above method, this application can improve the accuracy of speech recognition while reducing costs.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to speech recognition methods, speech recognition model training methods, devices and media. Background Technology

[0002] Automatic speech recognition, or speech recognition for short, is a technology that refers to a computer processor receiving speech signals, processing them, and converting them into text information that is understandable to humans. This technology is widely used in mobile phone voice assistants, input method software, in-vehicle navigation, and various AI wearable devices, and has significant application value. Mixture-Of-Experts (MoE) networks are currently a hot area in deep learning. Deep learning models can expand the number of model parameters while maintaining the original level of computation, thus significantly improving the overall performance of the model.

[0003] Existing speech recognition models based on expert sparse networks employ an unsupervised training method, where samples are randomly assigned to different experts during training. Model developers cannot determine the characteristics of the samples assigned to each expert, nor the required number of experts. Consequently, a large number of samples and experts are often needed for training, resulting in very high training costs. Furthermore, due to the unsupervised training method, the random assignment of samples to experts leads to lower feature accuracy compared to assigning samples to experts based on their attributes. This, in turn, results in low speech recognition accuracy for the speech recognition model. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a speech recognition method, a speech recognition model training method, a device, and a medium that can improve the accuracy of speech recognition while reducing costs.

[0005] To address the aforementioned technical problems, this application provides a speech recognition method, comprising: acquiring speech to be recognized and acquiring a trained speech recognition model; wherein the speech recognition model includes an encoding network and a decoding network; in each stage of encoding the speech to be recognized using the encoding network, firstly classifying the predicted attribute category to which the speech to be recognized belongs under the target speech attribute, and then encoding based on the predicted attribute category under the target speech attribute to obtain a first encoded feature; and decoding based on the decoding network to obtain the recognized text of the speech to be recognized; wherein the speech recognition model is adjusted at least based on a first loss, the first loss representing the difference between the preset attribute category of the sample speech annotation under the target speech attribute and the sample attribute category obtained by the speech recognition model.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a method for training a speech recognition model. This method includes: acquiring sample speech; in each stage of encoding the sample speech using the encoding network of the speech recognition model, first classifying the sample speech under target speech attributes to obtain the sample attribute category to which the sample speech belongs, and then encoding based on the sample attribute category under the target speech attributes to obtain a first sample encoding feature; decoding the first sample encoding feature using the decoding network of the speech recognition model to obtain the recognized text of the sample speech; determining a first loss based on the difference between the sample attribute category to which the sample speech belongs and the preset attribute category labeled on the sample speech, and determining a recognition loss based on the difference between the recognized text of the sample speech and the preset text labeled on the sample speech; and adjusting the network parameters of the speech recognition model based at least on the first loss and the recognition loss.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising a memory and a processor coupled to each other, wherein the memory stores program instructions; and the processor executes the program instructions stored in the memory to implement the methods provided in the first or second aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium for storing program instructions that can be executed to implement the methods provided in the first or second aspect.

[0009] The beneficial effects of this application are as follows: Unlike the prior art, in the training stage of the speech recognition model, this application adjusts the network parameters of the speech recognition model based on at least the first loss. The first loss represents the difference between the preset attribute category of the sample speech annotation and the sample attribute category obtained by the speech recognition model under the target speech attribute. That is, in the training stage, the target speech attribute and the preset attribute category can be determined first. In other words, the model developer can clearly define the characteristics of the sample assigned to each expert and the number of experts that need to be set. By adopting a supervised approach, a smaller number of samples and experts can be used to train the speech recognition model, thereby reducing costs.

[0010] Furthermore, by utilizing the encoding network of the trained speech recognition model in each stage of encoding the speech to be recognized, the predicted attribute category of the speech to be recognized is first determined by classifying it under the target speech attributes. Then, encoding is performed based on the predicted attribute category under the target speech attributes to obtain the first encoded feature. Finally, the first encoded feature is decoded based on the decoding network of the trained speech recognition model to obtain the recognized text of the speech to be recognized. By determining the predicted attribute category first and then encoding based on the predicted attribute category during the encoding process, a first encoded feature with higher accuracy can be obtained, thereby improving the speech recognition accuracy of the speech recognition model. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the first embodiment of the speech recognition method provided in this application;

[0012] Figure 2 This is a schematic diagram of the framework of one embodiment of the coding network provided in this application;

[0013] Figure 3 This is a flowchart illustrating the second embodiment of the speech recognition method provided in this application;

[0014] Figure 4 This is a schematic diagram of another implementation of the coding network provided in the application;

[0015] Figure 5 This is a flowchart illustrating the first embodiment of the speech recognition model training method provided in this application;

[0016] Figure 6 This is a flowchart illustrating the second embodiment of the speech recognition model training method provided in this application;

[0017] Figure 7 This is a schematic diagram of the framework of one embodiment of the speech recognition device provided in this application;

[0018] Figure 8 This is a schematic diagram of the framework of one embodiment of the speech recognition model training device provided in this application;

[0019] Figure 9 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application.

[0020] Figure 10 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that the embodiments of this application contain descriptions involving "first," "second," etc., which are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Please refer to the following: Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the speech recognition method provided in this application. Figure 2 This is a schematic diagram of the framework of an embodiment of the coding network provided in this application; the method includes:

[0025] S11: Obtain the speech to be recognized and obtain the trained speech recognition model.

[0026] In one embodiment, the speech recognition model may include an encoding network and a decoding network, such as a transformer-based or convolution-augmented transformer-based model. The encoding network may include a plurality of sequentially connected first network blocks, which may be associated with target speech attributes. These target speech attributes may include language, phonemes, attention span, importance, etc., and can be set by the user; no specific limitation is made here. Understandably, multiple first network blocks may be associated with the same target speech attribute or different target speech attributes. The plurality of first network blocks are used to perform different stages of encoding. In one specific implementation, several first network blocks can be divided into at least one network group. Each first network block in the same network group is associated with the same target speech attribute. Assuming that the speech recognition model is a transformer model containing 12 first network blocks, the shallow layers of the model often contain more phoneme information, and the deep layers contain more speech information. Based on this, the shallow first network blocks can be associated with phonemes. For example, 1-3 first network blocks can be divided into the first grid group and associated with phonemes; 4-5 first network blocks can be divided into the second grid group and associated with importance; 6-8 first network blocks can be divided into the third grid group and associated with attention field; and 9-12 first network blocks can be divided into the fourth grid group and associated with language.

[0027] The decoding network may also include a number of sequentially connected second network blocks. Similarly, these second network blocks can be associated with target speech attributes; multiple second network blocks can be associated with the same target speech attribute or different target speech attributes. These second network blocks are used to perform various stages at any given decoding time. Understandably, these second network blocks can be identical to or different from the first network blocks.

[0028] S12: In each stage of encoding the speech to be recognized using the coding network, the predicted attribute category to which the speech to be recognized belongs is first obtained by classifying under the target speech attribute, and then the first coding feature is obtained by encoding based on the predicted attribute category under the target speech attribute.

[0029] In one embodiment, the encoding network includes a plurality of sequentially connected first network blocks, each of which is used to perform various stages of encoding the speech to be recognized. Taking a transformer-based speech recognition model as an example, the transformer may include a plurality of transformer blocks (i.e., first network blocks), and each first network block may include a first classification layer and multiple first expert layers. The first classification layer is used to classify the speech under the target speech attribute. Specifically, the first classification layer can classify the speech to be recognized into the first probability of belonging to each preset attribute category under the target speech attribute. The preset attribute categories can be set according to the target speech attribute. For example, if the target speech attribute is a language, the preset attribute categories under the language may include Chinese, English, etc.; if the target speech attribute is a phoneme, the preset attribute categories under the language may include first vowel, second vowel, third vowel, fourth vowel, etc.; as another example, if the target speech attribute is the field of attention, the preset attribute categories under the language may include long field of attention and short field of attention; if the target speech attribute is importance, the preset attribute categories under the language may include important frames and unimportant frames. Multiple first expert layers correspond one-to-one with each preset attribute category under the target speech attribute; for example, if the target speech attribute is a language, which includes two preset attribute categories, Chinese and English, then a first expert layer can be set for each preset attribute category, such as a Chinese expert layer and an English expert layer; each first expert layer is used to encode the predicted attribute category based on the target speech attribute.

[0030] Specifically, the first network block corresponding to the current stage can be selected as the first target network block. For example, if the first network block corresponding to the current stage is the first first network block, then the first first network block can be used as the first target network block; similarly, if the first network block corresponding to the current stage is the last first network block, then the last first network block can be used as the first target network block. Classification is performed using the first classification layer in the first target network block to obtain the first probability that the speech to be recognized belongs to each preset attribute category under the target speech attribute. Based on the first probability that the speech to be recognized belongs to each preset attribute category, the predicted attribute category to which the speech to be recognized belongs is determined. Then, the first expert layer in the first target network block corresponding to the predicted attribute category is selected as the first target expert layer. Encoding is performed using the first target expert layer to obtain the first encoded feature.

[0031] In one embodiment, if the first network block corresponding to the current stage is the first first network block in the encoding network, the initial features of the speech to be recognized can be classified using the first classification layer of the first first network block to obtain the first probability that the speech to be recognized belongs to each preset attribute category under the target speech attribute; the preset attribute category corresponding to the largest first probability is selected as the predicted attribute category to which the speech to be recognized belongs; further, the first expert layer in the first first network block corresponding to the predicted attribute category is selected as the first target expert layer; the initial features of the speech to be recognized are encoded using the first target expert layer to obtain the first encoded features. The speech recognition model also includes an embedding layer and an attention layer, and the initial features of the speech to be recognized are obtained by processing the speech to be recognized using the embedding layer and the attention layer. The first coding feature obtained from the first first network block is input into the second first network block, and the second first network block performs the same steps as the first first network block to obtain the first coding feature output by the second first network block. This process is repeated until the first coding feature output by the last first network block is obtained. The first coding feature output by the last first network block is used as the first coding feature output by the coding network. Alternatively, residual processing and normalization operations are performed on the first coding feature output by the last first network block to obtain the first coding feature output by the coding network.

[0032] In another embodiment, for each stage corresponding to a first network block, the first classification layer in the first network block is used to classify the initial features of the speech to be recognized, and the first probability that the speech to be recognized belongs to each preset attribute category under the target speech attribute is obtained. The preset attribute category corresponding to the largest first probability is selected as the predicted attribute category to which the speech to be recognized belongs. The first expert layer in the first network block corresponding to the predicted attribute category is selected, and the initial features of the speech to be recognized are encoded using it to obtain the first encoded features. The first encoded features obtained from each first network block are fused to obtain the first encoded features finally output by the encoding network.

[0033] In one specific implementation, such as Figure 2 As shown, the encoding network includes sequentially connected self-attention layers, a first residual calculation and normalization layer, a first network block, and a second residual calculation and normalization layer. The initial features of the speech to be recognized are encoded by the encoding network to obtain the first encoded features. The first network block contains a first classification layer and several first expert layers.

[0034] S13: Decode the first encoded feature based on the decoding network to obtain the recognized text of the speech to be recognized.

[0035] In one implementation, the decoding network may employ an autoregressive decoder, a transformer decoder, or an attention-based decoder.

[0036] In one specific implementation, the decoding network employs a transformer model decoder, which includes sequentially connected attention layers, interactive attention processing layers, feedforward neural networks, fully connected layers, and normalization layers. The input to the decoding network includes the decoded characters obtained at previous decoding times and the first encoded features output by the encoding network. The decoding network uses the attention layer to perform attention processing on the features of the decoded characters obtained at previous decoding times to obtain a first feature vector. The first feature vector and the first encoded features are then used for interactive attention processing to obtain a second feature vector. After the second feature vector passes through the feedforward neural network, the fully connected layer, and the normalization layer in sequence, the decoded character at the current time can be obtained.

[0037] In another specific embodiment, the decoding network further includes several sequentially connected second network blocks. The second network blocks can be set before the interactive attention layer. That is, after processing the features of the decoded characters obtained at each previous decoding time using several second network blocks, a third feature vector is obtained. The third feature vector and the first encoded feature are then used for interactive attention processing to obtain a second feature vector.

[0038] Specifically, several second network blocks are associated with target speech attributes. Understandably, each second network block can be associated with the same target speech attribute or different target speech attributes. Each second network block includes a second classification layer for classifying based on the target speech attribute, and a second expert layer corresponding one-to-one with each preset attribute category under the target speech attribute. The second expert layers are used for decoding based on the preset attribute categories under the target speech attribute. Different second network blocks are used to execute different stages at any given decoding time.

[0039] At any given decoding time, the second network block corresponding to the current stage of that decoding time is selected as the second target network block. For example, in the first stage of the second decoding time, if the second network block corresponding to the first stage is the first second network block in the decoding network, then the first second network block is selected as the second target network block. The second classification layer in the second target network block is used to classify the decoded characters obtained from previous decoding times, obtaining the second probability that each decoded character belongs to each preset attribute category. For example, in the first decoding time, the decoded characters obtained from previous decoding times only include the start character; in the second decoding time, the decoded characters obtained from previous decoding times include both the decoded character obtained in the first decoding time and the start character. Based on the second probability that the decoded characters belong to each preset attribute category, a second expert layer in the second target network block used for decoding the decoded characters is determined as the second target expert layer. In one embodiment, the second network layer corresponding to the preset attribute category with the highest second probability can be selected as the second target expert layer; in another embodiment, any one of the second expert layers corresponding to the preset attribute categories with second probabilities greater than the preset probability can be selected as the second target expert layer. The second target expert layer is used to decode the character, obtaining the first decoded feature. If the decoding network contains only one second network block, the second network block outputs the first decoded feature, and then decodes based on the first encoded feature and the first decoded feature to obtain the decoded character at the current decoding time. If the decoding network contains multiple second network blocks, the second network block corresponding to the current stage outputs the first decoded feature, and then inputs the first decoded feature to the second network block corresponding to the next stage. The second network block corresponding to the next stage performs the same steps as the second network block corresponding to the current stage to obtain the first decoded feature output by the second network block corresponding to the next stage. This process continues until the second network block corresponding to the last stage. Decoding is performed based on the first decoded feature and the first encoded feature output by the second network block corresponding to the last stage to obtain the decoded character at the current decoding time.

[0040] After obtaining the decoded characters at each decoding moment using the above method, they can be combined to obtain the recognized text. By setting multiple second network blocks in the decoding network, the overall scale of the speech recognition model can be significantly expanded, further improving the model's computational performance.

[0041] In the above embodiments, the speech recognition model is adjusted at least based on a first loss, which represents the difference between the preset attribute category of the sample speech annotation and the sample attribute category obtained by the speech recognition model under the target speech attribute. Specifically, during training, the sample attribute categories classified by each first classification layer in the encoding network under the target speech attribute can be obtained. The first loss is determined based on the difference between the sample attribute categories and the preset attribute categories of the sample speech annotation, and the network parameters of the speech recognition model are adjusted at least based on the first loss. In other embodiments, the speech recognition model is also adjusted based on a second loss, which is determined based on the following factors: the proportion of sample characters in the sample text annotated by the sample speech under the target speech attribute belonging to each preset attribute category and the average probability of the sample characters belonging to each preset attribute category. The training process of the speech recognition model will not be described in detail here; please refer to the following text for a detailed description.

[0042] The above method allows for the generation of recognized text from speech recognition. Furthermore, because the speech recognition model is trained in a supervised manner during the training phase, the characteristics of the samples' attribute categories obtained from the recognition are clearly defined for each expert, as well as the required number of experts. This allows for training the speech recognition model with a smaller number of samples and experts, thus reducing costs. Moreover, by using the trained speech recognition model for speech recognition, and first determining the predicted attribute categories during the encoding process, followed by encoding based on these predicted attribute categories, a highly accurate first-level encoded feature can be obtained, thereby improving the speech recognition accuracy of the speech recognition model.

[0043] Please refer to the following: Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the speech recognition method provided in this application. Figure 4 This is a schematic diagram of another implementation of the coding network provided in the application; the method includes:

[0044] S31: Obtain the speech to be recognized and obtain the trained speech recognition model.

[0045] The speech recognition model includes an encoding network and a decoding network.

[0046] S32: Select the first network block corresponding to the current stage as the first target network block.

[0047] S33: Classify using the first classification layer in the first target network block to obtain the first probability that the speech to be recognized belongs to each preset attribute category under the target speech attribute.

[0048] S34: Based on the first probability that the speech to be recognized belongs to each preset attribute category, determine the predicted attribute category to which the speech to be recognized belongs.

[0049] S35: Select the first expert layer in the first target network block that corresponds to the predicted attribute category, and use it as the first target expert layer.

[0050] S36: Encode using the first target expert layer to obtain the first encoded feature.

[0051] For steps S31-S36, please refer to the first embodiment of the speech recognition method provided in this application, which will not be repeated here.

[0052] S37: Encode using a shared expert layer to obtain the second coding feature.

[0053] Furthermore, such as Figure 4 As shown, the coding network may also include a shared expert layer. Specifically, a shared expert layer may be set in each first network block. The shared expert layer is used to encode the initial features of the speech to be recognized input to the first network block or the first coding features output by the previous first network block to obtain the second coding features.

[0054] S38: The first coding feature and the second coding feature are fused to obtain the first coding feature of the final output of the first target network block.

[0055] The first coding feature output by the first target expert layer and the second coding feature output by the shared expert layer in the first network block corresponding to the current stage are fused to obtain the first coding feature finally output by the first target network block. The fusion of the first coding feature and the second coding feature can be achieved by adding or concatenating the first coding feature and the second coding feature.

[0056] S39: Decode the first encoded feature based on the decoding network to obtain the recognized text of the speech to be recognized.

[0057] For a detailed implementation of step S39, please refer to step S13 of the first embodiment of the speech recognition method provided in this application, which will not be repeated here.

[0058] Understandably, a shared expert layer can also be added to the decoding network. Specifically, a shared expert layer can be set in each second network block. The shared expert layer is used to decode the initial features of the decoded characters input to the second network block or the second decoded features output by the previous second network block to obtain the second decoded features. The first decoded features output by the second target expert layer in the current stage and the second decoded features output by the shared expert layer are fused to obtain the first decoded features finally output by the second target network block. The encoding step performed by the shared expert layer can be executed simultaneously with the encoding step performed by the first target expert layer, or it can be executed before the encoding step performed by the first target expert layer.

[0059] In this implementation, because the model developers pre-set the target speech attributes, preset attribute categories, and expert layers corresponding one-to-one with the preset attribute categories, the amount of speech data allocated to a particular expert layer for processing is relatively small, resulting in insufficient fitting ability of the speech recognition model for speech data of that preset attribute category. Therefore, adding a shared expert layer to the encoding and / or decoding network ensures that each speech data entering the network block passes through the shared expert layer. Since the shared expert layer has seen the features of each entering network block, it has a strong fitting ability, thus compensating for the poor fitting ability caused by uneven distribution of speech data.

[0060] Please see Figure 5 , Figure 5 This is a flowchart illustrating the first embodiment of the speech recognition model training method provided in this application. The method includes:

[0061] S51: Obtain sample speech.

[0062] S52: In each stage of encoding sample speech using the coding network of the speech recognition model, the sample attribute category to which the sample speech belongs is first obtained by classifying under the target speech attribute, and then the sample attribute category is encoded based on the target speech attribute to obtain the first sample coding feature.

[0063] S53: Use the decoding network of the speech recognition model to decode the encoded features of the first sample to obtain the recognized text of the sample speech.

[0064] The encoding process of the encoding network for the sample speech is the same as the encoding process of the encoding network for the speech to be recognized. The decoding process of the decoding network for the first sample encoded features is the same as the decoding process of the decoding network for the first encoded features, so it will not be described again here.

[0065] During the training phase of the speech recognition model, each sample speech is labeled with a preset attribute category. Understandably, a sample speech can be labeled with multiple preset attribute categories, such as Chinese characters or first vowel sounds. In one embodiment, the first classification layer in the first network block of the encoding network classifies the sample speech into sample attribute categories under the target speech attributes. Understandably, some first classification layers have learning parameters, i.e., the network parameters of the speech recognition model; some first classification layers may not have learning parameters. For example, language classification layers and phoneme classification layers may have learning parameters, while attention field classification layers and importance classification layers may not have learning parameters.

[0066] Taking the target speech attribute as language as an example, the first classification layer in the first network block associated with the language can be the language classification layer. That is, the first classification layer is used to determine the language of the sample speech. The calculation process of the language classification layer is as follows.

[0067]

[0068] e = argmax(softmax(r))

[0069]

[0070] in, Let X' be a D*n matrix, where D is the dimension of the hidden layers of the speech recognition model, and n is the number of languages ​​supported by the speech recognition model, i.e., the number of preset attribute categories under each language, which is also the number of first expert layers in the first network block corresponding to that language; Pooling represents the average pooling operation in the time dimension, FFNe represents the first target expert layer e that processes the initial features of the sample speech or the first encoded features output by the previous first network block; W and b are the network parameters of the language classification layer; X' T The initial features of the first network block or the first encoded features output by the previous first network block are used as input for the sample speech; X″ T The first encoded feature is the output of the first target expert layer e.

[0071] Taking the target speech attribute as phonemes as an example, the first classification layer in the first network block associated with the phonemes can be a phoneme classification layer. That is, the first classification layer is used to determine the phonemes of the sample speech. The calculation process of the phoneme classification layer is as follows.

[0072]

[0073] e T =argmax(softmax(r T ))

[0074]

[0075] in, Let D*m be the matrix, where D is the dimension of the hidden layer of the speech recognition model and m is the number of preset attribute categories under the phoneme, which is also the number of the first expert layers in the first network block corresponding to the phoneme. This represents the first target expert layer e, which processes the initial features of the sample speech or the first encoded features output by the previous first network block. t ;X′ t The initial features of the first network block are input to the sample speech, or the first encoded features output by the previous first network block; X″ t For the first target expert level e t The first encoded feature of the output.

[0076] Taking the target speech attribute as the field of interest (POI) as an example, the first classification layer in the first network block associated with the POI can be the POI classification layer. That is, the first classification layer is used to determine the attention a particular frame of the sample speech receives from other frames. Specifically, the Transformer-based model contains a self-attention module. The score calculated by the self-attention module can determine the attention a particular frame receives from other frames. By calculating the sum of the attention scores of the frame to its neighboring frames, the degree of attention the frame receives within its short field of interest can be obtained. An attention threshold is set to distinguish the length of a frame's POI. Frames with longer POIs contain unique information and can achieve higher scores when compared to distant frames, while frames with shorter POIs often only contain their own information, resulting in lower scores. Therefore, an attention threshold can be set to determine whether the attention score of a frame is greater than the threshold. If it is greater, the frame is considered to have a longer POI and is assigned to the long POI expert layer for processing; if it is less, the frame is considered to have a shorter POI and is assigned to the short POI expert layer for processing.

[0077] Taking the importance of the target speech attribute as an example, the first classification layer in the first network block associated with the importance can be an importance classification layer, that is, the first classification layer is used to determine the importance of a certain frame of the sample speech. Specifically, the sum of the attention scores of a certain speech frame to all other speech frames within a preset range can be calculated. If the sum of the attention scores is greater than a preset importance threshold, the speech frame is considered an important frame and is assigned to the important frame expert layer for processing; if the attention score is less than the preset importance threshold, the speech frame is considered an unimportant frame and is assigned to the unimportant frame expert layer for processing.

[0078] S54: Determine the first loss based on the difference between the sample attribute category to which the sample speech belongs and the preset attribute category of the sample speech annotation, and determine the recognition loss based on the difference between the recognized text of the sample speech and the preset text of the sample speech annotation.

[0079] For each first classification layer with learning parameters, the first loss is determined by the difference between the sample attribute category to which the classified sample speech belongs and the preset attribute category of the sample speech annotation. Specifically, the first loss can be the cross-entropy loss. Understandably, if the encoding network contains multiple first classification layers with learning parameters, multiple first losses can be obtained.

[0080] S55: Adjust the network parameters of the speech recognition model based at least on the first loss and the recognition loss.

[0081] The sum of multiple first losses and recognition losses is used as the total loss; the network parameters of the speech recognition model are adjusted based on the total loss.

[0082] Please see Figure 6 , Figure 6 This is a flowchart illustrating the second embodiment of the speech recognition model training method provided in this application. The method includes:

[0083] S61: Obtain sample speech.

[0084] S62: In each stage of encoding sample speech using the coding network of the speech recognition model, the sample attribute category to which the sample speech belongs is first obtained by classifying under the target speech attribute, and then the sample attribute category is encoded based on the target speech attribute to obtain the first sample coding feature.

[0085] S63: Use the decoding network of the speech recognition model to decode the encoded features of the first sample to obtain the recognized text of the sample speech.

[0086] The encoding process of the encoding network for the sample speech is the same as the encoding process of the encoding network for the speech to be recognized. The decoding process of the decoding network for the first sample encoded features is the same as the decoding process of the decoding network for the first encoded features, so it will not be repeated here. The specific implementation of steps S61-S63 can be referred to steps S51-S53 of the first implementation of the speech recognition model training method provided in this application, and will not be repeated here.

[0087] S64: Obtain the proportion of sample characters in the sample text labeled with the sample speech that belong to each of the preset attribute categories, and the average probability of sample characters belonging to each of the preset attribute categories.

[0088] In one embodiment, the decoding network further includes several sequentially connected second network blocks. To ensure that the decoded characters assigned to each second expert layer in the second network blocks are as evenly distributed as possible, a third loss function is set to constrain the distribution of each decoded character. The third loss function is shown below.

[0089]

[0090] Where n represents the number of second expert layers in the decoding network; s i P represents the proportion of each decoded character in a training set (i.e., a batch) that is assigned to the i-th second expert layer, i.e., the proportion of each decoded character in the training set belonging to the i-th sample attribute category; i The table represents the average probability of each decoded character being assigned to the i-th second expert layer, which is the average probability of each decoded character belonging to each sample attribute category.

[0091] In one embodiment, the sample characters in the sample text annotated with the sample speech can belong to the same training set. The total number of sample characters in the training set and the first number of sample characters belonging to each preset attribute category are obtained. The ratio of the first number to the total number is used as the proportion of each sample character in the sample text annotated with the sample speech belonging to a preset attribute category. For example, if the total number of sample characters is 10 and the first number of sample characters belonging to preset attribute category A is 3, then the proportion of each sample character belonging to preset attribute category A is 3 / 10. The probability of each sample character belonging to each preset attribute category is obtained, and the average probability of each sample character belonging to the same preset attribute category is calculated. For example, if the probability of sample character 1 belonging to preset attribute category A is 1 / 3 and the probability of sample character 2 belonging to preset attribute category A is 1 / 4, the average probability of the sample character belonging to preset attribute category A is obtained by averaging the two probabilities.

[0092] S65: Determine the first loss based on the difference between the sample attribute category to which the sample speech belongs and the preset attribute category of the sample speech annotation; determine the recognition loss based on the difference between the recognized text of the sample speech and the preset text of the sample speech annotation; and determine the second loss based on the proportion and average probability.

[0093] In one embodiment, determining the second loss based on the proportion and average probability includes: obtaining the product between the proportion of each sample character belonging to the same preset attribute category and the average probability of each sample character belonging to the same preset attribute category, as a probability parameter; and multiplying the sum of each first parameter by the number of sample attribute categories, as the second loss.

[0094] S66: Adjust the network parameters of the speech recognition model based on the first loss, the second loss, and the recognition loss.

[0095] In one implementation, the first loss, the second loss, and the recognition loss can be summed to obtain the total loss, and the network parameters of the speech recognition model can be adjusted based on the total loss.

[0096] In the training process of the aforementioned speech recognition model, the encoding network is trained in a supervised manner. Specifically, for each first classification layer with learning parameters, the first loss is determined by the difference between the sample attribute category of the classified sample speech and the preset attribute category of the sample speech annotation. The decoding network is trained unsupervised and constrained. Each second network block in the decoding network is classified under the target speech attribute to obtain the sample attribute category of the sample speech, and decoding is performed based on the sample attribute category under the target speech attribute. A third loss function is introduced during this process to ensure that the number of sample speech samples allocated to each second expert layer is as even as possible. Furthermore, the recognition loss is determined based on the difference between the recognized text of the sample speech and the preset text of the sample speech annotation. Finally, the parameters of the speech recognition model are adjusted based on the recognition loss, the first loss, and the second loss. Through this training, the speech recognition model can maintain good text recognition performance even with a small number of second expert layers.

[0097] Further, please refer to Figure 7 , Figure 7 This is a schematic diagram of the framework of an embodiment of the speech recognition device provided in this application. The speech recognition device 70 includes a first acquisition module 71, a first encoding module 72, and a first decoding module 73. The first acquisition module 71 is used to acquire the speech to be recognized and to acquire a trained speech recognition model. The speech recognition model includes an encoding network and a decoding network. The first encoding module 72 is used to, in each stage of encoding the speech to be recognized using the encoding network, first classify the predicted attribute category to which the speech to be recognized belongs under the target speech attribute, and then encode it based on the predicted attribute category under the target speech attribute to obtain a first encoded feature. The first decoding module 73 is used to decode the first encoded feature based on the decoding network to obtain the recognized text of the speech to be recognized. The speech recognition model is adjusted at least based on a first loss, which represents the difference between the preset attribute category of the sample speech annotation and the sample attribute category obtained by the speech recognition model under the target speech attribute.

[0098] In the above-described method, during the training phase of the speech recognition model, the network parameters of the speech recognition model are adjusted based on at least the first loss. The first loss represents the difference between the preset attribute category of the sample speech annotation and the sample attribute category obtained by the speech recognition model under the target speech attribute. That is, in the training phase, the target speech attribute and the preset attribute category can be determined first. In other words, the model developer can clearly define the characteristics of the sample assigned to each expert and the number of experts that need to be set. By adopting a supervised approach, a smaller number of samples and experts can be used to train the speech recognition model, thereby reducing costs.

[0099] Furthermore, by utilizing the encoding network of the trained speech recognition model in each stage of encoding the speech to be recognized, the predicted attribute category of the speech to be recognized is first determined by classifying it under the target speech attributes. Then, encoding is performed based on the predicted attribute category under the target speech attributes to obtain the first encoded feature. Finally, the first encoded feature is decoded based on the decoding network of the trained speech recognition model to obtain the recognized text of the speech to be recognized. By determining the predicted attribute category first and then encoding based on the predicted attribute category during the encoding process, a first encoded feature with higher accuracy can be obtained, thereby improving the speech recognition accuracy of the speech recognition model.

[0100] In one embodiment, the encoding network includes a plurality of first network blocks that are sequentially connected and associated with the target speech attribute, each used to perform different stages of encoding. The first network block associated with the target speech attribute includes a first classification layer for classifying under the target speech attribute, and a first expert layer that corresponds one-to-one with each preset attribute category under the target speech attribute. The first expert layers are used to encode based on the predicted attribute category under the target speech attribute.

[0101] In one embodiment, before classifying the predicted attribute category to which the speech to be recognized belongs under the target speech attribute, the method further includes: using the selection module of the speech recognition device 70 to select the first network block corresponding to the current stage as the first target network block;

[0102] The process of classifying the speech to be recognized under the target speech attribute to obtain the predicted attribute category includes: classifying the speech under the target speech attribute using the first classification layer in the first target network block to obtain the first probability that the speech to be recognized belongs to each preset attribute category under the target speech attribute; and determining the predicted attribute category to which the speech to be recognized belongs based on the first probability that the speech to be recognized belongs to each preset attribute category.

[0103] Encoding the predicted attribute category based on the target speech attribute to obtain the first encoded feature includes: selecting the first expert layer in the first target network block that corresponds to the predicted attribute category as the first target expert layer; and using the first target expert layer for encoding to obtain the first encoded feature.

[0104] In one embodiment, the method further includes: in response to the fact that the first network block corresponding to the current stage is not the last first network block in the coding network, inputting the first coding feature output by the first network block corresponding to the current stage to the next first network block, until the first network block corresponding to the current stage is the last first network block in the coding network; and selecting the first coding feature output by the last first network block as the first coding feature finally output by the coding network.

[0105] Therefore, by setting multiple first network blocks, the overall size of the speech recognition model can be greatly expanded, thereby improving the speech recognition performance of the speech recognition model.

[0106] In one embodiment, the first network block further includes a shared expert layer; after encoding using the first target expert layer to obtain a first encoded feature, the method further includes: encoding using the shared expert layer to obtain a second encoded feature; and fusing the first encoded feature and the second encoded feature to obtain the first encoded feature finally output by the first target network block. This step can also be performed by the first encoding module 72.

[0107] Therefore, the problem of insufficient sample fitting ability of speech recognition models can be solved by sharing an expert layer.

[0108] In one embodiment, a plurality of first network blocks are divided into at least one network group, and each first network block in the same network group is associated with the same target speech attribute.

[0109] In one embodiment, the decoding network includes a plurality of second network blocks that are sequentially connected and associated with the target speech attribute. The second network blocks associated with the target speech attribute include a second classification layer for classifying under the target speech attribute, and a second expert layer that corresponds one-to-one with each preset attribute category under the target speech attribute. The second expert layers are used to decode based on the preset attribute categories under the target speech attribute.

[0110] In one embodiment, the identified text is obtained by combining the decoded characters decoded at each decoding time, and each stage of any decoding time is executed by a different second network block. At any decoding time, the method includes: selecting the second network block corresponding to the current stage in the decoding time as the second target network block; classifying the decoded characters decoded at each previous decoding time using the second classification layer in the second target network block to obtain a second probability that each decoded character belongs to each preset attribute category; determining a second expert layer in the second target network block for decoding the decoded characters based on the second probability that the decoded characters belong to each preset attribute category, as the second target expert layer; decoding the decoded characters using the second target expert layer to obtain a first decoding feature; and decoding based on the first encoding feature and the first decoding feature to obtain the decoded character at the decoding time.

[0111] Therefore, multiple second network blocks are also set in the decoding network, which can further improve the speech recognition performance of the speech recognition model.

[0112] In one embodiment, the speech recognition model is further adjusted based on a second loss, which is determined based on the following factors: the proportion of sample characters in the sample text labeled with the sample speech belonging to each preset attribute category under the target speech attribute, and the average probability of the sample characters belonging to each preset attribute category.

[0113] Therefore, by adjusting the parameters of the speech recognition model using the second loss, the number of sample characters belonging to each preset attribute category can be made more uniform.

[0114] Please see Figure 8 , Figure 8 This is a schematic diagram of the framework of one embodiment of the speech recognition model training device provided in this application. The speech recognition model training device 80 includes a second acquisition module 81, a second encoding module 82, a second decoding module 83, a loss determination module 84, and an adjustment module 85. The second acquisition module 81 is used to acquire sample speech; the second encoding module 82 is used to, in each stage of encoding the sample speech using the encoding network of the speech recognition model, first classify the sample attribute category to which the sample speech belongs under the target speech attribute, and then encode based on the sample attribute category under the target speech attribute to obtain the first sample encoding feature; the second decoding module 83 is used to decode the first sample encoding feature using the decoding network of the speech recognition model to obtain the recognized text of the sample speech; the loss determination module 84 is used to determine the first loss based on the difference between the sample attribute category to which the sample speech belongs and the preset attribute category of the sample speech annotation, and to determine the recognition loss based on the difference between the recognized text of the sample speech and the preset text of the sample speech annotation; the adjustment module 85 is used to adjust the network parameters of the speech recognition model based at least on the first loss and the recognition loss.

[0115] In one embodiment, the second acquisition module 81 is further configured to acquire the proportion of sample characters in the sample text labeled with the sample speech belonging to each preset attribute category, and the average probability of the sample characters belonging to each preset attribute category; the loss determination module 84 is further configured to determine the second loss based on the proportion and the average probability; and the adjustment module 85 is configured to adjust the network parameters of the speech recognition model based on the first loss, the second loss and the recognition loss.

[0116] In one embodiment, determining the second loss based on proportion and average probability includes: obtaining the product between the proportion of each sample character belonging to the same preset attribute category and the average probability of each sample character belonging to the same preset attribute category, as a probability parameter; and multiplying the sum of the probability parameters by the number of preset attribute categories, as the second loss.

[0117] The above method introduces a third loss at the decoding network end, which makes the number of sample characters allocated to each second expert layer in the decoding network more even.

[0118] Please see Figure 9 , Figure 9 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application.

[0119] The electronic device 90 includes a memory 91 and a processor 92 coupled to each other. The memory 91 stores program instructions, and the processor 92 executes the program instructions stored in the memory 91 to implement the steps of any of the above-described method embodiments. In a specific implementation scenario, the electronic device 90 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 90 may also include mobile devices such as laptops and tablets, which are not limited here.

[0120] Specifically, processor 92 controls itself and memory 91 to implement the steps of any of the above-described method embodiments. Processor 92 may also be referred to as a CPU (Central Processing Unit). Processor 92 may be an integrated circuit chip with signal processing capabilities. Processor 92 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 92 may be implemented using integrated circuit chips.

[0121] Please see Figure 10 , Figure 10 This is a schematic diagram of a framework of one embodiment of the computer-readable storage medium provided in this application.

[0122] The computer-readable storage medium 100 stores program instructions 101, which, when executed by a processor, are used to implement the steps of any of the above-described method embodiments.

[0123] The computer-readable storage medium 100 may specifically be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium capable of storing computer programs. Alternatively, it may be a server storing the computer program, which can send the stored computer program to other devices for execution or can also execute the stored computer program itself.

[0124] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0130] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A voice recognition method, characterized by, The method comprises: acquiring a to-be-recognized speech and acquiring a trained speech recognition model; wherein the speech recognition model comprises an encoding network and a decoding network; in each stage of encoding the to-be-recognized speech by using the encoding network, first classifying to obtain a predicted attribute category to which the to-be-recognized speech belongs under a target speech attribute, and then encoding by using a first target expert layer corresponding to the predicted attribute category to obtain first encoding features; wherein the target speech attribute comprises at least one of a phoneme, an attention field of view, and an importance degree; the attention field of view represents the attention degree of a speech frame in the to-be-recognized speech to other speech frames different from the speech frame in the to-be-recognized speech; and the importance degree represents whether the speech frame is an important frame; decoding the first encoding features based on the decoding network to obtain recognized text of the to-be-recognized speech; wherein the speech recognition model is adjusted based on at least a first loss, and the first loss represents the difference between a preset attribute category of a sample speech label and a sample attribute category obtained by recognizing the sample speech by using the speech recognition model under the target speech attribute.

2. The method of claim 1, wherein, The encoding network comprises a plurality of first network blocks connected in sequence and associated with the target speech attribute, which are respectively used to perform different stages of encoding, and the first network block associated with the target speech attribute comprises a first classification layer used for classification under the target speech attribute, and a first expert layer corresponding to each preset attribute category under the target speech attribute one by one, which is respectively used for encoding based on the predicted attribute category under the target speech attribute.

3. The method of claim 2, wherein, Before the step of classifying to obtain the predicted attribute category to which the to-be-recognized speech belongs under the target speech attribute, the method further comprises: selecting a first network block corresponding to the current stage as a first target network block; wherein the first target expert layer is a first expert layer corresponding to the predicted attribute category selected from the first target network block; the step of classifying to obtain the predicted attribute category to which the to-be-recognized speech belongs under the target speech attribute comprises: classifying by using the first classification layer in the first target network block to obtain a first probability that the to-be-recognized speech belongs to each preset attribute category under the target speech attribute; determining the predicted attribute category to which the to-be-recognized speech belongs based on the first probability that the to-be-recognized speech belongs to each preset attribute category.

4. The method according to claim 2 or 3, characterized in that, The method further comprises: in response to the first network block corresponding to the current stage not being the last first network block in the encoding network, inputting the first encoding features output by the first network block corresponding to the current stage to the next first network block until the first network block corresponding to the current stage is the last first network block in the encoding network; selecting the first encoding features output by the last first network block as the first encoding features finally output by the encoding network.

5. The method of claim 3, wherein, The first network block further comprises a shared expert layer; After the first target expert layer is used for encoding to obtain the first encoding feature, the method further comprises: using the shared expert layer for encoding to obtain a second encoding feature; fusing the first encoding feature and the second encoding feature to obtain the first encoding feature finally output by the first target network block.

6. The method of claim 2, wherein, The plurality of first network blocks are divided into at least one network group, and each first network block in the same network group is associated with the same target speech attribute.

7. The method of claim 1, wherein, The decoding network comprises a plurality of second network blocks connected in sequence and associated with the target speech attribute, and the second network block associated with the target speech attribute comprises a second classification layer for classification under the target speech attribute, and a second expert layer corresponding to each preset attribute category under the target speech attribute, and the second expert layer is respectively used for decoding based on the preset attribute category under the target speech attribute.

8. The method of claim 1, wherein, The speech recognition model is further adjusted based on a second loss, and the second loss is determined based on the following factors: under the target speech attribute, the proportion of sample characters in the sample text annotated by the sample speech belonging to each preset attribute category and the average probability of the sample characters belonging to each preset attribute category. 9.A method for training a speech recognition model, the method comprising: Comprise: obtaining a sample speech; In each stage of encoding the sample speech by using the encoding network of the speech recognition model, first, the sample attribute category to which the sample speech belongs is classified under the target speech attribute, and then the first target expert layer corresponding to the sample attribute category is used for encoding to obtain the first sample encoding feature; wherein the target speech attribute comprises at least one of phoneme, attention field of view, and importance; the attention field of view represents the attention degree of a speech frame in the sample speech to other speech frames, and the other speech frames are frames different from the speech frame in the sample speech; the importance represents whether the speech frame is an important frame; using the decoding network of the speech recognition model to decode the first sample encoding feature to obtain the recognition text of the sample speech; based on the difference between the sample attribute category to which the sample speech belongs and the preset attribute category annotated by the sample speech, determining a first loss, and based on the difference between the recognition text of the sample speech and the preset text annotated by the sample speech, determining a recognition loss; adjusting the network parameters of the speech recognition model based on at least the first loss and the recognition loss.

10. The method of claim 9, wherein, The method further comprises: obtaining the proportion of sample characters in the sample text annotated by the sample speech belonging to each preset attribute category, and the average probability of the sample characters belonging to each preset attribute category; based on the proportion and the average probability, determining a second loss; the at least based on the first loss and the recognition loss, adjusting the parameters of the speech recognition model, comprising: based on the first loss, the second loss and the recognition loss, adjusting the network parameters of the speech recognition model.

11. An electronic device, comprising: comprising a memory storing program instructions and a processor configured to execute the program instructions to implement the speech recognition method of any one of claims 1-8 or the speech recognition model training method of any one of claims 9-10.

12. A computer-readable storage medium, characterized in that, The storage medium is configured to store program instructions executable by a processor to implement the speech recognition method of any one of claims 1-8 or the speech recognition model training method of any one of claims 9-10.

Images