Speech recognition method, related device, equipment and storage medium

Through the multi-layer fusion decoding technology of speech features, the accuracy of speech recognition under the influence of dialects and accents is solved, and the accuracy and robustness of speech recognition are improved.

CN118982986APending Publication Date: 2024-11-19IFLYTEK CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411007495.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The accuracy of existing speech recognition technology is affected when dealing with dialects and accents of different regions and educational levels, and it is difficult to effectively improve.

Method used

By obtaining the acoustic coding features, frame-level pronunciation features and segment-level pronunciation features in the speech feature, performing fusion decoding, using the first fusion feature of the acoustic coding features and the frame-level pronunciation features and the second fusion feature of the text coding features and segment-level pronunciation features and segment-level pronunciation features. Fusion features and deep fusion to improve the accuracy of speech recognition.

Benefits of technology

In the case of including accents or dialects, the accuracy of speech recognition is significantly improved, and the robustness and efficiency of speech recognition models are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982986A_ABST
    Figure CN118982986A_ABST
Patent Text Reader

Abstract

The invention discloses a voice recognition method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining an acoustic coding feature, a frame-level pronunciation feature and a segment-level pronunciation feature based on the voice feature of a to-be-recognized voice; performing prediction based on the acoustic coding features to obtain a predicted pronunciation sequence of the to-be-recognized speech, and extracting text coding features based on the predicted pronunciation sequence; obtaining a first fusion feature of the acoustic coding feature and the frame-level pronunciation feature, and obtaining a second fusion feature of the text coding feature and the segment-level pronunciation feature; and decoding based on the first fusion feature and the second fusion feature to obtain a speech recognition text of the speech to be recognized. According to the scheme, the accuracy of speech recognition can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and related devices, equipment and storage media. Background Art

[0002] With the development of end-to-end Automatic Speech Recognition (ASR) technology, the ASR framework based on the encoder-decoder (ED) structure has gradually become a mainstream solution and has been widely used in different scenarios.

[0003] However, due to factors such as region and education level, the existence of accents and dialects is inevitable. Since different dialects and accents have similar pronunciation units, they will cause crosstalk to ASR products, thus affecting the accuracy of speech recognition.

[0004] Therefore, how to improve the accuracy of speech recognition has become an urgent problem to be solved. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a speech recognition method and related devices, equipment and storage media, which can improve the accuracy of speech recognition.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a speech recognition method, which includes: obtaining acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized; predicting based on the acoustic coding features to obtain a predicted pronunciation sequence of the speech to be recognized, and extracting text coding features based on the predicted pronunciation sequence; obtaining a first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtaining a second fusion feature of the text coding features and the segment-level pronunciation features; decoding based on the first fusion feature and the second fusion feature to obtain a speech recognition text of the speech to be recognized.

[0007] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a speech recognition device, which includes an acquisition module, a prediction module, a fusion module and a decoding module; the acquisition module is used to obtain acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized; the prediction module is used to predict based on the acoustic coding features to obtain a predicted pronunciation sequence of the speech to be recognized, and extract text coding features based on the predicted pronunciation sequence; the fusion module is used to obtain a first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtain a second fusion feature of the text coding features and the segment-level pronunciation features; the decoding module is used to decode based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized.

[0008] In order to solve the above technical problem, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the speech recognition method in the above first aspect.

[0009] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the speech recognition method in the above first aspect is implemented.

[0010] The above scheme obtains acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized, and predicts based on the acoustic coding features to obtain the predicted pronunciation sequence of the speech to be recognized, and extracts text coding features based on the predicted pronunciation sequence, and then obtains the first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtains the second fusion feature of the text coding features and the segment-level pronunciation features, and then decodes based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized. Therefore, fusing the acoustic coding features with the finer-grained frame-level pronunciation features can capture deeper acoustic information related to pronunciation at the accent level, and fusing the text coding features with the coarser-grained segment-level pronunciation features can capture the relationship between different pronunciations and text information at the language level, and then using the complementarity of the first fusion feature and the second fusion feature in granularity and modality, it is helpful to promote the deep fusion between features to obtain the speech recognition text, thereby improving the accuracy of speech recognition, especially when the speech to be recognized contains an accent or dialect, the accuracy of speech recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of an embodiment of a speech recognition method of the present application;

[0012] Figure 2 A flowchart of an embodiment of a method for training a speech recognition model;

[0013] Figure 3 It is a structural diagram of an embodiment of a speech recognition model;

[0014] Figure 4 A schematic diagram of a framework of an embodiment of a speech recognition device of the present application;

[0015] Figure 5 A schematic diagram of a framework of an embodiment of an electronic device of the present application;

[0016] Figure 6 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0017] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0018] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0019] The term "multiple" herein means two or more than two. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, and may mean including any one or more elements selected from the set consisting of A, B, and C.

[0020] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech recognition method of the present application, specifically, comprising the following steps:

[0021] Step S11: Based on the speech features of the speech to be recognized, acoustic coding features, frame-level pronunciation features and segment-level pronunciation features are obtained.

[0022] In the embodiments of the present disclosure, the speech to be recognized can be collected in real time (such as online conferences, voice calls, etc.), or the speech to be recognized can be collected offline and stored in a storage medium such as a hard disk and then read from the storage medium. The acquisition channel of the speech to be recognized is not limited here. In addition, the language of the speech to be recognized may include but is not limited to: Chinese, English, German, etc., and the language of the speech to be recognized is also not limited here. It should be noted that according to the language differences of the speech to be recognized and the accent differences in the same language, the representation method of the pronunciation elements of the speech to be recognized may also be different. For example, the pronunciation elements of Chinese can be represented as pinyin letters. Under the influence of different dialects or accents, the pronunciation elements corresponding to the same Chinese character may be different. For example, the pronunciation elements of English can be represented as phonetic symbols. Under the influence of different accents, the pronunciation elements corresponding to the same word may be different. The same is true for other languages, and no examples are given here one by one.

[0023] In one implementation scenario, as a possible implementation method, in order to facilitate subsequent recognition processing, the acoustic feature information of the original speech signal of the speech to be recognized can be first extracted to obtain speech features, such as but not limited to: Fbank features (Mel filter bank), MFCC features (Mel-Frequency Cepstral Coefficients), etc. Taking the extraction of MFCC features as speech features as an example, the specific steps of speech feature extraction include pre-emphasis, framing, windowing, Fourier transform, Mel filtering, etc. The specific process can be referred to related technologies and will not be repeated here. It should be noted that the speech features can specifically include the acoustic feature information of each speech frame in the speech to be recognized.

[0024] In an implementation scenario, acoustic coding features can be obtained by encoding speech features, for example, the speech features are Fbank features, and the Fbank features can be encoded by encoders of network structures such as Transformer and Conformer to obtain acoustic coding features. It should be noted that by encoding the speech features, deeper acoustic feature information can be extracted. In other words, the acoustic coding features contain deeper acoustic feature information of each speech frame in the speech to be recognized. For example, encoding by the above-mentioned encoder can further model the relationship between speech frames based on the attention mechanism, such as for the acoustic feature information of any speech frame in the speech feature, it can be combined with the acoustic feature information of other speech frames that potentially correspond to the same pronunciation element in the speech feature for feature enhancement. For details, please refer to the technical details of the above-mentioned encoder, which will not be repeated here.

[0025] In an implementation scenario, the speech to be recognized is recognized according to the speaker recognition technology (Speaker Recognition, referred to as SR), and speaker information of different granularities can be obtained. For example, speaker information of different granularities may include speaker information corresponding to each speech frame in the speech to be recognized, and may also include speaker information corresponding to the entire segment of the speech to be recognized. In addition, the speech to be recognized may be divided into several speech segments, and the speaker information may also include speaker information corresponding to each speech segment. It should be noted that the speaker information represents the speaker pronunciation feature information of the speech to be recognized, that is, the pronunciation characteristics, which can be used to distinguish different speakers. In the embodiment of the present disclosure, the speaker pronunciation feature information corresponding to each speech frame in the speech to be recognized is used as the frame-level pronunciation feature, and the speaker pronunciation feature information of the entire segment of the speech to be recognized is used as the segment-level pronunciation feature. That is, the frame-level pronunciation feature and the segment-level pronunciation feature represent the same meaning, both of which represent the speaker pronunciation feature information, and the main difference is that the range of feature expression is different, the former is expressed at the fine-grained speech frame level, and the latter is expressed at the coarse-grained speech segment level. In addition, in order to improve the efficiency of acquiring frame-level pronunciation features and segment-level pronunciation features, the speech features of the speech to be recognized can be recognized through a specifically trained speaker recognition model to obtain frame-level pronunciation features and segment-level pronunciation features. The speaker recognition model can be trained with sample corpora of multiple specific speakers. The specific training process can be referred to related technologies, which will not be repeated here.

[0026] In a specific implementation scenario, the number of channels of the frame-level pronunciation feature is the same as the number of frames of the speech frame to be recognized. The segment-level pronunciation feature has a larger granularity than the frame-level pronunciation feature, and its number of channels is less than the number of channels of the frame-level pronunciation feature. The number of channels of the segment-level pronunciation feature can be 1, 2, 3, and so on. As a possible example, when the speaker's pronunciation features of the entire segment of speech to be recognized are used as segment-level pronunciation features, the number of channels of the segment-level pronunciation features is 1; as another possible example, when the speech to be recognized is divided into 3 speech segments, and the speaker's pronunciation features of the 3 speech segments are used as segment-level pronunciation features, the number of channels of the segment-level pronunciation features is 3. The determination of the number of channels of other numbers of segment-level pronunciation features is similar, and no examples are given here one by one. It should be noted that since the determination of the number of channels of the segment-level pronunciation features is independent of the number of frames of the speech frame to be recognized, the segment-level pronunciation features can be obtained by recognizing the speech to be recognized, as well as by recognizing other audio data of the speaker of the speech to be recognized in the corpus.

[0027] Step S12: Predicting based on the acoustic coding features to obtain a predicted pronunciation sequence of the speech to be recognized, and extracting text coding features based on the predicted pronunciation sequence.

[0028] In one implementation scenario, the acoustic coding features can be predicted by a decoder of a network structure such as DNN, Transformer, Conformer, etc. to obtain a predicted pronunciation sequence of the speech to be recognized.

[0029] In an implementation scenario, the speech to be recognized includes several speech frames, and the predicted pronunciation sequence includes pronunciation elements aligned to each speech frame. By obtaining the element embedding features (i.e., embedding) of each pronunciation element in the predicted pronunciation sequence, and encoding the element embedding features of each pronunciation element through an encoder of a network structure such as Transformer and Conformer, a text coding feature can be obtained. Therefore, by using the predicted pronunciation sequence predicted based on acoustic coding features as the data source of the text coding feature, the data in the same network can be recycled to improve data processing efficiency. It should be noted that, depending on the language of the speech to be recognized, the type of predicted pronunciation sequence is also different. For example, the pronunciation elements of Chinese are represented as pinyin letters. When the speech to be recognized is Chinese, the predicted pronunciation sequence is represented as a pinyin sequence. The pronunciation elements of English are represented as phonetic symbols. When the speech to be recognized is English, the predicted pronunciation sequence is represented as a phonetic symbol sequence. Other languages ​​are similar, and examples are not given one by one here.

[0030] Step S13: obtaining a first fusion feature of the acoustic coding feature and the frame-level pronunciation feature, and obtaining a second fusion feature of the text coding feature and the segment-level pronunciation feature.

[0031] In one implementation scenario, the speech to be recognized includes several speech frames, the predicted pronunciation sequence includes pronunciation elements aligned to each speech frame, and the acoustic coding features include first coding features corresponding to each speech frame. In order to obtain the first fusion feature, the first coding features corresponding to the same speech frame and the frame-level pronunciation features can be fused to obtain the first fusion feature. Therefore, by fusing the acoustic coding features with the finer-grained frame-level pronunciation features, it is helpful to capture deeper acoustic information related to pronunciation at the accent level, while capturing the pronunciation differences of different speakers in each speech frame, thereby improving the robustness of speech recognition.

[0032] For example, the speech to be recognized includes T speech frames, and the acoustic coding feature X fa The number of channels of the frame-level pronunciation feature F is consistent with and aligned with the number of speech frames, and the acoustic coding feature X fa The dimension of the frame-level pronunciation feature F is D-dimensional, that is, X fa ∈R T×D , F∈R T×D , which includes the acoustic coding feature X faIncluding the first coding features corresponding to each speech frame, and then fusing the first coding features corresponding to the same speech frame with the frame-level pronunciation features to obtain the first fusion feature

[0033]

[0034] Among them, concat(X fa ,F) is 2D, and after the linear layer of FusionLayer, the final The dimension of is still D. Of course, the acoustic encoding feature X fa The dimensions of the frame-level pronunciation feature F may be inconsistent. For example, the acoustic coding feature X fa The dimension of the frame-level pronunciation feature F can be D-dimensional, and the dimension of the frame-level pronunciation feature F can be E-dimensional. Here, the acoustic coding feature X fa The dimensions of the frame-level pronunciation feature F are not specifically limited. In addition, the linear layer of FusionLayer concat(X fa ,F) is adjusted to D dimension only as an example. You can also adjust concat(X fa ,F) is adjusted to other target dimensions, and the target dimensions are not specifically limited here.

[0035] In one implementation scenario, the speech to be recognized includes several speech frames, the predicted pronunciation sequence includes pronunciation elements aligned to each speech frame, and the text encoding feature includes a second encoding feature extracted from the pronunciation elements aligned to each speech frame. In order to obtain the second fusion feature, the number of channels of the segment-level pronunciation feature can be expanded to the same as the total number of several speech frames to obtain an expanded pronunciation feature, and then the second encoding feature corresponding to the same speech frame and the expanded pronunciation feature are fused to obtain the second fusion feature. Therefore, by fusing the text encoding feature with the coarser-grained segment-level pronunciation feature, it is helpful to capture the relationship between different pronunciations and text information at the language level, while capturing the pronunciation differences of different speakers in the entire speech, thereby improving the robustness of speech recognition.

[0036] For example, the speech to be recognized includes T speech frames, and the text encoding feature X py The number of channels of is consistent with the number of speech frames and is aligned, the number of channels of the segment-level pronunciation feature S1 is 1, and the text encoding feature X py The dimension of the segment-level pronunciation feature S1 is D, that is, X py ∈R T×D , S1∈R 1×D , where the text encoding feature X pyIt contains the second encoding features extracted from the pronunciation elements aligned to T speech frames. Since the number of channels of the segment-level pronunciation feature S1 is 1, it is necessary to expand the number of channels of the segment-level pronunciation feature S1 to the text encoding feature X py The number of channels is consistent and aligned, so as to obtain the expanded pronunciation feature S2, S2∈R T×D , to fuse the second encoding feature and the expanded pronunciation feature corresponding to the same speech frame to obtain the second fusion feature

[0037]

[0038] Among them, concat(X py ,S2) is 2D, and after the linear layer of FusionLayer, the final The dimension of is still D. Of course, the text encoding feature X py The dimensions of the segment-level pronunciation feature S1 may be inconsistent. For example, the text encoding feature X py The dimension of the segment-level pronunciation feature S1 can be D-dimensional, and the dimension of the segment-level pronunciation feature S1 can be C-dimensional. Here, the text encoding feature X py The dimension of the segment-level pronunciation feature S1 is not specifically limited. In addition, the linear layer of FusionLayer concat(X py ,S2) is adjusted to D dimension only as an example. You can also adjust concat(X py ,S2) is adjusted to other target dimensions, and the target dimensions are not specifically limited here.

[0039] Step S14: Decoding is performed based on the first fusion feature and the second fusion feature to obtain a speech recognition text of the speech to be recognized.

[0040] In one implementation scenario, the first fused feature and the second fused feature are directly decoded by a decoder of a network structure such as a Transformer or a Conformer to obtain a speech recognition text of the speech to be recognized. For the specific decoding process, please refer to the technical details of the speech recognition decoder, which will not be repeated here.

[0041] In another implementation scenario, unlike the above method of directly decoding the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized, in order to further improve the accuracy of speech recognition, the first fusion feature and the second fusion feature can be fused through the twin cross-attention mechanism to obtain the third fusion feature, and then decoded based on the third fusion feature to obtain the speech recognition text. Among them, the twin cross-attention mechanism uses two cross-attention modules to process the first fusion feature and the second fusion feature respectively. The two cross-attention modules can associate the first fusion feature and the second fusion feature of different modalities, thereby fusing the first fusion feature and the second fusion feature to obtain the third fusion feature. The specific process of fusion can refer to the relevant technology of the cross-attention mechanism, which will not be repeated here. Therefore, the twin attention mechanism makes full use of the complementarity and dependence of different modalities, promotes the in-depth fusion of different modal features, and helps to improve the accuracy of speech recognition.

[0042] In an implementation scenario, the speech recognition text is obtained by the speech recognition model recognizing the speech to be recognized, that is, after the speech to be recognized is input into the speech recognition model, the speech recognition model implements the above steps S11, S12, S13, and S14 respectively, and finally the speech recognition model outputs the speech recognition text of the speech to be recognized. Among them, the speech recognition model is pre-trained at least based on the sample speech, the sample speech is annotated with the sample real text, and the sample speech is also annotated with the sample real pronunciation sequence, and the sample real pronunciation sequence contains sample pronunciation elements that are respectively aligned with each sample speech frame in the sample speech. The pre-training process of the speech recognition model can refer to the relevant description in the subsequent embodiments, which will not be repeated here. Therefore, the speech recognition efficiency can be improved by obtaining the speech recognition text by recognizing the speech to be recognized through the trained speech recognition model.

[0043] The above scheme obtains acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized, and predicts based on the acoustic coding features to obtain the predicted pronunciation sequence of the speech to be recognized, and extracts text coding features based on the predicted pronunciation sequence, and then obtains the first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtains the second fusion feature of the text coding features and the segment-level pronunciation features, and then decodes based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized. Therefore, fusing the acoustic coding features with the finer-grained frame-level pronunciation features can capture deeper acoustic information related to pronunciation at the accent level, and fusing the text coding features with the coarser-grained segment-level pronunciation features can capture the relationship between different pronunciations and text information at the language level, and then using the complementarity of the first fusion feature and the second fusion feature in granularity and modality, it is helpful to promote the deep fusion between features to obtain the speech recognition text, thereby improving the accuracy of speech recognition, especially when the speech to be recognized contains an accent or dialect, the accuracy of speech recognition can be improved.

[0044] See also Figure 2 , Figure 2 This is a schematic diagram of the pre-training process of an embodiment of the speech recognition model of the present application. In the disclosed embodiment, the speech recognition model is pre-trained based on at least sample speech, the sample speech is annotated with sample real text, and the sample speech is also annotated with a sample real pronunciation sequence, the sample real pronunciation sequence includes sample pronunciation elements aligned to each sample speech frame in the sample speech, and the sample real pronunciation sequence can be obtained by converting the sample real text or directly recognizing the sample speech. Specifically, the pre-training of the speech recognition model includes the following steps:

[0045] Step S21: based on the sample speech features of the sample speech, obtain sample acoustic coding features, sample frame-level pronunciation features and sample segment-level pronunciation features.

[0046] For details, please refer to the relevant description of step S11 in the above embodiment, which will not be repeated here.

[0047] Step S22: Predicting based on the sample acoustic coding features to obtain a sample predicted pronunciation sequence of the sample speech, and extracting the sample text coding features based on either the sample predicted pronunciation sequence or the sample real pronunciation sequence.

[0048] In one implementation scenario, prediction is performed based on the acoustic coding features of the sample to obtain a sample predicted pronunciation sequence of the sample speech. Please refer to the relevant description of performing prediction based on the acoustic coding features in step S12 of the aforementioned embodiment to obtain a predicted pronunciation sequence of the speech to be recognized, which will not be repeated here.

[0049] In an implementation scenario, a speech recognition model is obtained through several rounds of training. In different training rounds of the speech recognition model, the method of obtaining the sample text coding features is different. For example, in the early training rounds, since the accuracy of the speech recognition model in predicting the sample predicted pronunciation sequence is low, the sample real pronunciation sequence can be used as the basis for extracting the sample text coding features. After multiple rounds of training, the accuracy of the speech recognition model in predicting the sample predicted pronunciation sequence has improved. At this time, the sample predicted pronunciation sequence can be used as the basis for extracting the sample text coding features. Therefore, by selecting either the sample predicted pronunciation sequence or the sample real pronunciation sequence in different training processes of the speech recognition model to extract the sample text coding features, it is possible to avoid the mismatch between the speech recognition model in the training process and the reasoning process as much as possible, and improve the flexibility of speech recognition model training.

[0050] In a specific implementation scenario, the size relationship between the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence and the preset difference threshold can be used as the basis for selecting the sample predicted pronunciation sequence or the sample real pronunciation sequence for encoding to obtain the sample text encoding feature. Among them, the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence can be the difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence in the latest training round of the speech recognition model. In addition, the preset difference threshold can be customized according to different training requirements, and the present application does not make specific restrictions on the preset difference threshold. Therefore, when the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence is greater than the preset difference threshold, encoding can be performed based on the sample real pronunciation sequence to obtain the sample text encoding feature; when the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence is less than the preset difference threshold, encoding can be performed based on the sample predicted pronunciation sequence to obtain the sample text encoding feature.

[0051] In another specific implementation scenario, the difference between the key pronunciation elements of the sample predicted pronunciation sequence and the sample real pronunciation sequence corresponding to the same sample speech frame in the latest training round of the speech recognition model can also be used as a basis for selecting the sample predicted pronunciation sequence or the sample real pronunciation sequence for encoding to obtain the sample text encoding feature. Among them, the key pronunciation element can be the pronunciation element with the largest weight value marked in advance among the pronunciation elements of each sample speech frame aligned with the sample real pronunciation sequence.

[0052] Step S23: obtaining a first sample fusion feature of the sample acoustic coding feature and the sample frame-level pronunciation feature, and obtaining a second sample fusion feature of the sample text coding feature and the sample segment-level pronunciation feature.

[0053] For details, please refer to the relevant description of step S13 in the above embodiment, which will not be repeated here.

[0054] Step S24: Decoding is performed based on the first sample fusion feature and the second sample fusion feature to obtain a sample recognition text of the sample speech.

[0055] For details, please refer to the relevant description of step S14 in the above embodiment, which will not be repeated here.

[0056] Step S25: Adjust the network parameters of the speech recognition model based at least on a first loss measured by the difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence, and a second loss measured by the difference between the sample recognized text and the sample real text.

[0057] In one implementation scenario, the difference measure between the sample predicted pronunciation sequence and the sample real pronunciation sequence can be implemented by a loss function such as cross entropy loss, thereby obtaining a first loss. The difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence can include the difference between the pronunciation elements of the sample predicted pronunciation sequence and the sample real pronunciation sequence aligned in the same sample speech frame. Exemplarily, the sample speech feature fbank is encoded using a conformer encoder to obtain the sample acoustic coding feature X fa =conformer(X fbank ), and continue to sample acoustic encoding features X through the conformer encoder fa Encode to get the conformer (X fa ), and then use the DNN decoder to decode the conformer (X fa ) to predict and obtain the sample predicted pronunciation sequence Thus, the first loss L is measured by the cross entropy loss function CE for:

[0058]

[0059] Among them, Y fa is the real pronunciation sequence of the sample.

[0060] In one implementation scenario, the difference between the sample recognition text and the sample real text can be measured by a loss function such as CTC loss (Connectionist Temporal Classification), thereby obtaining a second loss. The difference between the sample recognition text and the sample real text may include the difference between the characters at the same character position corresponding to the sample recognition text and the sample real text. The specific calculation process of the second loss can refer to the relevant technology of loss functions such as CTC loss, which will not be repeated here.

[0061] In an implementation scenario, before adjusting the network parameters of the speech recognition model, a comparison can also be made based on the first sample fusion feature and the second sample fusion feature, and a third loss can be obtained by measuring the contrast loss function such as InfoNCE loss (Information Noise-Contrastive Estimation). Among them, by contrast learning the first sample fusion feature and the second sample fusion feature, the first sample fusion feature and the second sample fusion feature can be mapped to a feature space, and the matching relationship between the first sample fusion feature (pronunciation level) and the second sample fusion feature (language level) can be learned by contrast, thereby combining the matching first sample fusion feature and the second sample fusion feature more closely together, and separating the distance between the unmatched first sample fusion feature and the second sample fusion feature in the feature space, thereby enhancing the correlation between the first sample fusion feature and the second sample fusion feature, and enabling the speech recognition model to better capture pronunciation details. The specific calculation process of the third loss can refer to the relevant technology of multimodal contrastive learning, which will not be repeated here.

[0062] In a specific implementation scenario, the network parameters of the speech recognition model are adjusted through the first loss, the second loss, and the third loss. For example, the network parameters of the speech recognition model can be adjusted through the results of fusion processing such as summation, weighting, and averaging of the first loss, the second loss, and the third loss. The specific process of adjusting the network parameters of the speech recognition model can refer to related technologies such as gradient descent that use training losses to adjust network parameters, which will not be repeated here.

[0063] In one implementation scenario, the speech recognition model includes at least a first encoding model, a second encoding model, a prediction model and a decoding model, the first encoding model is used to encode speech features, the prediction model is used to predict pronunciation sequences, the second encoding model is used to encode pronunciation sequences, the decoding model is used to decode and recognize text, and the speech recognition model is obtained through several rounds of training. As a possible implementation, when the latest difference between the predicted pronunciation sequence of the sample and the real pronunciation sequence of the sample is greater than the preset difference threshold, at least the first loss and the second loss are used to jointly adjust the network parameters of the speech recognition model; as another possible implementation, when the latest difference between the predicted pronunciation sequence of the sample and the real pronunciation sequence of the sample is less than the preset difference threshold, the network parameters of the prediction model are adjusted by the first loss, and the training loss other than the first loss and at least including the second loss is used to adjust the network parameters of the first encoding model, the second encoding model and the decoding model. Therefore, after the network parameter training of the prediction model is stable, the prediction model is made independent of other modules of the speech recognition model by cutting off the gradient propagation of the prediction model, which helps to improve the stability of the overall training of the speech recognition model.

[0064] In a specific implementation scenario, the speech recognition model also includes a comparison model, which is used to predict the matching relationship between two different modal input features. As a possible implementation, when the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence is greater than the preset difference threshold as mentioned above, the network parameters of the speech recognition model are adjusted together through the first loss, the second loss and the third loss; as another possible implementation, when the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence is less than the preset difference threshold as mentioned above, the network parameters of the prediction model are adjusted through the first loss, and the second loss and the third loss are used to adjust the network parameters of the first encoding model, the second encoding model, the decoding model and the comparison model.

[0065] See also Figure 3 , Figure 3 This is a structural diagram of an embodiment of a speech recognition model of the present application, and the speech recognition model includes a first encoding model, a second encoding model, a prediction model, a decoding model and a comparison model. The first encoding model is used to encode the speech features of the speech to be recognized, the prediction model is used to predict the pronunciation sequence, the second encoding model is used to encode the pronunciation sequence, and the decoding model is used to decode the recognized text. Among them, the first encoding model includes a first encoding subnetwork, a second encoding subnetwork and a first fusion subnetwork, the first encoding subnetwork and the second encoding subnetwork are used to encode the speech features Fbank to obtain the acoustic coding features, and the first fusion subnetwork is used to fuse the acoustic coding features and the frame-level pronunciation features obtained by recognizing Fbank to output the first fusion features; the prediction model includes a third encoding subnetwork and a prediction subnetwork, the third encoding subnetwork is used to encode the acoustic coding features, and the prediction subnetwork is used to predict the pronunciation sequence; the second encoding model includes an embedding subnetwork, a fourth encoding subnetwork and a second fusion subnetwork, and the embedding subnetwork is used to convert the pronunciation sequence into elements of each pronunciation element Embedding features, the fourth encoding subnetwork is used to encode the element embedding features into text encoding features, the second fusion subnetwork is used to fuse the text encoding features and the segment-level pronunciation features obtained by recognizing Fbank to output the second fusion features, the comparison model is used to compare and analyze the first fusion features and the second fusion features, the decoding model includes the third fusion subnetwork, the fifth encoding subnetwork and the decoding subnetwork, the third fusion subnetwork is used to fuse the first fusion features and the second fusion features based on the twin cross attention mechanism to output the third fusion features, the fifth encoding subnetwork is used to integrate the third fusion features, and the decoding subnetwork is used to decode the integrated third fusion features to obtain the speech recognition text.

[0066] The above scheme obtains sample acoustic coding features, sample frame-level pronunciation features and sample segment-level pronunciation features based on the sample speech features of the sample speech by training the speech recognition model, and predicts based on the sample acoustic coding features to obtain a sample predicted pronunciation sequence of the sample speech, and extracts sample text coding features based on either the sample predicted pronunciation sequence or the sample real pronunciation sequence, and then obtains a first sample fusion feature of the sample acoustic coding feature and the sample frame-level pronunciation feature, and obtains a second sample fusion feature of the sample text coding feature and the sample segment-level pronunciation feature, and then decodes based on the first sample fusion feature and the second sample fusion feature to obtain a sample recognized text of the sample speech, and then adjusts the network parameters of the speech recognition model based on at least a first loss obtained by measuring the difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence, and a second loss obtained by measuring the difference between the sample recognized text and the sample real text. Therefore, fusing the sample acoustic coding features with the finer-grained sample frame-level pronunciation features can train the speech recognition model to capture deeper pronunciation-related acoustic information at the accent level, and fusing the sample text coding features with the coarser-grained sample segment-level pronunciation features can train the speech recognition model to capture the relationship between different pronunciations and text information at the language level. Furthermore, utilizing the complementarity of the first sample fusion features and the second sample fusion features in granularity and modality respectively helps to promote the deep fusion of sample features to obtain sample recognition text, thereby improving the accuracy of speech recognition of the speech recognition model, especially improving the accuracy of speech recognition of the speech recognition model under the influence of different dialects or accents.

[0067] See also Figure 4 , Figure 4 This is a framework diagram of an embodiment of a speech recognition device of the present application, wherein the speech recognition device 40 includes an acquisition module 41, a prediction module 42, a fusion module 43 and a decoding module 44; the acquisition module 41 is used to obtain acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized; the prediction module 42 is used to predict based on the acoustic coding features to obtain a predicted pronunciation sequence of the speech to be recognized, and extract text coding features based on the predicted pronunciation sequence; the fusion module 43 is used to obtain a first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtain a second fusion feature of the text coding features and the segment-level pronunciation features; the decoding module 44 is used to decode based on the first fusion feature and the second fusion feature to obtain a speech recognition text of the speech to be recognized.

[0068] In the above scheme, the speech recognition device 40 obtains acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized, and predicts based on the acoustic coding features to obtain the predicted pronunciation sequence of the speech to be recognized, and extracts text coding features based on the predicted pronunciation sequence, and then obtains the first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtains the second fusion feature of the text coding features and the segment-level pronunciation features, and then decodes based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized. Therefore, the fusion of acoustic coding features with finer-grained frame-level pronunciation features can capture deeper acoustic information related to pronunciation at the accent level, and the fusion of text coding features with coarser-grained segment-level pronunciation features can capture the relationship between different pronunciations and text information at the language level, and then the complementarity of the first fusion feature and the second fusion feature in granularity and modality respectively is used to promote the deep fusion between features to obtain speech recognition text, thereby improving the accuracy of speech recognition, especially when the speech to be recognized contains an accent or dialect, the accuracy of speech recognition can be improved.

[0069] In some disclosed embodiments, the speech to be recognized includes several speech frames, the acoustic coding features include first coding features corresponding to each speech frame, and the fusion module 43 includes a first fusion submodule, which is used to obtain a first fusion feature of the acoustic coding features and the frame-level pronunciation features, and is specifically used to fuse the first coding features and frame-level pronunciation features corresponding to the same speech frame to obtain the first fusion feature.

[0070] Therefore, by fusing acoustic coding features with finer-grained frame-level pronunciation features, it helps to capture deeper pronunciation-related acoustic information at the accent level while capturing the pronunciation differences of different speakers in each speech frame, thereby improving the robustness of speech recognition.

[0071] In some disclosed embodiments, the speech to be recognized includes several speech frames, the predicted pronunciation sequence includes pronunciation elements respectively aligned with each speech frame, the text encoding feature includes a second encoding feature extracted from the pronunciation elements respectively aligned with each speech frame, and the fusion module 43 includes a second fusion submodule, which is used to obtain a second fusion feature of the text encoding feature and the segment-level pronunciation feature, specifically for expanding the number of channels of the segment-level pronunciation feature to the same as the total number of several speech frames to obtain an expanded pronunciation feature; based on the second encoding feature and the expanded pronunciation feature corresponding to the same speech frame, a fusion is performed to obtain a second fusion feature.

[0072] Therefore, by fusing text encoding features with coarser-grained segment-level pronunciation features, it helps to capture the relationship between different pronunciations and text information at the language level, while capturing the pronunciation differences of different speakers in the entire speech segment, thereby improving the robustness of speech recognition.

[0073] In some disclosed embodiments, the predicted pronunciation sequence includes pronunciation elements respectively aligned with each speech frame, and the prediction module 42 is specifically used to obtain element embedding features of each pronunciation element in the predicted pronunciation sequence; and encode based on the element embedding features of each pronunciation element to obtain text encoding features.

[0074] Therefore, by adopting the predicted pronunciation sequence based on acoustic coding feature prediction as the data source of text coding features, the data in the same network can be recycled to improve data processing efficiency.

[0075] In some disclosed embodiments, the decoding module 44 is specifically used to fuse the first fused feature and the second fused feature based on the twin cross attention mechanism to obtain a third fused feature; and decode based on the third fused feature to obtain a speech recognition text.

[0076] Therefore, the twin attention mechanism can make full use of the complementarity and dependency of different modalities, promote the in-depth integration of different modal features, and help improve the accuracy of speech recognition.

[0077] In some disclosed embodiments, speech recognition text is obtained by a speech recognition model recognizing speech to be recognized, and the speech recognition model is pre-trained based on at least sample speech. The sample speech is annotated with sample real text, and the sample speech is also annotated with a sample real pronunciation sequence, and the sample real pronunciation sequence includes sample pronunciation elements that are respectively aligned with each sample speech frame in the sample speech.

[0078] Therefore, by recognizing the speech to be recognized through the trained speech recognition model to obtain the speech recognition text, the efficiency of speech recognition can be improved.

[0079] In some disclosed embodiments, speech recognition also includes a training module, which is used to train a speech recognition model, specifically to obtain sample acoustic coding features, sample frame-level pronunciation features and sample segment-level pronunciation features based on sample speech features of the sample speech; to predict based on the sample acoustic coding features to obtain a sample predicted pronunciation sequence of the sample speech, and to extract sample text coding features based on either the sample predicted pronunciation sequence or the sample real pronunciation sequence; to obtain a first sample fusion feature of the sample acoustic coding feature and the sample frame-level pronunciation feature, and to obtain a second sample fusion feature of the sample text coding feature and the sample segment-level pronunciation feature; to decode based on the first sample fusion feature and the second sample fusion feature to obtain a sample recognized text of the sample speech; and to adjust the network parameters of the speech recognition model based at least on a first loss obtained by measuring the difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence, and a second loss obtained by measuring the difference between the sample recognized text and the sample real text.

[0080] Therefore, by fusing the sample acoustic coding features with the finer-grained sample frame-level pronunciation features, the speech recognition model can be trained to capture deeper pronunciation-related acoustic information at the accent level, and by fusing the sample text coding features with the coarser-grained sample segment-level pronunciation features, the speech recognition model can be trained to capture the relationship between different pronunciations and text information at the language level. Furthermore, by utilizing the complementarity of the first sample fusion features and the second sample fusion features in granularity and modality respectively, the sample recognition text can be fused as deeply as possible, thereby improving the accuracy of speech recognition of the speech recognition model.

[0081] In some disclosed embodiments, the training module is also used to compare the first sample fusion feature and the second sample fusion feature to obtain a third loss before adjusting the network parameters of the speech recognition model at least based on a first loss obtained by measuring the difference between the sample predicted pronunciation sequence and the sample actual pronunciation sequence, and a second loss obtained by measuring the difference between the sample recognized text and the sample actual text; and adjust the network parameters of the speech recognition model based on the first loss, the second loss and the third loss.

[0082] Therefore, by comparing and learning the matching relationship between the first sample fusion features (pronunciation level) and the second sample fusion features (language level), the correlation between the first sample fusion features and the second sample fusion features can be enhanced, enabling the speech recognition model to better capture the pronunciation details.

[0083] In some disclosed embodiments, the speech recognition model is obtained through several rounds of training, and the training module is also specifically used to, in response to the latest difference between the sample predicted pronunciation sequence and the sample actual pronunciation sequence being greater than a preset difference threshold, encode based on the sample actual pronunciation sequence to obtain the sample text encoding feature; in response to the latest difference between the sample predicted pronunciation sequence and the sample actual pronunciation sequence being less than a preset difference threshold, encode based on the sample predicted pronunciation sequence to obtain the sample text encoding feature.

[0084] Therefore, by selecting either the sample predicted pronunciation sequence or the sample actual pronunciation sequence in different training processes of the speech recognition model to extract the sample text encoding features, it is possible to avoid the mismatch between the training process and the inference process of the speech recognition model as much as possible and improve the flexibility of speech recognition model training.

[0085] In some disclosed embodiments, the speech recognition model includes at least a first encoding model, a second encoding model, a prediction model and a decoding model, the first encoding model is used to encode speech features, the prediction model is used to predict pronunciation sequences, the second encoding model is used to encode pronunciation sequences, the decoding model is used to decode and recognize text, and the speech recognition model is obtained through several rounds of training, and the training module is also specifically used to adjust the network parameters of the prediction model based on the first loss in response to the latest difference between the sample predicted pronunciation sequence and the sample true pronunciation sequence being less than a preset difference threshold, and adjust the network parameters of the first encoding model, the second encoding model and the decoding model based on the training loss other than the first loss and at least including the second loss.

[0086] Therefore, after the network parameter training of the prediction model is stabilized, cutting off the gradient propagation of the prediction model makes the prediction model independent of other modules of the speech recognition model, which helps to improve the stability of the overall training of the speech recognition model.

[0087] See also Figure 5 , Figure 5 Schematic diagram of the framework of an embodiment of an electronic device 50 of the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above-mentioned speech recognition method embodiments. Specifically, the electronic device 50 may include but is not limited to: a microcomputer, a server, a laptop computer, a tablet computer, a smart phone, etc., which are not limited here.

[0088] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of any of the above-mentioned speech recognition method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0089] In the above scheme, the electronic device 50 obtains acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized, and predicts based on the acoustic coding features to obtain the predicted pronunciation sequence of the speech to be recognized, and extracts text coding features based on the predicted pronunciation sequence, and then obtains the first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtains the second fusion feature of the text coding features and the segment-level pronunciation features, and then decodes based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized. Therefore, the fusion of acoustic coding features with the finer-grained frame-level pronunciation features can capture deeper acoustic information related to pronunciation at the accent level, and the fusion of text coding features with the coarser-grained segment-level pronunciation features can capture the relationship between different pronunciations and text information at the language level, and then the complementarity of the first fusion feature and the second fusion feature in granularity and modality is used to promote the deep fusion between features to obtain the speech recognition text, thereby improving the accuracy of speech recognition, especially when the speech to be recognized contains an accent or dialect, the accuracy of speech recognition can be improved.

[0090] See also Figure 6 , Figure 6 The schematic diagram of the framework of an embodiment of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor, and the program instructions 61 are used to implement the steps of the above-mentioned speech recognition method embodiment.

[0091] In the above scheme, the computer-readable storage medium 60 obtains acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized, and predicts based on the acoustic coding features to obtain the predicted pronunciation sequence of the speech to be recognized, and extracts text coding features based on the predicted pronunciation sequence, and then obtains the first fusion feature of the acoustic coding features and the frame-level pronunciation features, and obtains the second fusion feature of the text coding features and the segment-level pronunciation features, and then decodes based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized. Therefore, the fusion of the acoustic coding features with the finer-grained frame-level pronunciation features can capture deeper acoustic information related to pronunciation at the accent level, and the fusion of the text coding features with the coarser-grained segment-level pronunciation features can capture the relationship between different pronunciations and text information at the language level, and then the complementarity of the first fusion feature and the second fusion feature in granularity and modality is used to promote the deep fusion between features to obtain the speech recognition text, thereby improving the accuracy of speech recognition, especially when the speech to be recognized contains an accent or dialect, the accuracy of speech recognition can be improved.

[0092] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0093] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0094] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0096] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0097] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech recognition method, characterized in that: include: Based on the speech features of the speech to be recognized, acoustic coding features, frame-level pronunciation features and segment-level pronunciation features are obtained; Predicting based on the acoustic coding features to obtain a predicted pronunciation sequence of the speech to be recognized, and extracting text coding features based on the predicted pronunciation sequence; Acquire a first fusion feature of the acoustic coding feature and the frame-level pronunciation feature, and acquire a second fusion feature of the text coding feature and the segment-level pronunciation feature; Decoding is performed based on the first fusion feature and the second fusion feature to obtain a speech recognition text of the speech to be recognized.

2. The method according to claim 1, characterized in that The speech to be recognized includes a plurality of speech frames, the acoustic coding features include first coding features corresponding to each of the speech frames, and the first fusion feature of the acoustic coding features and the frame-level pronunciation features is obtained, including: The first fused feature is obtained by fusing the first coding feature and the frame-level pronunciation feature corresponding to the same speech frame.

3. The method according to claim 1, characterized in that The speech to be recognized includes a plurality of speech frames, the predicted pronunciation sequence includes pronunciation elements respectively aligned with each of the speech frames, the text encoding feature includes a second encoding feature extracted from the pronunciation elements respectively aligned with each of the speech frames, and the second fusion feature of the text encoding feature and the segment-level pronunciation feature is obtained, including: Expanding the number of channels of the segment-level pronunciation feature to be the same as the total number of the plurality of speech frames to obtain an expanded pronunciation feature; The second fused feature is obtained by fusing the second coding feature and the expanded pronunciation feature corresponding to the same speech frame.

4. The method according to claim 1, characterized in that: The speech to be recognized includes a plurality of speech frames, the predicted pronunciation sequence includes pronunciation elements respectively aligned with each of the speech frames, and the text encoding features are extracted based on the predicted pronunciation sequence, including: Obtaining element embedding features of each of the pronunciation elements in the predicted pronunciation sequence; Encoding is performed based on the element embedding features of each of the pronunciation elements to obtain the text encoding features.

5. The method according to claim 1, characterized in that The decoding based on the first fusion feature and the second fusion feature to obtain the speech recognition text of the speech to be recognized includes: Based on the twin cross attention mechanism, the first fused feature and the second fused feature are fused to obtain a third fused feature; Decoding is performed based on the third fusion feature to obtain the speech recognition text.

6. The method according to claim 1, characterized in that The speech recognition text is obtained by recognizing the speech to be recognized by a speech recognition model, and the speech recognition model is pre-trained based on at least a sample speech. The sample speech is annotated with a sample real text, and the sample speech is also annotated with a sample real pronunciation sequence, and the sample real pronunciation sequence includes sample pronunciation elements that are respectively aligned with each sample speech frame in the sample speech.

7. The method according to claim 6, characterized in that The training steps of the speech recognition model include: Based on the sample speech features of the sample speech, obtaining sample acoustic coding features, sample frame-level pronunciation features and sample segment-level pronunciation features; Predicting based on the sample acoustic coding features to obtain a sample predicted pronunciation sequence of the sample speech, and extracting a sample text coding feature based on either the sample predicted pronunciation sequence or the sample real pronunciation sequence; Acquire a first sample fusion feature of the sample acoustic coding feature and the sample frame-level pronunciation feature, and acquire a second sample fusion feature of the sample text coding feature and the sample segment-level pronunciation feature; Decoding is performed based on the first sample fusion feature and the second sample fusion feature to obtain a sample recognition text of the sample speech; The network parameters of the speech recognition model are adjusted based at least on a first loss measured by the difference between the sample predicted pronunciation sequence and the sample true pronunciation sequence, and a second loss measured by the difference between the sample recognized text and the sample true text.

8. The method according to claim 7, characterized in that Before adjusting the network parameters of the speech recognition model based at least on the first loss obtained by measuring the difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence, and the second loss obtained by measuring the difference between the sample recognized text and the sample real text, the method further includes: Comparing the first sample fusion feature with the second sample fusion feature to obtain a third loss; The method of adjusting the network parameters of the speech recognition model based on at least a first loss obtained by measuring the difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence, and a second loss obtained by measuring the difference between the sample recognized text and the sample real text, comprises: Based on the first loss, the second loss and the third loss, the network parameters of the speech recognition model are adjusted.

9. The method according to claim 7, characterized in that: The speech recognition model is obtained through several rounds of training, and the sample text encoding features are extracted based on either the sample predicted pronunciation sequence or the sample real pronunciation sequence, including: In response to the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence being greater than a preset difference threshold, encoding is performed based on the sample real pronunciation sequence to obtain the sample text encoding feature; In response to the latest difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence being less than the preset difference threshold, encoding is performed based on the sample predicted pronunciation sequence to obtain the sample text encoding feature.

10. The method according to claim 7 or 9, characterized in that: The speech recognition model at least includes a first encoding model, a second encoding model, a prediction model and a decoding model, wherein the first encoding model is used to encode speech features, the prediction model is used to predict a pronunciation sequence, the second encoding model is used to encode a pronunciation sequence, and the decoding model is used to decode and recognize text, and the speech recognition model is obtained through several rounds of training, and the network parameters of the speech recognition model are adjusted based on at least a first loss measured by a difference between the sample predicted pronunciation sequence and the sample real pronunciation sequence, and a second loss measured by a difference between the sample recognized text and the sample real text, including: In response to the latest difference between the sample predicted pronunciation sequence and the sample actual pronunciation sequence being less than a preset difference threshold, based on the first loss, the network parameters of the prediction model are adjusted, and based on the training loss other than the first loss and at least including the second loss, the network parameters of the first encoding model, the second encoding model and the decoding model are adjusted.

11. A speech recognition device, characterized in that: include: An acquisition module, used for obtaining acoustic coding features, frame-level pronunciation features and segment-level pronunciation features based on the speech features of the speech to be recognized; A prediction module, configured to predict based on the acoustic coding features to obtain a predicted pronunciation sequence of the speech to be recognized, and extract text coding features based on the predicted pronunciation sequence; A fusion module, used to obtain a first fusion feature of the acoustic coding feature and the frame-level pronunciation feature, and to obtain a second fusion feature of the text coding feature and the segment-level pronunciation feature; A decoding module is used to perform decoding based on the first fusion feature and the second fusion feature to obtain a speech recognition text of the speech to be recognized.

12. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech recognition method according to any one of claims 1 to 10.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the speech recognition method according to any one of claims 1 to 10 is implemented.

Citation Information

Cited By

  • Voice interaction method, vehicle and computer readable storage medium

    CN119181363A

  • Voice interaction method, vehicle, and computer-readable storage medium

    CN119181363B

  • Speech recognition model training method, using method and related device

    CN121708906A

  • Speech data processing method and apparatus, and device and readable storage medium

    WO2026170942A1