Speech recognition method, related device, equipment and storage medium

Abstract

The invention discloses a speech recognition method, a related device, equipment and a storage medium. The speech recognition method comprises the following steps: acquiring a first decoding feature of a current round; wherein the first decoding feature of the current round comprises feature information of the decoded characters output by the last round; acquiring an alignment probability between the coding feature of each voice frame in the to-be-recognized voice and the first decoding feature, and based on the alignment probability, selecting the coding features of at least part of the voice frames to be fused with the first decoding feature to obtain a second decoding feature; and performing decoding based on the second decoding feature to obtain the decoding character output by the current round, returning to the step of obtaining the first decoding feature of the current round for iteration until the latest output decoding character represents that the speech recognition is finished, and obtaining the recognition text of the speech to be recognized based on the decoding characters of each round. According to the scheme, the speech recognition efficiency and accuracy, especially the streaming speech recognition efficiency and accuracy, can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and related devices, equipment and storage media. Background Art

[0002] Automatic Speech Recognition (ASR) technology can extract text information from speech and convert it into corresponding text. As a form of speech recognition technology, streaming speech recognition can process continuous speech input in real time and provide instant feedback. This feature makes it have important application value in many fields such as real-time communication, intelligent assistants and simultaneous interpretation.

[0003] With the improvement of computing power of speech recognition models and the advancement of algorithms, speech recognition technology has gradually matured and has been widely used. However, due to background noise, accents, sound overlap, changes in speech speed, differences in speaking styles, and model limitations, speech recognition accuracy is not high. In particular, streaming speech recognition systems need to provide recognition results in real time or near real time and need to start generating text after receiving a small portion of speech signals, which further reduces the accuracy of speech recognition. In order to improve accuracy, the system often needs to collect more audio data to make decisions, which in turn reduces the efficiency of speech recognition.

[0004] Therefore, how to improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition, has become a problem that needs to be solved urgently. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a speech recognition method and related devices, equipment and storage media, which can improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition.

[0006] In order to solve the above technical problems, the first aspect of the present application provides a speech recognition method, which includes: obtaining a first decoding feature of the current round; wherein the first decoding feature of the current round contains feature information of the decoded characters output in the previous round; obtaining the alignment probability between the coding features of each speech frame in the speech to be recognized and the first decoding feature, and based on the alignment probability, selecting the coding features of at least part of the speech frames to merge with the first decoding feature to obtain a second decoding feature; decoding based on the second decoding feature to obtain the decoded characters output in the current round, and returning to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and obtaining the recognized text of the speech to be recognized based on the decoded characters of each round.

[0007] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a speech recognition device, which includes a first acquisition module, a second acquisition module and a decoding module; the first acquisition module is used to obtain the first decoding feature of the current round; wherein the first decoding feature of the current round contains the feature information of the decoded characters output in the previous round; the second acquisition module is used to obtain the alignment probability between the encoding features of each speech frame in the speech to be recognized and the first decoding feature, and based on the alignment probability, select the encoding features of at least part of the speech frames to merge with the first decoding feature to obtain the second decoding feature; the decoding module is used to decode based on the second decoding feature to obtain the decoded characters output in the current round, and return to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded character represents the end of speech recognition, and based on the decoded characters of each round, the recognized text of the speech to be recognized is obtained.

[0008] In order to solve the above technical problem, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the speech recognition method in the above first aspect.

[0009] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the speech recognition method in the above first aspect is implemented.

[0010] The above scheme obtains the first decoding feature of the current round, wherein the first decoding feature of the current round includes the feature information of the decoded characters output in the previous round, and obtains the alignment probability between the coding features of each speech frame in the speech to be recognized and the first decoding feature, so as to select the coding features of at least part of the speech frames based on the alignment probability to merge with the first decoding feature to obtain the second decoding feature, and then decode based on the second decoding feature to obtain the decoded characters output in the current round, and return to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and finally obtain the recognized text of the speech to be recognized based on the decoded characters of each round. Therefore, on the one hand, compared with referring to all speech frames, selecting the coding features of at least part of the speech frames to merge with the first decoding features can improve data processing efficiency. On the other hand, selecting coding features to merge with the first decoding features according to the alignment probability can help improve the data validity of the decoding process of speech recognition, thereby improving the accuracy of speech recognition. In the scenario of streaming speech recognition, it is possible to selectively select coding features at and before the alignment moment with the first decoding feature of the current round according to the alignment probability, thereby reducing the reliance on coding features after the alignment moment in the speech recognition process, thereby improving the accuracy of streaming speech recognition. Therefore, it is possible to improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of an embodiment of a speech recognition method of the present application; Figure 2 This is a schematic diagram of the structure of an embodiment of a downsampling network of the present application; Figure 3 A schematic diagram of a framework of an embodiment of a target speech recognition model of the present application; Figure 4 for Figure 3 A schematic diagram of an embodiment of a sample speech of a target speech recognition model; Figure 5 A schematic diagram of a framework of an embodiment of a speech recognition device of the present application; Figure 6 A schematic diagram of a framework of an embodiment of an electronic device of the present application; Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0012] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0013] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0014] The term "multiple" herein means two or more than two. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, and may mean including any one or more elements selected from the set consisting of A, B, and C.

[0015] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech recognition method of the present application, specifically, comprising the following steps: S11: Obtain the first decoding feature of the current round; wherein the first decoding feature of the current round includes feature information of the decoded characters output in the previous round.

[0016] In an implementation scenario, the first decoding feature of the current round can be obtained by extracting features from the decoded characters output in the previous round. For example, feature extraction of decoded characters can be implemented through a Transformer decoder. Specifically, the first decoding feature can be obtained by obtaining the embedding features of the decoded characters output in the previous round and learning the embedding features in combination with an attention mechanism. The specific process can be referred to the technical details of the Transformer decoder, which will not be repeated here.

[0017] In one implementation scenario, when the current round is the first round, since there are no decoded characters output from the previous round, the initial character representing the start of speech recognition can be used as the decoded character output from the previous round, that is, the first decoding feature of the first round contains the feature information of the initial character.

[0018] S12: Obtain the alignment probability between the coding features of each speech frame in the speech to be recognized and the first decoding feature, and select the coding features of at least part of the speech frames to merge with the first decoding feature based on the alignment probability to obtain a second decoding feature.

[0019] In one implementation scenario, the source of the speech to be recognized may include, but is not limited to, intelligent assistants, real-time translation, customer service robots, real-time subtitle generation, etc. In addition, the speech to be recognized may be collected in real time, that is, the speech to be recognized may be streaming speech, or the speech to be recognized may be collected offline and stored in a storage medium such as a hard disk and then read from the storage medium. The method of obtaining the speech to be recognized is not limited here.

[0020] In one implementation scenario, the coding features of each speech frame in the speech to be recognized can be obtained by encoding the speech features of the speech to be recognized through a Conformer encoder, a Transformer encoder, etc. The specific process can refer to the technical details of the above encoders, which will not be repeated here.

[0021] In a specific implementation scenario, the speech features of the speech to be recognized may include acoustic feature information of each speech frame. The types of speech features may include but are not limited to: Fbank features (Filter bank), MFCC features (Mel-Frequency Cepstral Coefficients), etc. The process of acquiring speech features can refer to related technologies such as Fbank features and MFCC features, which will not be repeated here.

[0022] In a specific implementation scenario, in order to improve the efficiency of obtaining the coding features of the speech to be recognized, the speech features of the speech to be recognized can be processed through a downsampling network before encoding the speech features of the speech to be recognized, thereby encoding the local information of the speech features and downsampling them to speed up the network training speed and improve the computing efficiency. As a possible example, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of an embodiment of a downsampling network of the present application. Figure 2 In the figure, the downsampling network is a DenseNet feature extractor, which includes two dense blocks, each of which is followed by a transition layer to reduce the size of the feature map. The dense block consists of several bottleneck layers, each of which consists of a BN-ReLU-Conv-BN-ReLU-Conv structure, where BN is batch normalization, ReLU is a rectified linear unit, and Conv is a convolution. The pooling layer after the transition layer is used to downsample in time and feature dimensions to improve computational efficiency, and the projection layer is used to map the output dimension to the same as the encoder embedding dimension. It should be noted that the DenseNet feature extractor includes two dense blocks only as an example, and the number of dense blocks can also include three, four, and so on. In addition, the downsampling network can also include feature extractors such as MobileNet and ShuffleNet, and the structure of the downsampling network is not specifically limited here.

[0023] In one implementation scenario, the alignment probability between the encoding features of each speech frame in the speech to be recognized and the first decoding feature can be obtained through a monotonic attention mechanism. The monotonic attention mechanism may include but is not limited to: Hard Monotonic Attention (HMA), Monotonic Chunkwise Attention (MOCHA), Monotonic Infinite Lookback Attention (MILK), etc.

[0024] In a specific implementation scenario, a hard monotone attention mechanism is used to obtain the alignment probability. Specifically, the hard monotone attention mechanism is used to obtain the monotone energy between the encoding feature and the first decoding feature, and the alignment probability is obtained according to the monotone energy. For example, the encoding feature of each speech frame in the speech to be recognized is represented as (j represents the frame number of the speech frame, j=1, 2, ..., T), the first decoding feature is expressed as (i represents the sequence number of the current round), the monotonic energy between the encoding feature and the first decoding feature for:

[0025] According to the monotonic energy Get the alignment probability for:

[0026] in, An attention energy metric that represents a monotonic temporal sequence, used to make the movement of attention computations in the sequence smooth and orderly. ， , Characterizes the selection probability of the i-th first decoding feature for the j-th encoding feature, is a random noise that conforms to a standard Gaussian distribution, used to help in training Become sharp, for The expected estimate of is the cumulative multiplication operation, is an accumulation operation. The derivation process of the formula can be found in the related technologies of the monotone attention mechanism, which will not be repeated here. In addition, the alignment probability can also be obtained through the monotone window-level attention mechanism. The monotone window-level attention mechanism combines the hard monotone attention mechanism and the window-level energy. The way to obtain the alignment probability is the same as the process of obtaining the alignment probability through the aforementioned hard monotone attention mechanism.

[0027] In one implementation scenario, after obtaining the alignment probability, a coding feature for fusion with the first decoding feature can be selected according to the alignment probability. As a possible implementation, when obtaining the alignment probability through a monotonic window-level attention mechanism, the speech frame to which the coding feature with the highest alignment probability with the first decoding feature belongs can be used as a reference to obtain the coding features of a window of speech frames forward to fuse with the first decoding feature to obtain a second decoding feature. Specifically, in addition to obtaining the alignment probability through the monotonic window-level attention mechanism as described above with the hard monotonic attention mechanism, In addition, the coding features of each speech frame are also obtained. The first decoding feature The window energy between :

[0028] in, Indicates which parts (i.e., elements within the window) should be paid more attention to in the attention calculation, and then according to the window-level energy Get the attention probability distribution :

[0029] Among them, w is the window size of the monotonic window-level attention mechanism, , b and f represent the number of covers forward and backward starting from position n (including position n). The derivation process of can refer to the related technology of monotonic window-level attention mechanism, which will not be repeated here. With the alignment probability The determined encoding features are fused to obtain the second decoding features.

[0030] As another possible implementation, the alignment probabilities between the coding features of each speech frame and the first decoding feature can be sorted in descending order, and the coding features of several speech frames with the highest sorting order can be selected and fused with the first decoding feature to obtain the second decoding feature.

[0031] In an implementation scenario, the monotone attention mechanism in the aforementioned embodiment can be further combined with the multi-head attention mechanism to obtain a monotone multi-head attention mechanism, so that each attention head outputs the alignment probability between each encoding feature and the first decoding feature. Exemplarily, when the monotone multi-head attention mechanism is obtained by combining the hard monotone attention mechanism with the multi-head attention mechanism, the monotone energy can be correspondingly expressed as:

[0032] For another example, when the monotonic attention mechanism is a combination of the monotonic window-level attention mechanism and the multi-head attention mechanism, the window-level energy can be expressed as:

[0033] in, represents the weight of the monotone energy mapping layer, represents the weight of the window-level energy mapping layer. In this case, the alignment probability of each attention head output is , h represents the head number of the attention head.

[0034] In one implementation scenario, in order to obtain the second decoding feature, the speech frame to which the coding feature aligned with the first decoding feature belongs and the first number of speech frames before it can be selected as the first reference speech frame through the alignment probability, and the coding feature of the first reference speech frame and the first decoding feature are fused to obtain the intermediate decoding feature, and according to the alignment probability, the speech frame to which the coding feature aligned with the first decoding feature belongs is selected as the target speech frame, and then the target speech frame and the second number of speech frames before it are selected as the second reference speech frame, and finally the coding feature of the second reference speech frame and the intermediate decoding feature are fused to obtain the second decoding feature. Therefore, after obtaining the intermediate decoding feature through the first decoding feature, and then obtaining the second decoding feature through the intermediate decoding feature, the first decoding feature can be decoupled from the second decoding feature, thereby realizing the focus on obtaining the alignment probability in the process of obtaining the intermediate decoding, and the focus on obtaining the context information in the process of obtaining the second decoding feature, so as to improve the flexibility of speech recognition. In a specific implementation scenario, the first reference speech frame and the second reference speech frame may be the same, or the first reference speech frame and the second reference speech frame may be different.

[0035] In a specific implementation scenario, the second decoding feature is predicted by the first decoding network of the target speech recognition model, wherein the target speech recognition model includes a first encoding network and a first decoding network, and the first decoding network includes a first decoding layer and a second decoding layer. Specifically, the encoding features of each speech frame in the speech to be recognized are obtained by the first encoding network, and the first decoding feature of the current round is obtained by the first decoding layer to decode and output the alignment probability and the intermediate decoding feature to the second decoding layer, and the intermediate decoding feature is obtained by fusing the encoding feature of the first reference speech frame determined by the first decoding layer according to the alignment probability with the first decoding feature, and then the encoding feature of the second reference speech frame is determined by the second decoding layer according to the alignment probability, so as to fuse with the intermediate decoding feature and decode and output the second decoding feature. It should be noted that the specific network structure and training process of the target speech recognition model can be found in the relevant description of the subsequent embodiments, which will not be repeated here.

[0036] As a possible implementation, the first decoding layer uses a monotone attention mechanism to obtain the alignment probability and the intermediate decoding features. For example, the first decoding layer obtains the alignment probability by using the monotone window-level attention mechanism as described above. and the attention probability distribution On the basis of The encoding feature of the first reference speech frame as a first weight factor Weighted to obtain intermediate decoding features :

[0037] The first reference speech frame is the frame number in [z The speech frame of the interval.

[0038] As another possible implementation, the second decoding layer uses a cross attention mechanism to obtain the second decoding feature. For example, the second decoding layer uses the alignment probability output by the first decoding layer to obtain the second decoding feature. and intermediate decoding features , select the speech frame with the largest alignment probability as the target speech frame, and the frame number of the target speech frame is (i.e. alignment time), the corresponding second reference speech frame is the frame number in [ The second decoding layer further obtains the encoding features and intermediate decoding features of each speech frame under the cross attention mechanism. The energy between :

[0039] in, Used to measure the encoding features and intermediate decoding features of each speech frame That is, when the energy formula is used to process input data, it calculates an "energy" value for each input element, which reflects the relevance or importance of the element to the current task goal. By calculating these energy values, it is possible to determine how much "attention" should be given to each element when processing data, that is, how much resources or weight should be allocated to each input element. , obtain the second weight factor corresponding to the coding feature of the second reference speech frame :

[0040] Thus, according to the second weight factor By weighting the encoding features of the second reference speech frame, the second decoding features can be obtained :

[0041] In a specific implementation scenario, the alignment probability is obtained by a multi-head attention mechanism, and each attention head outputs the alignment probability between each encoding feature and the first decoding feature. In this case, in order to obtain the intermediate decoding feature, the candidate decoding feature of the attention head can be obtained based on the first decoding feature and the encoding feature of the first reference speech frame determined by the attention head, and then the candidate decoding features based on each attention head are fused to obtain the intermediate decoding feature. For example, in the above-mentioned monotonic window-level attention mechanism to obtain the alignment probability and the attention probability distribution Based on this, we get the intermediate decoding features of each attention head :

[0042] And according to the intermediate decoding features of each attention head, the final intermediate decoding features are obtained:

[0043] Therefore, the multi-head attention mechanism can capture information in different representation subspaces by executing multiple attention functions in parallel, and the dimension of each attention head is low, which can improve the efficiency of acquiring intermediate decoding features.

[0044] In a specific implementation scenario, the alignment probability is obtained by a multi-head attention mechanism, and each attention head outputs the alignment probability between each encoding feature and the first decoding feature. In this case, in order to obtain the target speech frame, the speech frame to which the encoding feature aligned with the first decoding feature belongs in each attention head can be determined as a candidate speech frame through the alignment probability of each attention head, thereby selecting the candidate speech frame with the largest frame number as the target speech frame. As a possible implementation method, the alignment probability of each attention head is obtained by the monotone multi-head attention mechanism as described above. , the speech frame with the highest alignment probability in each attention head is taken as the candidate speech frame, and the candidate speech frame with the largest frame number is selected as the target speech frame :

[0045] in, The function is used to find the parameters that make the function reach its maximum value, so as to use The function can determine the speech frame corresponding to the maximum alignment probability of each attention head.

[0046] Correspondingly, the second decoded feature in each attention head :

[0047] And according to the second decoding features in each attention head, the final second decoding features are obtained:

[0048] Therefore, on the one hand, the multi-head attention mechanism can capture information in different representation subspaces by executing multiple attention functions in parallel, and the dimension of each attention head is low, which can improve the efficiency of obtaining the second decoding features; on the other hand, all attention heads share the encoding features of the second reference speech frame, which can reduce the amount of data calculation and help to further improve the efficiency of obtaining the second decoding features.

[0049] In another implementation scenario, instead of selecting the speech frame to which the coding feature aligned with the first decoding feature belongs and the first number or second number of speech frames before it to obtain the first reference speech frame or the second reference speech frame, it is also possible to select all speech frames before the speech frame to which the coding feature aligned with the first decoding feature belongs as the third reference speech frame, so that the second decoding feature can be obtained by fusing the coding feature of the third reference speech frame with the first decoding feature.

[0050] S13: Decode based on the second decoding feature to obtain the decoded characters output in the current round, and return to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and obtain the recognized text of the speech to be recognized based on the decoded characters of each round.

[0051] In an implementation scenario, the recognition text of the speech to be recognized is predicted by the target speech recognition model. Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the target speech recognition model of the present application. The target speech recognition model includes at least a first encoding network and a first decoding network. The first decoding network includes a first decoding layer and a second decoding layer. The output end of the first encoding network is respectively connected to the input end of the first decoding layer and the input end of the second decoding layer. The output end of the first decoding layer is connected to the input end of the second decoding layer. The target speech recognition model is trained based on a first sample speech, and the first sample speech is annotated with a first target text; wherein the first encoding network is used to encode the input speech, the first decoding layer is at least used to decode the output alignment probability, and the second decoding layer is used to output the decoded characters. Therefore, the recognition text of the speech to be recognized is predicted according to the specifically trained target speech recognition model, which can improve the efficiency of speech recognition.

[0052] In a specific implementation scenario, the first encoding network may be a Conformer network or a Transformer network, and the first decoding network may be a Transformer network.

[0053] In a specific implementation scenario, the first encoding network obtains the encoding features of each speech frame in the speech to be recognized, the first decoding layer obtains the first decoding features of the current round, and decodes and outputs the alignment probabilities between the encoding features of each speech frame in the speech to be recognized and the first decoding features, and then the second decoding layer decodes and outputs the second decoding features, and then further decodes based on the second decoding features to obtain the decoded characters output by the current round, and returns to the step of obtaining the first decoding features of the current round by the first decoding layer for iteration until the latest output decoding characters represent the end of speech recognition, and finally, according to the decoded characters of each round, the recognition text of the speech to be recognized can be obtained. Therefore, the first decoding network of the target speech recognition model is divided into the first decoding layer and the second decoding layer, and the first decoding layer and the second decoding layer are designed to focus on learning and realizing different feature information, such as the first decoding layer mainly focuses on obtaining the alignment probability, and the second decoding layer mainly focuses on outputting the decoded characters, so that the internal structure of the first decoding network can be decoupled, and the flexibility of the speech recognition decoding process can be improved.

[0054] In a specific implementation scenario, the first decoding layer can also be used to decode and output intermediate decoding features. In this case, the first decoding layer outputs the alignment probability and the intermediate decoding features to the second decoding layer. The second decoding layer fuses the encoding features and the intermediate decoding features of the second reference speech frame to obtain the second decoding features. The specific process can be found in the relevant description of the aforementioned embodiment and will not be repeated here.

[0055] In a specific implementation scenario, please continue to refer to Figure 3 , the first decoding network may include multiple first decoding layers and multiple second decoding layers, and the number of the first decoding layers and the second decoding layers can be flexibly set according to actual needs. As a possible example, by designing 1 first decoding layer and 23 second decoding layers, a first decoding network including 24 decoding layers is obtained. Since the first decoding network only needs to learn the alignment probability between the encoding features and the decoding features in one decoding layer, the overall computing power of the target speech recognition model can be saved, and the multiple second decoding layers focus on learning the context feature information according to the alignment probability obtained by the first decoding layer, which can improve the accuracy of speech recognition. Therefore, the efficiency of speech recognition can be improved while improving the accuracy of speech recognition. Of course, the number of first decoding layers can also include 2 layers, 3 layers, 4 layers, etc., and the number of second decoding layers can also include 20 layers, 21 layers, 22 layers, 24 layers, etc., and the number of first decoding layers and second decoding layers is not specifically limited here.

[0056] In a specific implementation scenario, the first decoding layer and the second decoding layer adopt different attention mechanisms. For example, the first decoding layer adopts a monotonic attention mechanism, and the second decoding layer adopts a cross attention mechanism. Therefore, the first decoding layer and the second decoding layer learn different feature information by adopting different attention mechanisms, which can decouple the internal structure of the first decoding network and improve the flexibility of the speech recognition decoding process.

[0057] In a specific implementation scenario, before training the target speech recognition model, the complete speech sequence can be intercepted by a sliding window to obtain several speech segments, and at least several of the speech segments can be used as the first sample speech. Therefore, by means of a dynamic window, different speech data are randomly selected and truncated with windows of different sizes, which can improve the compatibility of the target speech recognition model with speech recognition delays caused by different application scenarios, especially in streaming speech recognition scenarios. In addition, the complete speech sequence and the speech segments intercepted based on the complete speech sequence can be used as the first sample speech, or the speech segments obtained by other means (such as segmenting the speech sequence by speech endpoint detection technology) can be used as the first sample speech. The source of the first sample speech is not limited here.

[0058] For example, see Figure 4 , Figure 4 for Figure 3 Schematic diagram of a sample speech of a target speech recognition model in an embodiment of Figure 4 In the figure, the complete speech sequence is "abcdef". After intercepting the complete speech sequence through the sliding window, multiple speech segments "00abcd", "0abcde", "abcdef", and "bcdef0" are obtained respectively. Each square represents a data window of several frames in length, 0 represents padding data, and the length of each speech segment is consistent. The third column of speech segments "abcde" represents the current window of the complete speech sequence, and the fourth column of speech segments "bcdef0" represents the future window spliced ​​by the current window. By splicing the future window, it is possible to enrich the context information of the current window without adding too much delay. The first column of speech segments "00abcd" and the second column of speech segments "0abcde" are both historical windows spliced ​​by the current window. The historical window can be determined according to the specific task. For tasks that rely on longer time context, more historical windows can be spliced. Of course, the lengths of the intercepted speech segments can also be inconsistent, such as multiple speech segments can be "abcd", "abcde", "def", etc. Therefore, by dividing the complete speech sequence into several speech segments that lack contextual information, it is possible to simulate streaming speech in actual application scenarios, thereby helping to improve the robustness of the target speech recognition model for speech recognition.

[0059] In a specific implementation scenario, the first target text can be obtained by annotation by a labeler, or by transcribing the first sample speech through a general speech recognition model. For example, the first sample speech can be recognized by the first speech recognition model or the second speech recognition model mentioned below. The method for obtaining the first target text is not limited here.

[0060] In a specific implementation scenario, the speech data contained in the first sample speech can also be expanded by data enhancement. For example, data enhancement can be performed by adding noise, reverberation, speed disturbance, volume disturbance, frequency domain masking, etc., and the noise can be selected from white noise, music noise, etc. to simulate the actual application scenario. In addition, the noisy data of the first target text can be generated by operations such as word fragment deletion, random word fragment insertion, and word fragment disorder. Therefore, data enhancement can not only effectively enrich the training data, but also introduce noise and disturbance into the training of the target speech recognition model, thereby improving the robustness of the target speech recognition model in practical applications.

[0061] In an implementation scenario, please continue to refer to Figure 3 In order to train the target speech recognition model, the first sample speech can be recognized by the target speech recognition model to obtain the first predicted text, and the network parameters of the target speech recognition model can be adjusted according to the difference between the first predicted text and the first target text. Among them, the network parameters of the target speech recognition model can be adjusted by loss functions such as cross entropy. It should be noted that the step of the target speech recognition model recognizing the first sample speech to obtain the first predicted text is similar to the step of the target speech recognition model recognizing the speech to be recognized to obtain the recognized text, which will not be repeated here. In addition, for the encoding features, first decoding features, alignment probabilities, second decoding features, and decoded characters generated in the process of the target speech recognition model recognizing the speech to be recognized, the corresponding sample encoding features, first sample decoding features, sample alignment probabilities, second sample decoding features, sample decoded characters, etc. will also be generated in the training process of the target speech recognition model recognizing the first sample speech, which will not be repeated here.

[0062] In a specific implementation scenario, the target speech recognition model is obtained by replacing the decoding network based on the trained first speech recognition model, the first speech recognition model has an encoding network with the same structure as the target speech recognition model, and a decoding network with a different structure from the target speech recognition model, the decoding network of the first speech recognition model only includes the second decoding layer, the first speech recognition model is obtained based on the second sample speech training, and the second sample speech is annotated with the second target text. As a possible implementation method, the second sample speech is a complete speech sequence, and the method for obtaining the second target text is similar to the method for obtaining the first target text, which will not be repeated here. Therefore, since the encoding network of the first speech recognition model is the same as the first encoding network, the target speech recognition model is obtained by replacing the decoding network with the trained first speech recognition model. Compared with the target speech recognition model obtained by direct training, the training speed of the target speech recognition model can be improved, thereby improving the efficiency of speech recognition.

[0063] In another specific implementation scenario, the target speech recognition model is obtained based on the trained second speech recognition model, the target speech recognition model and the second speech recognition model have the same network structure, the second speech recognition model is obtained based on the third sample speech training, and the third sample speech is annotated with a third target text. Among them, the third sample speech is similar to the first sample speech in form and acquisition method, and the third target text is similar to the acquisition method of the first target text, which will not be repeated here. In this case, in order to train the second speech recognition model, the third sample speech can be recognized by the second speech recognition model to obtain the second predicted text, and according to the difference between the second predicted text and the third target text, the network parameters of the first decoding network in the second speech recognition model are adjusted, and the network parameters of the first encoding network in the second speech recognition model are fixed. Therefore, before training the target speech recognition model, the second speech recognition model with the same network structure as the target speech recognition model is locally trained, that is, only the network parameters of the decoding network are trained, and the network parameters of the freezing encoding network are not gradient updated, which can gradually improve the adaptability of the decoding network of the target speech recognition model to data.

[0064] As a possible implementation, in the process of recognizing the third sample speech through the second speech recognition model to obtain the second predicted text, the sample speech frame to which the sample encoding feature aligned with the first sample decoding feature in the current sample round belongs can be selected as the sample target speech frame through the sample alignment probability in the current sample round, and based on whether the current sample round reaches the target sample round, the sample speech frame is selected forward and / or backward based on the sample target speech frame to obtain the sample reference speech frame. Therefore, adopting different training strategies at different stages of the second speech recognition model training can improve the flexibility of the second speech recognition model training.

[0065] As a possible example, depending on the different detection results of whether the current sample round reaches the target sample round, the method of obtaining the sample reference speech frame is also different. When the current sample round does not reach the target sample round, the sample target speech frame is used as a reference and the sample speech frames of the first sample number are selected backward and the sample speech frames of the second sample number are selected forward as the sample reference speech frames; when the current sample round reaches the target sample round, the sample target speech frame and the sample speech frames of the second sample number before it are selected as the sample reference speech frames. Therefore, the second speech recognition model is divided into two training stages according to whether the target sample round is reached, and different training strategies are adopted in these two training stages. In particular, before the target sample round is not reached, the sample target speech frame is used as a reference and the sample speech frames of the first sample number are selected backward as the sample reference speech frames. This can achieve the use of some future information as a reference in the early stage of training the second speech recognition model, thereby improving the accuracy of speech recognition of the second speech recognition model.

[0066] As another possible example, the second speech recognition model can also be divided into two training stages according to whether the difference between the second predicted text and the third target text is greater than a preset difference threshold, and different sample reference speech frames can be obtained in different training stages, that is, when the difference between the second predicted text and the third target text is greater than the preset difference threshold, the sample target speech frame is used as a reference and the first sample number of sample speech frames are selected backward and the second sample number of sample speech frames are selected forward as sample reference speech frames; when the difference between the second predicted text and the third target text is less than or equal to the preset difference threshold, the sample target speech frame and the second sample number of sample speech frames before it are selected as sample reference speech frames.

[0067] As another possible implementation, in the process of recognizing the third sample speech through the second speech recognition model and obtaining the second predicted text, the sample alignment probability between the sample coding features output by each attention head in the multi-head attention mechanism and the first sample decoding feature in the current sample round can also be obtained. And according to the sample alignment probability output by the attention head, the sample speech frame to which the sample coding feature aligned with the first sample decoding feature belongs is determined as the sample candidate speech frame in the attention head, and finally the sample candidate speech frame with the largest frame number is selected as the sample target speech frame. Therefore, under the multi-head attention mechanism, the alignment moment is selected the latest, that is, the alignment moment of the attention head with the largest delay, and all attention heads pay attention to the same maximum delay moment, so that each attention head can obtain enough information for identifying the predicted text, thereby improving the accuracy of speech recognition.

[0068] As another possible implementation, in the process of recognizing the third sample speech through the second speech recognition model to obtain the second predicted text, the sample encoding features aligned with the first sample decoding features in the current sample round can be used to select the sample speech frame to which the sample encoding features aligned with the first sample decoding features belong as the sample target speech frame, and the sample encoding features of the sample target speech frame and all previous sample speech frames and the first sample decoding features are fused to obtain the second sample decoding features. Therefore, taking the alignment moment determined by the alignment probability as a reference, looking forward at all historical encoding features can help the second speech recognition model establish longer long-term dependencies, learn richer contextual feature information, and improve the accuracy of speech recognition.

[0069] In another specific implementation scenario, the target speech recognition model is obtained based on the trained second speech recognition model, the second speech recognition model is obtained by replacing the decoding network based on the trained first speech recognition model, the target speech recognition model and the second speech recognition model have the same network structure, the first speech recognition model has an encoding network with the same structure as the second speech recognition model, and a decoding network with a different structure from the second speech recognition model, the decoding network of the first speech recognition model only includes the second decoding layer, the first speech recognition model is obtained based on the training of the second sample speech, the second sample speech is annotated with the second target text, the second speech recognition model is obtained based on the training of the third sample speech, and the third sample speech is annotated with the third target text. Among them, the first sample speech includes a complete speech sequence and a speech segment intercepted by a sliding window, and the second sample speech and the third sample speech are both complete speech sequences. Therefore, by training the first speech recognition model, the second speech recognition model, and the target speech recognition model in sequence, the final target speech recognition model is obtained. The basic speech recognition ability of the first speech recognition model can be trained by using the complete speech sequence as the second sample speech, and then the decoding network of the first speech recognition model is further replaced by the decoding network of the second speech recognition model. The second speech recognition model is trained using the complete speech sequence as the third sample speech, so that the replaced decoding network learns monotonic time alignment on the whole sentence of speech, and then transitions to training the target speech recognition model through the first sample speech including the complete speech sequence and speech fragments, which helps to improve the stability and convergence speed of the target speech recognition model training.

[0070] The above scheme obtains the first decoding feature of the current round, wherein the first decoding feature of the current round includes the feature information of the decoded characters output in the previous round, and obtains the alignment probability between the coding features of each speech frame in the speech to be recognized and the first decoding feature, so as to select the coding features of at least part of the speech frames based on the alignment probability to merge with the first decoding feature to obtain the second decoding feature, and then decode based on the second decoding feature to obtain the decoded characters output in the current round, and return to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and finally obtain the recognized text of the speech to be recognized based on the decoded characters of each round. Therefore, on the one hand, compared with referring to all speech frames, selecting the coding features of at least part of the speech frames to merge with the first decoding features can improve data processing efficiency. On the other hand, selecting coding features to merge with the first decoding features according to the alignment probability can help improve the data validity of the decoding process of speech recognition, thereby improving the accuracy of speech recognition. In the scenario of streaming speech recognition, it is possible to selectively select coding features at and before the alignment moment with the first decoding feature of the current round according to the alignment probability, thereby reducing the reliance on coding features after the alignment moment in the speech recognition process, thereby improving the accuracy of streaming speech recognition. Therefore, it is possible to improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition.

[0071] See also Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of a speech recognition device of the present application, wherein the speech recognition device 50 includes a first acquisition module 51, a second acquisition module 52 and a decoding module 53, wherein the first acquisition module 51 is used to acquire the first decoding feature of the current round; wherein the first decoding feature of the current round includes feature information of the decoded characters output in the previous round; the second acquisition module 52 is used to acquire the alignment probability between the encoding features of each speech frame in the speech to be recognized and the first decoding feature, respectively, and based on the alignment probability, select the encoding features of at least part of the speech frames to be fused with the first decoding feature to obtain the second decoding feature; the decoding module 53 is used to decode based on the second decoding feature to obtain the decoded characters output in the current round, and return to the step of acquiring the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and based on the decoded characters of each round, obtain the recognized text of the speech to be recognized.

[0072] In the above scheme, the speech recognition device 50 obtains the first decoding feature of the current round, wherein the first decoding feature of the current round includes the feature information of the decoded characters output in the previous round, and obtains the alignment probability between the encoding features of each speech frame in the speech to be recognized and the first decoding feature, so as to select the encoding features of at least part of the speech frames based on the alignment probability to merge with the first decoding feature to obtain the second decoding feature, and then decodes based on the second decoding feature to obtain the decoded characters output in the current round, and returns to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and finally obtains the recognized text of the speech to be recognized based on the decoded characters of each round. Therefore, on the one hand, compared with referring to all speech frames, selecting the coding features of at least part of the speech frames to merge with the first decoding features can improve data processing efficiency. On the other hand, selecting coding features to merge with the first decoding features according to the alignment probability can help improve the data validity of the decoding process of speech recognition, thereby improving the accuracy of speech recognition. In the scenario of streaming speech recognition, it is possible to selectively select coding features at and before the alignment moment with the first decoding feature of the current round according to the alignment probability, thereby reducing the reliance on coding features after the alignment moment in the speech recognition process, thereby improving the accuracy of streaming speech recognition. Therefore, it is possible to improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition.

[0073] In some disclosed embodiments, the second acquisition module 52 includes a fusion submodule, which is used to select the coding features of at least part of the speech frames based on the alignment probability to fuse with the first decoding features to obtain the second decoding features, and specifically to select the speech frame to which the coding features aligned with the first decoding features belong and a first number of speech frames before it as the first reference speech frame based on the alignment probability; to fuse the coding features of the first reference speech frame and the first decoding features to obtain an intermediate decoding feature; to select the speech frame to which the coding features aligned with the first decoding features belong as the target speech frame based on the alignment probability, and to select the target speech frame and a second number of speech frames before it as the second reference speech frame; and to fuse the coding features of the second reference speech frame and the intermediate decoding features to obtain the second decoding feature.

[0074] Therefore, after obtaining the intermediate decoding feature through the first decoding feature, and then obtaining the second decoding feature through the intermediate decoding feature, the first decoding feature can be decoupled from the second decoding feature, thereby achieving the goal of focusing on obtaining the alignment probability in the process of obtaining the intermediate decoding, and focusing on obtaining the context information in the process of obtaining the second decoding feature, so as to improve the flexibility of speech recognition.

[0075] In some disclosed embodiments, the alignment probability is obtained by a multi-head attention mechanism, and each attention head outputs the alignment probability between each encoding feature and the first decoding feature. The fusion submodule is specifically used to obtain the candidate decoding features of the attention head based on the first decoding feature and the encoding features of the first reference speech frame determined by the attention head; and to obtain the intermediate decoding features based on the candidate decoding features of each attention head.

[0076] Therefore, the multi-head attention mechanism can capture information in different representation subspaces by executing multiple attention functions in parallel, and the dimension of each attention head is low, which can improve the efficiency of acquiring intermediate decoding features.

[0077] In some disclosed embodiments, the alignment probability is obtained by a multi-head attention mechanism, and each attention head outputs the alignment probability between each encoding feature and the first decoding feature. The fusion submodule is also specifically used to determine, based on the alignment probability of each attention head, the speech frames to which the encoding features aligned with the first decoding feature in each attention head belong as candidate speech frames; and select the candidate speech frame with the largest frame number as the target speech frame.

[0078] Therefore, on the one hand, the multi-head attention mechanism can capture information in different representation subspaces by executing multiple attention functions in parallel, and the dimension of each attention head is low, which can improve the efficiency of obtaining the second decoding features; on the other hand, all attention heads share the encoding features of the second reference speech frame, which can reduce the amount of data calculation and help to further improve the efficiency of obtaining the second decoding features.

[0079] In some disclosed embodiments, the recognition text of the speech to be recognized is predicted by a target speech recognition model, the target speech recognition model includes at least a first encoding network and a first decoding network, the first decoding network includes a first decoding layer and a second decoding layer, the output end of the first encoding network is respectively connected to the input end of the first decoding layer and the input end of the second decoding layer, the output end of the first decoding layer is connected to the input end of the second decoding layer, the target speech recognition model is trained based on a first sample speech, and the first sample speech is annotated with a first target text; wherein the first encoding network is used to encode the input speech, the first decoding layer is at least used to decode the output alignment probability, and the second decoding layer is used to output the decoded characters.

[0080] Therefore, predicting the recognition text of the speech to be recognized based on the specifically trained target speech recognition model can improve the efficiency of speech recognition.

[0081] In some disclosed embodiments, the speech recognition device 50 also includes a training module, which is used to intercept the complete speech sequence based on a sliding window to obtain several speech segments before training the target speech recognition model, and at least use several speech segments as the first sample speech.

[0082] Therefore, by using a dynamic window method to randomly select windows of different sizes for truncation of different speech data, the target speech recognition model can be improved in its ability to perform speech recognition with delays caused by different application scenarios, especially in streaming speech recognition scenarios.

[0083] In some disclosed embodiments, a target speech recognition model is obtained by replacing a decoding network based on a trained first speech recognition model, the first speech recognition model has an encoding network with the same structure as the target speech recognition model, and a decoding network with a different structure from the target speech recognition model, the decoding network of the first speech recognition model only includes a second decoding layer, the first speech recognition model is trained based on a second sample speech, and the second sample speech is annotated with a second target text.

[0084] Therefore, since the encoding network of the first speech recognition model is the same as the first encoding network, the target speech recognition model is obtained by replacing the decoding network with the trained first speech recognition model. Compared with directly training the target speech recognition model, the training speed of the target speech recognition model can be improved, thereby improving the efficiency of speech recognition.

[0085] In some disclosed embodiments, a target speech recognition model is obtained based on a trained second speech recognition model, the target speech recognition model and the second speech recognition model have the same network structure, the second speech recognition model is obtained based on a third sample speech training, and the third sample speech is annotated with a third target text, and the training module is also used to train the second speech recognition model, and the training steps include: recognizing the third sample speech based on the second speech recognition model to obtain a second predicted text; based on the difference between the second predicted text and the third target text, adjusting the network parameters of the first decoding network in the second speech recognition model, and fixing the network parameters of the first encoding network in the second speech recognition model.

[0086] Therefore, before training the target speech recognition model, the adaptability of the decoding network of the target speech recognition model to data can be gradually improved by locally training a second speech recognition model with the same network structure as the target speech recognition model, that is, only training the network parameters of the decoding network, while freezing the network parameters of the encoding network without gradient updating.

[0087] In some disclosed embodiments, in the process of the training module executing recognition of the third sample speech based on the second speech recognition model to obtain the second predicted text, it is also specifically used to select, based on the sample alignment probability in the current sample round, a sample speech frame belonging to a sample encoding feature that is aligned with the first sample decoding feature in the current sample round as a sample target speech frame; based on whether the current sample round reaches the target sample round, select sample speech frames forward and / or backward based on the sample target speech frame to obtain a sample reference speech frame.

[0088] Therefore, adopting different training strategies at different stages of the second speech recognition model training can improve the flexibility of the second speech recognition model training.

[0089] In some disclosed embodiments, the training module is specifically used to select sample speech frames forward and / or backward based on whether the current sample round reaches the target sample round, with the sample target speech frame as a reference, to obtain sample reference speech frames, including at least one of the following: in response to the current sample round not reaching the target sample round, with the sample target speech frame as a reference and selecting a first sample number of sample speech frames backward and a second sample number of sample speech frames forward as sample reference speech frames; in response to the current sample round reaching the target sample round, selecting the sample target speech frame and the second sample number of sample speech frames before it as sample reference speech frames.

[0090] Therefore, the second speech recognition model is divided into two training stages according to whether the target sample round is reached, and different training strategies are adopted in these two training stages. In particular, before the target sample round is reached, the sample target speech frame is used as a benchmark and the first sample number of sample speech frames are selected backward as sample reference speech frames. This allows part of the future information to be used as a reference in the early stage of training the second speech recognition model, thereby improving the accuracy of speech recognition by the second speech recognition model.

[0091] In some disclosed embodiments, in the process of the training module executing recognition of the third sample speech based on the second speech recognition model to obtain the second predicted text, it is also specifically used to obtain the sample alignment probability between the sample encoding features output by each attention head in the multi-head attention mechanism and the first sample decoding feature in the current sample round; based on the sample alignment probability output by the attention head, determine the sample speech frame to which the sample encoding feature aligned with the first sample decoding feature belongs as the sample candidate speech frame in the attention head; select the sample candidate speech frame with the largest frame number as the sample target speech frame.

[0092] Therefore, under the multi-head attention mechanism, the alignment time is selected at the latest, that is, the alignment time of the attention head with the largest delay. All attention heads focus on the same maximum delay time, so that each attention head can obtain enough information to identify the predicted text and improve the accuracy of speech recognition.

[0093] In some disclosed embodiments, in the process of the training module executing recognition of the third sample speech based on the second speech recognition model to obtain the second predicted text, the training module is also specifically used to select the sample speech frame to which the sample coding feature aligned with the first sample decoding feature belongs as the sample target speech frame based on the sample alignment probability between each sample coding feature and the first sample decoding feature in the current sample round; and to fuse the sample coding features of the sample target speech frame and all previous sample speech frames with the first sample decoding feature to obtain the second sample decoding feature.

[0094] Therefore, taking the alignment moment determined by the alignment probability as a benchmark and looking forward at all historical encoding features can help the second speech recognition model establish longer long-term dependencies, learn richer contextual feature information, and improve the accuracy of speech recognition.

[0095] In some disclosed embodiments, a target speech recognition model is obtained based on a trained second speech recognition model, the second speech recognition model is obtained by replacing a decoding network based on a trained first speech recognition model, the target speech recognition model and the second speech recognition model have the same network structure, the first speech recognition model has an encoding network with the same structure as the second speech recognition model, and a decoding network with a different structure from the second speech recognition model, the decoding network of the first speech recognition model only includes a second decoding layer, the first speech recognition model is obtained by training based on a second sample speech, the second sample speech is annotated with a second target text, and the second speech recognition model is obtained by training based on a third sample speech, and the third sample speech is annotated with a third target text.

[0096] Therefore, by training the first speech recognition model, the second speech recognition model, and the target speech recognition model in sequence, the final target speech recognition model is obtained. The basic speech recognition ability of the first speech recognition model can be trained by using the complete speech sequence as the second sample speech, and then the decoding network of the first speech recognition model is further replaced by the decoding network of the second speech recognition model. The second speech recognition model is trained using the complete speech sequence as the third sample speech, so that the replaced decoding network learns monotonic time alignment on the whole sentence of speech, and then transitions to training the target speech recognition model through the first sample speech including the complete speech sequence and speech fragments, which helps to improve the stability and convergence speed of the target speech recognition model training.

[0097] See also Figure 6 , Figure 6 Schematic diagram of the framework of an embodiment of an electronic device 60 of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in any of the above-mentioned speech recognition method embodiments. Specifically, the electronic device 60 may include but is not limited to: a microcomputer, a server, a laptop computer, a tablet computer, a smart phone, etc., which are not limited here.

[0098] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any of the above-mentioned speech recognition method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0099] In the above scheme, the electronic device 60 obtains the first decoding feature of the current round, wherein the first decoding feature of the current round includes the feature information of the decoded characters output in the previous round, and obtains the alignment probability between the coding features of each speech frame in the speech to be recognized and the first decoding feature, so as to select the coding features of at least part of the speech frames based on the alignment probability to merge with the first decoding feature to obtain the second decoding feature, and then decodes based on the second decoding feature to obtain the decoded characters output in the current round, and returns to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and finally obtains the recognized text of the speech to be recognized based on the decoded characters of each round. Therefore, on the one hand, compared with referring to all speech frames, selecting the coding features of at least part of the speech frames to merge with the first decoding features can improve data processing efficiency. On the other hand, selecting coding features to merge with the first decoding features according to the alignment probability can help improve the data validity of the decoding process of speech recognition, thereby improving the accuracy of speech recognition. In the scenario of streaming speech recognition, it is possible to selectively select coding features at and before the alignment moment with the first decoding feature of the current round according to the alignment probability, thereby reducing the reliance on coding features after the alignment moment in the speech recognition process, thereby improving the accuracy of streaming speech recognition. Therefore, it is possible to improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition.

[0100] See also Figure 7 , Figure 7 The schematic diagram of the framework of an embodiment of a computer-readable storage medium 70 of the present application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps of the above-mentioned speech recognition method embodiment.

[0101] In the above scheme, the computer-readable storage medium 70 obtains the first decoding feature of the current round, wherein the first decoding feature of the current round includes the feature information of the decoded characters output in the previous round, and obtains the alignment probability between the encoding features of each speech frame in the speech to be recognized and the first decoding feature, so as to select the encoding features of at least part of the speech frames based on the alignment probability to merge with the first decoding feature to obtain the second decoding feature, and then decode based on the second decoding feature to obtain the decoded characters output in the current round, and return to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded characters represent the end of speech recognition, and finally obtain the recognized text of the speech to be recognized based on the decoded characters of each round. Therefore, on the one hand, compared with referring to all speech frames, selecting the coding features of at least part of the speech frames to merge with the first decoding features can improve data processing efficiency. On the other hand, selecting coding features to merge with the first decoding features according to the alignment probability can help improve the data validity of the decoding process of speech recognition, thereby improving the accuracy of speech recognition. In the scenario of streaming speech recognition, it is possible to selectively select coding features at and before the alignment moment with the first decoding feature of the current round according to the alignment probability, thereby reducing the reliance on coding features after the alignment moment in the speech recognition process, thereby improving the accuracy of streaming speech recognition. Therefore, it is possible to improve the efficiency and accuracy of speech recognition, especially the efficiency and accuracy of streaming speech recognition.

[0102] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0103] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0105] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0107] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech recognition method, characterized in that: include: Obtaining a first decoding feature of the current round; wherein the first decoding feature of the current round includes feature information of a decoded character outputted in a previous round; Obtaining the alignment probability between the encoding features of each speech frame in the speech to be recognized and the first decoding feature, and selecting at least part of the encoding features of the speech frames to be merged with the first decoding feature based on the alignment probability to obtain a second decoding feature; Decoding is performed based on the second decoding feature to obtain the decoded characters output in the current round, and the step of obtaining the first decoding feature of the current round is returned to iterate until the latest output decoding character represents the end of speech recognition, and the recognition text of the speech to be recognized is obtained based on the decoded characters of each round.

2. The method according to claim 1, characterized in that The step of selecting, based on the alignment probability, at least part of the encoding features of the speech frames and fusing them with the first decoding features to obtain a second decoding feature comprises: Based on the alignment probability, selecting a speech frame to which the encoding feature aligned with the first decoding feature belongs and a first number of speech frames before it as a first reference speech frame; fusing the encoding feature of the first reference speech frame and the first decoding feature to obtain an intermediate decoding feature; Based on the alignment probability, select a speech frame to which the encoding feature aligned with the first decoding feature belongs as a target speech frame, and select the target speech frame and a second number of speech frames before the target speech frame as second reference speech frames; The second decoding feature is obtained by fusing the encoding feature of the second reference speech frame and the intermediate decoding feature.

3. The method according to claim 2, characterized in that The alignment probability is obtained by a multi-head attention mechanism, and each attention head outputs the alignment probability between each of the encoding features and the first decoding feature, and the encoding feature based on the first reference speech frame and the first decoding feature are fused to obtain an intermediate decoding feature, including: Obtaining candidate decoding features of the attention head based on the first decoding features and the encoding features of the first reference speech frame determined by the attention head; The candidate decoding features of each of the attention heads are fused to obtain the intermediate decoding features.

4. The method according to claim 2, characterized in that: The alignment probability is obtained by a multi-head attention mechanism, and each attention head outputs the alignment probability between each of the encoding features and the first decoding feature, and based on the alignment probability, selecting the speech frame to which the encoding feature aligned with the first decoding feature belongs as the target speech frame, including: Based on the alignment probabilities of the attention heads, respectively determine the speech frames to which the encoding features aligned with the first decoding features in the attention heads belong as candidate speech frames; The candidate speech frame with the largest frame number is selected as the target speech frame.

5. The method according to claim 1, characterized in that The recognition text of the speech to be recognized is predicted by a target speech recognition model, the target speech recognition model at least includes a first encoding network and a first decoding network, the first decoding network includes a first decoding layer and a second decoding layer, the output end of the first encoding network is respectively connected to the input end of the first decoding layer and the input end of the second decoding layer, the output end of the first decoding layer is connected to the input end of the second decoding layer, the target speech recognition model is trained based on a first sample speech, and the first sample speech is annotated with a first target text; The first encoding network is used to encode the input speech, the first decoding layer is used at least to decode and output the alignment probability, and the second decoding layer is used to output the decoded characters.

6. The method according to claim 5, characterized in that Before training the target speech recognition model, the method further includes: The complete speech sequence is intercepted based on the sliding window to obtain a number of speech segments, and at least the number of speech segments are used as the first sample speech.

7. The method according to claim 5, characterized in that The target speech recognition model is obtained by replacing the decoding network based on the trained first speech recognition model. The first speech recognition model has an encoding network with the same structure as the target speech recognition model, and a decoding network with a different structure from the target speech recognition model. The decoding network of the first speech recognition model only includes the second decoding layer. The first speech recognition model is trained based on a second sample speech, and the second sample speech is annotated with a second target text.

8. The method according to claim 5, characterized in that The target speech recognition model is obtained based on the trained second speech recognition model, the target speech recognition model and the second speech recognition model have the same network structure, the second speech recognition model is obtained based on the third sample speech training, and the third sample speech is annotated with a third target text, and the training steps of the second speech recognition model include: Recognize the third sample speech based on the second speech recognition model to obtain a second predicted text; Based on the difference between the second predicted text and the third target text, the network parameters of the first decoding network in the second speech recognition model are adjusted, and the network parameters of the first encoding network in the second speech recognition model are fixed.

9. The method according to claim 8, characterized in that In the process of performing the step of recognizing the third sample speech based on the second speech recognition model to obtain the second predicted text, the method further includes: Based on the sample alignment probability in the current sample round, selecting a sample speech frame to which a sample encoding feature aligned with a first sample decoding feature in the current sample round belongs as a sample target speech frame; Based on whether the current sample round reaches the target sample round, the sample speech frame is selected forward and / or backward based on the sample target speech frame to obtain a sample reference speech frame.

10. The method according to claim 9, characterized in that The step of selecting the sample speech frame forward and / or backward based on whether the current sample round reaches the target sample round and taking the sample target speech frame as a reference to obtain the sample reference speech frame includes at least one of the following: In response to the current sample round not reaching the target sample round, taking the sample target speech frame as a reference and selecting a first sample number of sample speech frames backward and a second sample number of sample speech frames forward as the sample reference speech frames; In response to the current sample round reaching the target sample round, the sample target speech frame and a second number of sample speech frames before it are selected as the sample reference speech frames.

11. The method according to claim 8, characterized in that In the process of performing the step of recognizing the third sample speech based on the second speech recognition model to obtain the second predicted text, the method further includes: Obtain the sample alignment probability between the sample encoding features output by each attention head in the multi-head attention mechanism and the first sample decoding feature in the current sample round; Based on the sample alignment probability output by the attention head, determine the sample speech frame to which the sample encoding feature aligned with the first sample decoding feature belongs as the sample candidate speech frame in the attention head; The sample candidate speech frame with the largest frame number is selected as the sample target speech frame.

12. The method according to claim 8, characterized in that In the process of performing the step of recognizing the third sample speech based on the second speech recognition model to obtain the second predicted text, the method further includes: Based on the sample alignment probabilities between each sample coding feature and the first sample decoding feature in the current sample round, select the sample speech frame to which the sample coding feature aligned with the first sample decoding feature belongs as the sample target speech frame; The sample encoding features of the sample target speech frame and all previous sample speech frames are fused with the first sample decoding features to obtain second sample decoding features.

13. The method according to claim 5, characterized in that The target speech recognition model is obtained based on a trained second speech recognition model, and the second speech recognition model is obtained by replacing a decoding network based on the trained first speech recognition model. The target speech recognition model and the second speech recognition model have the same network structure, the first speech recognition model has an encoding network with the same structure as the second speech recognition model, and a decoding network with a different structure from the second speech recognition model, the decoding network of the first speech recognition model only includes the second decoding layer, the first speech recognition model is obtained based on a second sample speech training, and the second sample speech is annotated with a second target text, and the second speech recognition model is obtained based on a third sample speech training, and the third sample speech is annotated with a third target text.

14. A speech recognition device, characterized in that: include: A first acquisition module, used to acquire a first decoding feature of a current round; wherein the first decoding feature of the current round includes feature information of a decoded character outputted in a previous round; A second acquisition module is used to obtain the alignment probability between the coding features of each speech frame in the speech to be recognized and the first decoding feature, and select the coding features of at least part of the speech frames to be merged with the first decoding feature based on the alignment probability to obtain a second decoding feature; A decoding module is used to decode based on the second decoding feature to obtain the decoded characters output by the current round, and return to the step of obtaining the first decoding feature of the current round to iterate until the latest output decoded character represents the end of speech recognition, and obtain the recognized text of the speech to be recognized based on the decoded characters of each round.

15. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech recognition method according to any one of claims 1 to 13.

16. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the speech recognition method according to any one of claims 1 to 13 is implemented.

Images