Speech recognition method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice recognition method and device, equipment and a storage medium, and the method comprises the steps: receiving a target voice signal which comprises the voice signals of one or more speakers; obtaining a target acoustic feature sequence of the target voice signal; generating a target coding sequence according to the target acoustic feature sequence; generating a masking text corresponding to the target voice signal according to the target coding sequence; and generating speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text. According to the method, the inherent problems of context fragmentation and semantic confusion of a multi-speaker automatic speech recognition system can be effectively reduced, and the automatic speech recognition performance is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of speech signal technology, and in particular to a speech recognition method and device, an electronic device and a storage medium. Background Art

[0002] With the rapid development of large language models (LLM) and deep neural networks, speech is becoming an important interface for people to interact with machines, especially real-time (or streaming) automatic speech recognition (ASR) with speaker identity information, which has produced extremely extensive and important applications. More and more applications not only need to obtain what the user said, but also need to identify the speaker's identity to provide personalized services. In addition, training such as LLM requires a large amount of text data, which also relies on a robust and reliable ASR system.

[0003] Existing streaming end-to-end multi-speaker automatic speech recognition systems still rely on clustering-based log modules or additional speaker embedding vector extraction modules, and streaming application scenarios mean that the system faces limited historical information. This makes the ASR system prone to semantic confusion problems during frequent alternating conversations between multiple people, and the resulting historical errors will accumulate, greatly affecting system performance.

[0004] Therefore, how to reduce the semantic confusion problem that may occur in automatic speech recognition during frequent alternating conversations between multiple people and improve ASR performance has become an urgent problem to be solved. Summary of the invention

[0005] The embodiment of the present application provides a speech recognition method to solve the semantic confusion problem that may occur in automatic speech recognition during frequent alternating conversations between multiple people.

[0006] Correspondingly, the embodiments of the present application also provide a speech recognition device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application discloses a speech recognition method, which includes:

[0008] receiving a target speech signal, wherein the target speech signal includes speech signals of one or more speakers;

[0009] Acquiring a target acoustic feature sequence of a target speech signal;

[0010] generating a target coding sequence according to the target acoustic feature sequence;

[0011] generating a masking text corresponding to a target speech signal according to a target coding sequence;

[0012] Generate speech recognition content corresponding to one or more speakers based on the target encoding sequence and the masking text.

[0013] Optionally, obtaining a target acoustic feature sequence of a target speech signal includes:

[0014] Obtaining the frequency spectrum characteristics of the target speech signal;

[0015] A target acoustic feature sequence of the target speech signal is extracted from the spectral features.

[0016] Optionally, generating a target coding sequence according to the target acoustic feature sequence includes:

[0017] The target acoustic feature sequence is encoded to generate a target encoding sequence.

[0018] Optionally, encoding the target acoustic feature sequence to generate a target encoding sequence includes:

[0019] Inputting the target acoustic feature sequence into a first encoding block of an encoder to generate a first encoding sequence;

[0020] generating a first state sequence according to a first encoding sequence;

[0021] generating a first fine-tuning sequence according to the first encoding sequence;

[0022] splicing the first state sequence and the first fine-tuning sequence to obtain a first spliced ​​sequence;

[0023] The first concatenated sequence is fed into the next coding block until the Nth concatenated sequence is generated, and the Nth concatenated sequence is used as the target coding sequence; wherein N is the number of coding blocks stacked in the encoder, and N is an integer greater than 1.

[0024] Optionally, generating a masked text corresponding to the target speech signal according to the target coding sequence includes:

[0025] Predicting a predicted text corresponding to a target speech signal according to a target coding sequence;

[0026] Convert the predicted text into the masked text corresponding to the target speech signal.

[0027] Optionally, converting the predicted text into a masked text corresponding to the target speech signal includes:

[0028] Convert the target state sequence into a mask matrix; the mask matrix is ​​a mask matrix with the number of predicted speakers or speakers as the vertical axis and the length of the target acoustic feature as the horizontal axis;

[0029] The predicted text is masked using a mask matrix to obtain a masked text.

[0030] Optionally, speech recognition content corresponding to one or more speakers is generated according to the target encoding sequence and the masking text, including:

[0031] The target state sequence and the masked text are jointly decoded to obtain the speech recognition content corresponding to one or more speakers.

[0032] The present application also discloses a speech recognition device, which includes:

[0033] A receiving module, used for receiving a target speech signal, wherein the target speech signal includes speech signals of one or more speakers;

[0034] An acquisition module, used for acquiring a target acoustic feature sequence of a target speech signal;

[0035] A target coding sequence generation module, used for generating a target coding sequence according to a target acoustic feature sequence;

[0036] A masked text generation module, used to generate a masked text corresponding to a target speech signal according to a target coding sequence;

[0037] The recognition module is used to generate speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text.

[0038] An embodiment of the present application also discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, one or more speech recognition methods in the embodiments of the present application are implemented.

[0039] The embodiments of the present application also disclose one or more machine-readable media, on which a computer program is stored. When the computer program is executed by a processor, one or more speech recognition methods as described in the embodiments of the present application are implemented.

[0040] Compared with the prior art, the embodiments of the present application have the following advantages:

[0041] In an embodiment of the present application, a target speech signal is received, and the target speech signal includes speech signals of one or more speakers. A target acoustic feature sequence of the target speech signal is obtained, and a target coding sequence is generated based on the target acoustic feature sequence. The identity information of the speaker can be integrated into the end-to-end ASR system. A masked text corresponding to the target speech signal is generated based on the target coding sequence, and speech recognition content corresponding to one or more speakers is generated based on the target coding sequence and the masked text. The masked decoding strategy is used to effectively reduce the inherent context fragmentation and semantic confusion problems of the multi-speaker ASR system, and significantly improve the ASR performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flowchart of a speech recognition method embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of an ASR encoder in an embodiment of a speech recognition method of the present application;

[0044] Figure 3 It is an abstract schematic diagram of the encoding and decoding strategy of another speech recognition method embodiment of the present application;

[0045] Figure 4 is a flowchart of another embodiment of a speech recognition method of the present application;

[0046] Figure 5 It is a structural block diagram of an embodiment of a speech recognition device of the present application. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0048] Existing streaming end-to-end multi-speaker ASR systems still rely on clustering-based log modules or additional speaker embedding vector extraction modules. These non-differentiable modules often need to be optimized separately to obtain optimal performance. The existence of these modules limits the joint optimization of the entire system through gradient backpropagation, resulting in the inability to quickly deploy when facing some domain-specific datasets.

[0049] In addition, streaming application scenarios mean that the system faces limited historical information and cannot fully utilize mature speaker recognition technology to obtain robust speaker identity representation. In conversations where multiple speakers frequently alternate, streaming ASR systems trained using the t-SOT (Token-level Serialized Output Training) strategy often have semantic confusion problems when predicting speaker transition markers. The content of another person destroys the context of the previous speaker's content, resulting in a decline in ASR performance.

[0050] In addition, historical errors will accumulate during the recognition process of the streaming system, which will make the limited historical information even more unreliable. In the end-to-end multi-speaker ASR system based on the online clustering log module, this often manifests as more speaker counts and speaker assignment errors, which cannot be corrected, greatly affecting the system performance.

[0051] In order to effectively reduce the inherent context fragmentation and semantic confusion problems of multi-speaker automatic speech recognition systems and significantly improve the performance of automatic speech recognition, the present application proposes a speech recognition method.

[0052] Reference Figure 1 , is a flowchart of a speech recognition method embodiment of the present application, comprising the following steps:

[0053] Step 101: receiving a target speech signal, wherein the target speech signal includes speech signals of one or more speakers;

[0054] The target speech signal may be an audio signal containing frequent alternating conversations between multiple speakers in various complex application scenarios, such as conference recording audio, classroom recording, group discussion audio, etc.

[0055] Step 102: Acquire a target acoustic feature sequence of the target speech signal;

[0056] Among them, the target acoustic feature sequence contains key information in the target speech signal, which helps the model to recognize the speech content later.

[0057] Step 103: generating a target coding sequence according to the target acoustic feature sequence;

[0058] Among them, the target coding sequence is the hidden state feature sequence obtained after inputting the target acoustic features into the ASR encoder;

[0059] Step 104: generating a masked text corresponding to the target speech signal according to the target coding sequence;

[0060] The masked text is obtained by performing a masking operation on the text output sequence recognized from the target speech signal.

[0061] Step 105: Generate speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text.

[0062] When the target speech signal contains only one speaker's speech, a speech recognition text corresponding to the current speaker is generated. When the target speech signal contains multiple speakers' speech, speech recognition texts corresponding to different speakers are generated respectively.

[0063] The embodiment of the present application realizes multi-speaker speech recognition through the above steps. The multi-speaker speech recognition (Multi-speaker Automatic Speech Recognition, Multi-speaker ASR) system can recognize the speech of multiple speakers at the same time, separate it according to the speaker and convert it into text, which is suitable for multi-person conversation scenarios such as meetings and classrooms. When performing speech recognition, a target speech signal to be recognized can be received, and the target speech signal contains the speech signals of one or more speakers. The target acoustic feature sequence of the target speech signal is obtained, and the target coding sequence is generated according to the target acoustic feature sequence, and the speaker state is embedded in the acoustic coding feature of the streaming multi-speaker ASR. The masked text corresponding to the target speech signal is generated according to the target coding sequence; and the speech recognition content corresponding to one or more speakers is generated according to the target coding sequence and the masked text. Through the mask decoding strategy, the context of the same speaker is connected as much as possible without being interrupted by the speaker conversion mark, so as to effectively reduce the problems of context fragmentation and semantic confusion inherent in the multi-speaker automatic speech recognition system, and significantly improve the automatic speech recognition performance.

[0064] Optionally, in an embodiment of the present application, obtaining a target acoustic feature sequence of the target speech signal may include:

[0065] Acquire frequency spectrum characteristics of the target speech signal;

[0066] A target acoustic feature sequence of the target speech signal is extracted from the frequency spectrum features.

[0067] In this embodiment, the received target speech signal can be preprocessed, including framing, pre-emphasis, spectrum calculation, etc., so as to obtain the spectrum characteristics of the target speech signal. Exemplarily, the received target speech signal can be pre-emphasized by a first-order high-pass digital filter to enhance the high-frequency part of the speech signal and increase the high-frequency resolution of the speech. The pre-emphasized target speech signal can be divided into several short time frames, and each frame can be windowed. The windowed frame can also be Fourier transformed to obtain the spectrum characteristics of the target speech signal. After obtaining the spectrum characteristics of the target speech signal, the acoustic characteristics of the target speech signal are extracted from the spectrum characteristics to obtain the target acoustic feature sequence. Exemplarily, the acoustic features such as MFCC (Mel Frequency Cepstral Coefficients) or fBank (Mel Filter Bank, Mel filter bank coefficients) can be extracted by Mel filter. Through the above steps, the target speech signal is pre-processed and the acoustic features are extracted, and the feature parameters can be better extracted by removing noise and improving the high-frequency resolution, thereby improving the subsequent speech recognition effect.

[0068] Optionally, in an embodiment of the present application, generating a target coding sequence according to the target acoustic feature sequence may include:

[0069] The target acoustic feature sequence is encoded to generate the target encoding sequence.

[0070] In this embodiment, the extracted target acoustic feature sequence can be sent to an ASR encoder that integrates speaker states, the speaker states are encoded through a differentiable state space model, and the speaker states are embedded in the acoustic coding features of the streaming multi-speaker ASR to generate a target coding sequence.

[0071] Reference Figure 2 , the present application provides a schematic diagram of the local structure of an ASR encoder of an embodiment of a speech recognition method. Optionally, in an embodiment of the present application, encoding the target acoustic feature sequence to generate the target encoding sequence may include:

[0072] Inputting the target acoustic feature sequence into a first encoding block of an encoder to generate a first encoding sequence;

[0073] The encoder of the ASR model is composed of multiple ASR encoder blocks stacked together. Each encoder block processes a part of the input target acoustic features and passes the processing results to the next block. In this way, the encoder can gradually build a contextual representation of the entire input feature.

[0074] generating a first state sequence according to the first coding sequence;

[0075] The first coding sequence generated by the first coding block is input into the speaker state space model. As an effective tool for processing time series data, the state space model (SSM) can naturally express the dynamic process of the system changing over time, so it is very suitable for encoding and analyzing the speaker state. After the first coding sequence is processed by the speaker SSM, it will be generated and output as a first state sequence.

[0076] generating a first fine-tuning sequence according to the first encoding sequence;

[0077] The first coding sequence generated by the first coding block is input into the ASR Adapter module, and the ASR Adapter module processes and generates a first fine-tuning sequence. The ASR Adapter transforms the output of the coding block by compressing the dimension and enlarging the dimension, and fine-tunes the first coding sequence in this process. For example, the first coding sequence input into the ASR Adapter module may be 64-dimensional, compressed to 16-dimensional in the ASR Adapter module, and restored to 64-dimensional when output.

[0078] splicing the first state sequence and the first fine-tuning sequence to obtain a first spliced ​​sequence;

[0079] The first concatenated sequence is fed into the next coding block until an Nth coding sequence is generated, and the Nth coding sequence is used as the target coding sequence; wherein N is the number of coding blocks stacked in the encoder, and N is an integer greater than 1.

[0080] Among them, the ASR encoder is composed of N stacked encoding blocks. The input target acoustic features need to pass through each encoding block and finally output the hidden state features corresponding to the target acoustic features. Figure 2 The figure shows the topological relationship between the ASR Adapter, the speaker SSM and the encoder part of the ASR model. The speaker SSM and ASRAdapter modules are added after each encoding block in the encoder. The output of the encoding block will be sent to the speaker SSM and ASR Adapter modules respectively, and then the outputs of the speaker SSM and ASR Adapter modules will be spliced ​​and input into the next encoding block. The final output target encoding sequence will be a hidden state feature sequence that integrates the speaker state features.

[0081] In this embodiment, the target acoustic feature sequence of the extracted target speech signal can be input into the ASR encoder. The ASR encoder can include N coding blocks, and a speaker SSM and an ASR Adapter module are added after each coding block, where N is an integer greater than one. The coding sequence generated by each coding block can be sent to the speaker SSM and the ASR Adapter module respectively. The speaker SSM can encode the speaker state, and the ASR Adapter module can fine-tune the input parameters. The outputs of the speaker SSM and the ASR Adapter module are then spliced ​​to obtain a spliced ​​sequence. The speaker state can be embedded in the acoustic coding features of the streaming multi-speaker ASR, and then the obtained spliced ​​sequence is provided as a new input to the next coding block. The fusion of the speaker state and the target acoustic features is achieved.

[0082] Without loss of generality, we take S4 (Structured State Space Sequence Model) as an example of SSM for speaker state encoding. The S4 model is an advanced structured state space model for sequence modeling, which can efficiently process long sequence data. In the S4 model, the state vector h t There is the following relationship between the input feature sequence xt and the output feature sequence yt.

[0083] ht=Ah t-1 +Bx t

[0084] y t =Ch t

[0085] Where A, B, and C are learnable parameter matrices.

[0086] After discretization using the ZOH (Zero Order Holding) method, the discretized features have the following relationship.

[0087]

[0088] y t =C T h t

[0089] Where Δ is the learnable step size, and is the discrete parameter matrix and I is the identity matrix.

[0090]

[0091] Exemplarily, the first coding sequence x generated by the first coding block may be t , sent to the speaker state model Speaker SSM for encoding to obtain the first state sequence h t ,h t The speaker state features obtained by inputting a layer of encoder blocks to the Speaker SSM.

[0092]

[0093] It should be noted that the above S4 model is only a function It can also be implemented through more advanced SSM models such as Mamba, and this application does not limit this.

[0094] The first coding sequence generated by the first coding block is input into the ASR Adapter module to generate the first fine-tuning sequence. The first state sequence and the first fine-tuning sequence are fused by splicing to obtain the first splicing sequence, and then the first splicing sequence is provided as a new input to the second coding block. The above steps are repeated until the N coding blocks are stacked in the ASR adapter to obtain the Nth splicing sequence, where N is an integer greater than 1. The Nth splicing sequence is used as the target coding sequence e t , which is the output of the ASR encoder. Figure 2 The ASR encoder structure shown in the figure can make full use of the existing speaker-unaware ASR system through the above steps. It only needs a certain number of steps of fine-tuning training to enable the existing ASR system to have multi-speaker decoding capabilities, and will not cause the original ASR performance to degrade due to the disaster of forgetting. At the same time, the context memory ability of the speaker state space model can improve the accuracy of speaker time boundary judgment.

[0095] Reference Figure 3 , the present application provides an abstract schematic diagram of the encoding and decoding strategy of another embodiment of the speech recognition method. Optionally, in an embodiment of the present application, generating a masked text corresponding to the target speech signal according to the target encoding sequence may include:

[0096] Predicting a predicted text corresponding to the target speech signal according to the target coding sequence;

[0097] The predicted text is converted into a masked text corresponding to the target speech signal.

[0098] In this embodiment, the target coding sequence e output by the ASR encoder can be t Input to the text decoder Prediction, predict the text sequence Y corresponding to the target speech signal, Y = Y 1， Y 2， …,,Y t , where t is the number of characters output corresponding to the predicted text. Then the predicted character sequence Y is output as the predicted text. The predicted text can be masked to convert the predicted text into masked text.

[0099] Optionally, in an embodiment of the present application, converting the predicted text into a masked text corresponding to the target speech signal may include:

[0100] Converting the target state sequence into a mask matrix; the mask matrix is ​​a mask matrix with the predicted number of one or more speakers as the vertical axis and the target acoustic feature length as the horizontal axis;

[0101] The predicted text is masked using the mask matrix to obtain the masked text.

[0102] In this embodiment, if Figure 3 As shown, Figure 3 An abstract representation of a mask decoding strategy of the present application is given, which includes an encoder, a text decoder, a speaker predictor, and a joint decoder. Figure 3 The encoder is an abstract representation of the ASR encoder that incorporates the speaker state encoding. The target feature sequence x t Input the encoder and input the target encoding sequence output by the encoder into the speaker prediction module (Speaker Prediction). The Speaker Prediction module can convert the sequence e t Transformed into a mask matrix of shape S*T, where T represents the entire acoustic feature length, T=1,2,…,t. S is the predicted number of global speakers, that is, the mask matrix corresponding to the target state sequence is a mask matrix with the number of speakers contained in the predicted target speech signal as the vertical axis and the target acoustic feature length as the horizontal axis. The mask matrix obtained by the Speaker Prediction module is used to mask the predicted text obtained by the text decoder Prediction to obtain the corresponding masked text. Exemplarily, a 01 mask matrix can be selected to mask the predicted text. When the predicted number of global speakers is increased, the number of rows of the corresponding generated mask matrix will also change. For example, if the originally predicted number of global speakers is S, the generated mask matrix shape is S*T. When the predicted number of global speakers is increased by 1, the generated mask matrix will also be updated to (S+1)*T accordingly.

[0103] In an embodiment of the present application, generating speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text may include:

[0104] The target state sequence and the masked text are jointly decoded to obtain speech recognition content corresponding to one or more speakers.

[0105] In this embodiment, the joint network can be used as a joint decoder to jointly decode the obtained target state sequence and the masked text. The Joint Network can learn two or more tasks at the same time, so that the representations of these tasks can be shared, thereby improving learning efficiency and performance. By masking the predicted text with a mask matrix, the decoding process of the Joint Network can be controlled to the same speaker, such as Figure 3 As shown in Figure 1, the final decoding result can output the target speech signal containing one or more speakers as speech recognition content corresponding to different speakers. For example, the Joint Network finally outputs the predicted speaker s i,{s i |s1,s2,…,s t-1 ,s t} and speaker s i The corresponding speech recognition result Y t , speaker i The corresponding speech recognition result is a recognition text sequence obtained by separating the target speech signal according to the speaker and then converting it. Through the above steps, the semantic confusion problem caused by predicting the speaker conversion can be avoided.

[0106] Take the streaming RNN-T multi-speaker ASR system trained by t-SOT as an example. Among them, SOT (Serialized Output Training) can be trained to transcribe the speech of multiple speakers into a sequence text to solve the problem of overlapping speech. Based on SOT, t-SOT can correspond each word to the correct speaker to improve recognition accuracy. The recurrent neural network RNN-T (Recurrent Neural Network-Transducer) is a model for speech recognition that combines recursive neural networks and sequence-to-sequence mapping. It can predict and align speech input and text output in real time, and is suitable for streaming ASR tasks.

[0107] refer to Figure 4 A flow chart of the steps of an embodiment of a speech recognition method is shown. A target speech signal containing multiple speakers is input into the streaming RNN-T multi-speaker ASR system, the system preprocesses the target speech signal, and then extracts acoustic features from the preprocessed target speech signal to obtain a target acoustic feature sequence x t . The target acoustic feature sequence x t Input encoder Encoder to get the target encoding sequence w that incorporates the speaker state t The encoder contains N encoder blocks that integrate the speaker state. t The target speech signal is sent to the speaker prediction module. Speaker Prediction predicts the number of global speakers S contained in the target speech signal, and generates a 01 mask matrix of shape S*T according to the length T of the target acoustic feature sequence. The text decoder Prediction in the system outputs the predicted text corresponding to the target speech signal. The mask matrix obtained by Speaker Prediction is combined with the first t-1 text output sequence Y encoded by Prediction. t-1 After the mask operation, the Joint Network decodes it jointly. When predicting the tth word, Y1 to Yt-1 The historical prediction text information and target coding sequence e1 to e t . Target coding sequence e1 to e t and the target acoustic features x1 to x in the input target speech signal t Corresponding.

[0108] As shown in the following table, the target acoustic feature sequence length is T, T = 1, 2, ..., 12. The predicted text is "Today's weather <sc>Hello <sc>How is it? <sc>To predict the speaker switch marker, it means that the speaker identity has switched, and the text before and after the marker corresponds to different speakers. When the number of predicted global speakers is 2, Speaker Prediction generates a 2*12 01 mask matrix to mask the predicted text. In the table, Masked Speaker 1 and Masked Speaker 2 are the 2-speaker mask matrices output by the Speaker Prediction module, and 1 represents the person who is speaking. If the current time t = 10, the text decoded at the previous time is obtained <sc>, which means that the speaker has switched at the previous moment. According to the mask matrix obtained by the speaker predictor, speaker 1 should be speaking at the moment, so the predicted text "Hello" of speaker 2 is masked. At this time, only the predicted text "Today's weather" corresponding to speaker 1 is used as the historical recognition result of the current speaker 1. When t=12, according to the above mask decoding strategy, the predicted two speakers' voices in the input voice signal can be separated and converted into corresponding texts, and the speech recognition content of speaker 1 "How is today's weather" and the speech recognition content of speaker 2 "Hello" are obtained respectively.

[0109] Time t 1 2 3 4 5 6 7 8 9 10 11 12 Predictive text now sky of sky gas <sc> < / sc> you good <sc> < / sc> How What Sample Masked Speaker 1 1 1 1 1 1 - 0 0 - 1 1 1 Masked Speaker 2 0 0 0 0 0 - 1 1 - 0 0 0

[0110] Through the above masking operation, the decoding process of the Joint Network can be controlled to the same speaker, the context of the same speaker can be connected as much as possible without being interrupted by the speaker switch marker, different speakers can be separated and the corresponding recognition text can be output, avoiding the problem of predicting speaker switch markers. <sc>The semantic confusion brought about by this further improves the speech recognition performance.

[0111] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0112] On the basis of the above embodiments, this embodiment further provides a speech recognition device, which is applied to electronic devices such as terminal devices and servers.

[0113] Reference Figure 5 , shows a structural block diagram of an embodiment of a speech recognition device of the present application, which may specifically include the following modules:

[0114] The receiving module 501 is used to receive a target speech signal, wherein the target speech signal includes speech signals of one or more speakers;

[0115] An acquisition module 502 is used to acquire a target acoustic feature sequence of the target speech signal;

[0116] A target coding sequence generating module 503 is used to generate a target coding sequence according to the target acoustic feature sequence;

[0117] A masked text generation module 504, configured to generate a masked text corresponding to the target speech signal according to the target coding sequence;

[0118] The recognition module 505 is used to generate speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text.

[0119] Optionally, the acquisition module includes:

[0120] A frequency spectrum feature acquisition submodule, used to acquire the frequency spectrum features of the target speech signal;

[0121] The acoustic feature extraction submodule is used to extract the target acoustic feature sequence of the target speech signal from the frequency spectrum feature.

[0122] Optionally, the target state sequence generation module is further used to:

[0123] The target acoustic feature sequence is encoded to generate the target encoding sequence.

[0124] Optionally, the target coding sequence generation module includes:

[0125] Inputting the target acoustic feature sequence into a first encoding block of an encoder to generate a first encoding sequence;

[0126] A state sequence generating submodule, used for generating a first state sequence according to the first coding sequence;

[0127] A fine-tuning sequence generating submodule, configured to generate a first fine-tuning sequence according to the first encoding sequence;

[0128] A splicing submodule, used for splicing the first state sequence and the first fine-tuning sequence to obtain a first spliced ​​sequence;

[0129] A loop submodule is used to send the first spliced ​​sequence to the next coding block until an Nth spliced ​​sequence is generated, and the Nth spliced ​​sequence is used as the target coding sequence; wherein N is the number of coding blocks stacked in the encoder, and N is an integer greater than 1.

[0130] Optionally, the masked text generation module includes:

[0131] A predicted text generation submodule, used for predicting a predicted text corresponding to the target speech signal according to the target coding sequence;

[0132] The masked text generation submodule is used to convert the predicted text into the masked text corresponding to the target speech signal.

[0133] Optionally, the masked text generation submodule includes:

[0134] A mask matrix generating unit, configured to convert the target state sequence into a mask matrix; the mask matrix is ​​a mask matrix with the number of predicted speakers or speakers as the vertical axis and the target acoustic feature length as the horizontal axis;

[0135] A masking unit is used to mask the predicted text using the mask matrix to obtain the masked text.

[0136] Optionally, the identification module includes:

[0137] The joint decoding submodule is used to jointly decode the target state sequence and the masked text to obtain speech recognition content corresponding to one or more speakers.

[0138] A speech recognition device provided by an embodiment of the present application can receive a target speech signal to be recognized when performing speech recognition, and the target speech signal includes speech signals of one or more speakers. A target acoustic feature sequence of the target speech signal is obtained, a target coding sequence is generated according to the target acoustic feature sequence, and the speaker state is embedded in the acoustic coding features of the streaming multi-speaker ASR. A masked text corresponding to the target speech signal is generated according to the target coding sequence; and speech recognition content corresponding to one or more speakers is generated according to the target coding sequence and the masked text. Through the mask decoding strategy, the context of the same speaker is connected as much as possible without being interrupted by speaker transition marks, thereby effectively reducing the problems of context fragmentation and semantic confusion inherent in multi-speaker automatic speech recognition systems, and significantly improving the performance of automatic speech recognition.

[0139] An embodiment of the present application also provides a speech recognition device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, one or more speech recognition methods in the above embodiments are implemented.

[0140] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0141] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0142] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0143] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0144] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable speech recognition terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable speech recognition terminal device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable speech recognition terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable speech recognition terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0147] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0148] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0149] The above is a detailed introduction to a speech recognition method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.< / sc> < / sc> < / sc> < / sc> < / sc>

Claims

1. A speech recognition method, characterized in that: The method comprises: receiving a target speech signal, wherein the target speech signal includes speech signals of one or more speakers; Acquiring a target acoustic feature sequence of the target speech signal; generating a target coding sequence according to the target acoustic feature sequence; Generating a masked text corresponding to the target speech signal according to the target coding sequence; Speech recognition content corresponding to one or more speakers is generated according to the target coding sequence and the masking text.

2. The method according to claim 1, characterized in that The step of acquiring a target acoustic feature sequence of the target speech signal comprises: Acquire frequency spectrum characteristics of the target speech signal; A target acoustic feature sequence of the target speech signal is extracted from the frequency spectrum features.

3. The method according to claim 1, characterized in that Generating a target coding sequence according to the target acoustic feature sequence includes: The target acoustic feature sequence is encoded to generate the target encoding sequence.

4. The method according to claim 3, characterized in that The step of encoding the target acoustic feature sequence to generate the target encoding sequence includes: Inputting the target acoustic feature sequence into a first encoding block of an encoder to generate a first encoding sequence; generating a first state sequence according to the first coding sequence; generating a first fine-tuning sequence according to the first encoding sequence; splicing the first state sequence and the first fine-tuning sequence to obtain a first spliced ​​sequence; The first concatenated sequence is fed into the next coding block until an Nth concatenated sequence is generated, and the Nth concatenated sequence is used as the target coding sequence; wherein N is the number of coding blocks stacked in the encoder, and N is an integer greater than 1.

5. The method according to claim 1, characterized in that The step of generating a masked text corresponding to the target speech signal according to the target coding sequence comprises: Predicting a predicted text corresponding to the target speech signal according to the target coding sequence; The predicted text is converted into a masked text corresponding to the target speech signal.

6. The method according to claim 5, characterized in that The converting the predicted text into a masked text corresponding to the target speech signal comprises: Converting the target state sequence into a mask matrix; the mask matrix is ​​a mask matrix with the number of predicted one or more speakers as the vertical axis and the length of the target acoustic feature as the horizontal axis; The predicted text is masked using the mask matrix to obtain the masked text.

7. The method according to claim 1, characterized in that The generating speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text comprises: The target state sequence and the masked text are jointly decoded to obtain speech recognition content corresponding to one or more speakers.

8. A speech recognition device, characterized in that: The device comprises: A receiving module, configured to receive a target speech signal, wherein the target speech signal includes speech signals of one or more speakers; An acquisition module, used for acquiring a target acoustic feature sequence of the target speech signal; A target coding sequence generation module, used to generate a target coding sequence according to the target acoustic feature sequence; A masked text generation module, used to generate a masked text corresponding to the target speech signal according to the target coding sequence; A recognition module is used to generate speech recognition content corresponding to one or more speakers according to the target coding sequence and the masking text.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the speech recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech recognition method according to any one of claims 1 to 7 is implemented.

Images