Method of streaming speech recognition, method and apparatus for training a speech recognition model

Abstract

Embodiments of the present application disclose a streaming speech recognition method, a method and device for training a speech recognition model. The method comprises: obtaining a speech audio stream; inputting continuous first audio blocks obtained by dividing the speech audio stream by a first time unit into a first speech recognition model to obtain recognition results of the first audio blocks for display; obtaining hidden vectors of each frame obtained by encoding the speech audio stream, and predicting a first sequence corresponding to the speech audio stream by using the hidden vectors, the first sequence comprising weight values of each frame in the speech audio stream; dividing the speech audio stream by using the first sequence to obtain continuous second audio blocks; inputting the continuous second audio blocks into a second speech recognition model to obtain recognition results of the second audio blocks, and updating the displayed recognition results of the corresponding first audio blocks by using the recognition results of the second audio blocks. The present application improves the display effect of real-time speech recognition and enhances user experience.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a method for streaming speech recognition, a method for training a speech recognition model, and an apparatus. Background Technology

[0002] Streaming ASR (Automatic Speech Recognition), also known as real-time speech recognition, is a technology that converts audio into text in real time, achieving the effect of "text appearing on the screen as you speak." Currently, real-time speech recognition products mainly fall into two technical categories:

[0003] The first approach is to use a low-latency speech recognition model to output the speech recognition result. However, in order to ensure a low-latency on-screen experience, low-latency speech recognition models encode and decode audio blocks of relatively short duration as a unit, thus obtaining recognition results for shorter audio blocks. This shorter audio block duration means that the low-latency speech recognition model relies on less contextual information during encoding and decoding, inevitably leading to lower recognition accuracy.

[0004] The second approach combines low-latency and high-latency speech recognition models. The low-latency model outputs the recognition results for shorter audio segments in real time, while the high-latency model performs speech recognition on longer audio segments. The results from these longer segments are then used to update the low-latency model's earlier findings. This approach ensures real-time display of results while allowing for more accurate updates after a certain time. However, a key challenge in this approach is ensuring proper audio segmentation so that the high-latency model can update the displayed results appropriately, thus guaranteeing a positive user experience. Summary of the Invention

[0005] In view of this, this application provides a method for streaming speech recognition, a method and apparatus for training a speech recognition model, in order to improve the display effect of speech recognition results and enhance user experience.

[0006] This application provides the following solution:

[0007] In a first aspect, a method for streaming speech recognition is provided, the method comprising:

[0008] Get the audio stream;

[0009] The continuous first audio blocks obtained by segmenting the speech audio stream according to the first duration are input into the first speech recognition model to obtain the recognition results of each first audio block for display.

[0010] Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame.

[0011] The first sequence is used to segment the speech audio stream to obtain continuous second audio blocks;

[0012] The continuous second audio blocks are input into the second speech recognition model to obtain the recognition results of each second audio block. The recognition results of the corresponding first audio block that has been displayed are then updated using the recognition results of each second audio block.

[0013] According to one achievable method in the embodiments of this application, the operation of encoding the speech audio stream is performed by the encoding module in the first speech recognition model, and the operation of predicting the first sequence corresponding to the speech audio stream using the latent vector is performed by the prediction module in the first speech recognition model.

[0014] According to one achievable method in an embodiment of this application, segmenting the speech audio stream using the first sequence to obtain consecutive second audio blocks includes:

[0015] The first sequence is convolved using a full-1 convolution kernel to obtain the second sequence.

[0016] In the second sequence, the speech audio stream is segmented at frame positions where the value is less than or equal to a preset first threshold to obtain continuous second audio blocks.

[0017] According to one achievable method in an embodiment of this application, before performing convolution processing on the first sequence, the method further includes: performing smoothing processing on the first sequence to set the weight values ​​in the first sequence that are less than or equal to a preset second threshold to 0.

[0018] The preset first threshold is 0.

[0019] According to one achievable method in an embodiment of this application, segmenting the speech audio stream at frame positions in the second sequence corresponding to values ​​less than or equal to a preset first threshold includes:

[0020] If there are multiple consecutive frames in the second sequence whose values ​​are less than or equal to a preset first threshold, then the speech audio stream is segmented at one of the multiple consecutive frames.

[0021] According to one achievable method in the embodiments of this application, if no frame with a value less than or equal to a preset first threshold appears in the second sequence for more than a preset second duration starting from the latest segmentation position, the speech audio stream is segmented from a position of preset second duration away from the latest segmentation position;

[0022] The second duration is longer than the first duration.

[0023] According to one achievable method in an embodiment of this application, inputting the continuous second audio blocks into the second speech recognition model includes:

[0024] The acoustic features of the continuous second audio blocks and the latent vector representation output by the encoding module in the first speech recognition model are concatenated, and the concatenated feature representation is downsampled and then input into the second speech recognition model.

[0025] Secondly, a method for streaming speech recognition is provided, the method comprising:

[0026] Acquire the audio stream generated during a real-time meeting;

[0027] The first audio blocks obtained by segmenting the audio stream according to the first duration are input into the first speech recognition model to obtain the recognition results of each first audio block, which are then displayed on the user terminal of the real-time conference.

[0028] Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame.

[0029] The first sequence is used to segment the speech audio stream to obtain continuous second audio blocks;

[0030] The continuous second audio blocks are input into the second speech recognition model to obtain the recognition results of each second audio block. The recognition results of each second audio block are then used to update the recognition results of the corresponding first audio block that have been displayed on the user terminal of the real-time conference.

[0031] Thirdly, a method for streaming speech recognition is provided, executed by a cloud server, the method comprising:

[0032] Acquire the voice audio stream from the terminal device;

[0033] The first audio block obtained by segmenting the speech audio stream according to the first duration is input into the first speech recognition model to obtain the recognition result of each first audio block, and the recognition result of each first audio block is sent to the terminal device for display.

[0034] Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame.

[0035] The first sequence is used to segment the speech audio stream to obtain continuous second audio blocks;

[0036] The continuous second audio blocks are input into the second speech recognition model to obtain the recognition results of each second audio block. The recognition results of the corresponding first audio block already displayed by the terminal device are then updated using the recognition results of each second audio block.

[0037] Fourthly, a method for training a speech recognition model is provided, the method comprising:

[0038] Acquire training data containing multiple training samples, wherein the training samples include speech audio samples and the recognition result labels corresponding to the speech audio samples;

[0039] The training data is used to train a second speech recognition model. The training includes: obtaining latent vectors for each frame of the encoded speech audio sample; using the latent vectors to predict a first sequence corresponding to the speech audio sample, the first sequence containing weight values ​​for each frame in the speech audio sample, the weight values ​​representing the amount of acoustic information contained in the corresponding frame; using the first sequence to segment the speech audio stream to obtain continuous second audio blocks; inputting the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each audio block obtained by the second speech recognition model; the training objective includes minimizing the difference between the recognition results obtained by the second speech recognition model for the speech audio sample and the corresponding recognition result labels.

[0040] Fifthly, a streaming speech recognition apparatus is provided, the apparatus comprising:

[0041] The audio stream acquisition unit is configured to acquire the speech audio stream;

[0042] The first recognition unit is configured to input continuous first audio blocks obtained by segmenting the speech audio stream according to a first duration into the first speech recognition model, and to obtain the recognition results of each first audio block for display.

[0043] The speech segmentation unit is configured to acquire latent vectors of each frame obtained by encoding the speech audio stream, predict a first sequence corresponding to the speech audio stream using the latent vectors, the first sequence containing weight values ​​of each frame in the speech audio stream, the weight values ​​being used to characterize the amount of acoustic information contained in the corresponding frame; and segment the speech audio stream using the first sequence to obtain continuous second audio blocks.

[0044] The second recognition unit is configured to input the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each second audio block;

[0045] The result update unit is configured to update the recognition result of the corresponding first audio block that has been displayed using the recognition result of each second audio block.

[0046] Sixthly, an apparatus for training a speech recognition model is provided, the apparatus comprising:

[0047] The sample acquisition unit is configured to acquire training data containing multiple training samples, wherein the training samples include speech audio samples and recognition result labels corresponding to the speech audio samples;

[0048] The model training unit is configured to train a second speech recognition model using the training data. The training includes: acquiring latent vectors for each frame encoded from the speech audio samples; predicting a first sequence corresponding to the speech audio samples using the latent vectors, the first sequence containing weight values ​​for each frame in the speech audio samples, the weight values ​​representing the amount of acoustic information contained in the corresponding frame; segmenting the speech audio stream using the first sequence to obtain continuous second audio blocks; inputting the continuous second audio blocks into the second speech recognition model to obtain recognition results for each audio block obtained by the second speech recognition model; the training objective includes minimizing the difference between the recognition results obtained by the second speech recognition model for the speech audio samples and the corresponding recognition result labels.

[0049] According to a seventh aspect, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any one of the first to fourth aspects.

[0050] According to the eighth aspect, an electronic device is provided, comprising:

[0051] One or more processors; and

[0052] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first to fourth aspects.

[0053] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0054] 1) In this application, the first speech recognition model performs speech recognition on audio blocks of shorter duration units to ensure the real-time display of recognition results. The second speech recognition model no longer updates the first model's recognition result using the entire sentence recognition result after detecting the end of the sentence. Instead, it predicts the first sequence using the latent vectors of each frame obtained from the speech audio stream encoding, then segments the speech audio stream using the weight values ​​representing the acoustic information content of the corresponding frames in the first sequence, obtaining continuous second audio blocks. The second speech recognition model then performs speech recognition on each second audio block and updates the displayed recognition result of the corresponding first audio block. This method of segmenting the speech audio stream based on acoustic information content ensures that the segmentation position is not necessarily at the end of the sentence, thereby improving the display effect of real-time speech recognition and enhancing the user experience.

[0055] 2) The operations of encoding the speech audio stream and predicting the first sequence in this application can be performed by the encoding module and prediction module in the first speech recognition model. That is, the results from the encoding and prediction modules in the first speech recognition model can be directly reused, minimizing the amount of additional parameters and processing time, thereby improving the efficiency of speech recognition. Furthermore, compared to traditional silence detection methods such as VAD (VoiceActiveDetection), the encoding and prediction capabilities of the first speech recognition model are more powerful and more conducive to the accuracy of speech recognition.

[0056] 3) In this application, the first sequence is convolved to obtain the second sequence, and the speech audio stream is segmented at the frame position in the second sequence where the value is less than or equal to the preset first threshold, so that the segmentation position is in the silent segment. Compared with the traditional fixed duration segmentation method, this method can segment at short pauses in the middle of the sentence, instead of forcibly segmenting in the middle of the pronunciation, thus ensuring the complete semantics within the updated paragraph as much as possible, and is more in line with the user's habits.

[0057] 4) In this application, if no frame with a value less than or equal to a preset first threshold appears in the second sequence after a preset second duration starting from the latest segmentation position, the voice audio stream is forcibly segmented from a position of preset second duration away from the latest segmentation position, thereby ensuring that there are no excessively long second audio blocks and ensuring user experience.

[0058] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is an exemplary system architecture diagram that can be applied to embodiments of this application;

[0061] Figure 2 A flowchart of a streaming speech recognition method provided in an embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the structure of the first speech recognition model provided in an embodiment of this application;

[0063] Figure 4 A schematic diagram illustrating the segmentation of a speech audio stream using a second sequence, provided as an embodiment of this application;

[0064] Figure 5 A structural schematic diagram illustrating the combination of a first speech recognition model and a second speech recognition model provided in the embodiments of this application;

[0065] Figure 6 A schematic diagram illustrating the updating of displayed recognition results using the recognition results of the second audio block, as provided in an embodiment of this application;

[0066] Figure 7 A flowchart illustrating the method for training a second speech recognition model provided in this application embodiment;

[0067] Figure 8 A schematic block diagram of a streaming speech recognition device provided in an embodiment of this application;

[0068] Figure 9 A schematic block diagram of an apparatus for training a speech recognition model provided in an embodiment of this application;

[0069] Figure 10 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0071] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0072] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0073] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0074] In existing methods combining low-latency and high-latency speech recognition models, some employ Voice-Active Detection (VAD) to detect the end of sentences in the audio. The high-latency language model continuously encodes and decodes until the end of the sentence is detected, then updates the low-latency speech recognition model's results using the entire sentence recognition result. This whole-sentence update method leads to the modification of recognition content from a long time ago (a user's sentence typically lasts for tens of seconds or even longer), especially when the sentence is very long or the speaker habitually pauses very briefly. This results in recognition results not being updated for a long time, leading to inaccurate low-latency speech recognition model results being displayed for an extended period. However, directly removing the sentence-end update logic causes a sharp drop in the accuracy of the displayed recognition results.

[0075] Some existing technologies use fixed-duration segmentation of the speech audio stream. Shorter segments are used to provide the audio stream to a low-latency speech recognition model, while longer segments are used to provide it to a high-latency speech recognition model. However, this approach results in a fixed refresh time, and the segmentation position is often at a specific word. This causes the audio blocks before and after the segmentation position to have incomplete semantics, thus compromising the accuracy of the refreshed recognition results.

[0076] In view of this, this application proposes a novel approach, namely a new audio segmentation method, to enable high-latency speech recognition models to update their recognition results. To facilitate understanding of this application, the system architecture to which this application is applied is first briefly described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture includes terminal devices and streaming speech recognition devices located on the service side.

[0077] Users can interact with the streaming speech recognition device on the server side via a network using their terminal devices. The user's input audio stream is sent by the terminal device to the streaming speech recognition device on the server side. The streaming speech recognition device performs speech recognition on the audio stream based on a first speech recognition model and a second speech recognition model, and returns the recognition result, which is then displayed to the user in real time by the terminal device.

[0078] Terminal devices are various devices that support voice interaction, including but not limited to smart mobile terminals, smart home devices, wearable devices, and PCs (personal computers). Smart mobile devices can include devices such as mobile phones, tablets, laptops, PDAs (personal digital assistants), and connected cars. Smart home devices can include devices such as smart TVs and smart refrigerators. Wearable devices can include devices such as smartwatches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual and augmented reality).

[0079] Networks can include various connection types, such as wired and wireless communication links or fiber optic cables, etc.

[0080] Streaming speech recognition devices can be configured as standalone servers, within the same server or server cluster, or on a separate or shared cloud server. Cloud servers, also known as cloud computing servers or cloud hosts, are a host product within the cloud computing service system, designed to address the management difficulties and weak service scalability inherent in traditional physical hosts and Virtual Private Servers (VPs). Furthermore, streaming speech recognition devices can, in addition to... Figure 1The settings shown can be configured not only on the server side, but also on terminal devices with strong computing capabilities.

[0081] It should be understood that Figure 1 The number of streaming speech recognition devices, first speech recognition models, second speech recognition models, and terminal devices shown in the diagram is merely illustrative. Depending on the implementation requirements, any number of streaming speech recognition devices, first speech recognition models, second speech recognition models, and terminal devices can be included.

[0082] It should be noted that the terms "first" and "second" used in this disclosure do not have any limitations on size, order, or quantity; they are merely used to distinguish between the two. For example, "first speech recognition model" and "second speech recognition model" are used to distinguish between two speech recognition models in terms of name, and "first duration" and "second duration" are used to distinguish between two durations in terms of name, and so on.

[0083] Figure 2 This is a flowchart of a streaming speech recognition method provided in an embodiment of this application. The process can be performed by… Figure 1 The speech recognition device in the system architecture shown performs the operation. For example... Figure 2 As shown, the method may include the following steps:

[0084] Step 202: Obtain the audio stream.

[0085] Step 204: Input the continuous first audio blocks obtained by segmenting the speech audio stream according to the first duration into the first speech recognition model to obtain the recognition results of each first audio block for display.

[0086] Step 206: Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame.

[0087] Step 208: Use the first sequence to segment the speech audio stream to obtain continuous second audio blocks.

[0088] Step 210: Input the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each second audio block, and use the recognition results of each second audio block to update the recognition results of the corresponding first audio block that has been displayed.

[0089] As can be seen from the above process, in this application, the first speech recognition model performs speech recognition on audio blocks of shorter duration units to ensure the real-time display of recognition results. The second speech recognition model no longer updates the recognition result of the first model using the recognition result of the entire sentence after detecting the end of the sentence. Instead, it uses the latent vectors of each frame obtained from the speech audio stream encoding to predict the first sequence, then uses the weight values ​​representing the acoustic information content of the corresponding frames in the first sequence to segment the speech audio stream, obtaining continuous second audio blocks. The second speech recognition model then performs speech recognition on each second audio block and updates the recognition result of the corresponding first audio block that has already been displayed. This method of segmenting the speech audio stream based on acoustic information content ensures that the segmentation position is not necessarily at the end of the sentence, thereby improving the display effect of real-time speech recognition and enhancing the user experience.

[0090] The following is a detailed description of each step in the above process. The audio stream acquired in step 202 can be generated by the user in real time. For example... Figure 1 As shown, the user inputs voice in real time through a terminal device, and the terminal device sends the voice audio stream generated by the user's input to the voice recognition device, which then acquires the voice audio stream in real time for voice recognition.

[0091] Furthermore, the methods for generating audio streams differ depending on the application scenario. For example, in a live video streaming scenario, the audio stream generated by the live video stream is acquired. In an online meeting scenario, the audio stream generated by the real-time meeting is acquired. In a voice input scenario, the audio stream input by the user in the human-computer interaction system or in an input method application can be acquired.

[0092] The following describes in detail step 204, namely, "inputting the continuous first audio blocks obtained by segmenting the speech audio stream according to the first duration into the first speech recognition model, and obtaining the recognition results of each first audio block for display," with reference to the embodiments.

[0093] The first speech recognition model in this embodiment is a low-latency model. For the acquired speech audio stream, starting from the beginning of the speech audio stream, the speech audio stream is segmented according to a first duration, that is, the speech audio stream is segmented into continuous first audio blocks, each with a duration of the first duration. As a low-latency model, the first duration used for segmentation in this first speech recognition model is relatively short, typically on the order of milliseconds. The specific duration can be selected according to the actual needs of the application scenario, such as 100 milliseconds, 200ms, 300ms, etc.

[0094] Each time an audio segment is generated, the first speech recognition model immediately performs speech recognition processing on the newly generated segment, and the result is displayed immediately. For example, it is immediately returned to the terminal device for display. In other words, the first duration unit used by the first speech recognition model reflects the latency of streaming speech recognition. The user's experience is that the recognition result displayed on the terminal device is one duration unit later than the speech, rather than waiting until the speech is finished before performing speech recognition; it's equivalent to "displaying the recognition result while speaking."

[0095] As one possible implementation method, the structure of the first speech recognition model used in the embodiments of this application may include a first encoding module and a first decoding module.

[0096] After segmenting the audio into first audio blocks, the acoustic features extracted from the first audio blocks are input into the first encoding module for encoding to obtain a vector representation of the first audio block. During encoding, the first encoding module can combine the acoustic features of previous first audio blocks (which need to be continuous with the current first audio block) to encode the acoustic features of the current first audio block. These acoustic features may include, but are not limited to, at least one of MFCC (Mel-Frequency Cepstral Coefficients), LPCC (Linear Predictive Cepstral Coefficients), average amplitude change rate, short-time average energy, and Fbank features.

[0097] The first decoding module decodes the vector representation of the first audio block and outputs the recognition result of the first audio block, i.e., the corresponding text. When decoding the current first audio block, the first decoding module combines the recognition result of the previous first audio block (which needs to be continuous with the current first audio block) to decode the vector representation of the current first audio block, thus obtaining the recognition result of the current first audio block. This recognition result is actually a text sequence composed of consecutive elements (Tokens). These elements can be text, characters, or words, etc.

[0098] As another possible approach, the structure of the first speech recognition model used in this embodiment can be as follows: Figure 3 As shown, it includes a second encoding module, a prediction module, and a second decoding module.

[0099] In this process, after segmenting the audio into a first audio block, the first audio block is input into a second encoding module. The second encoding module encodes the first audio block to obtain an implicit vector representation H.

[0100] The prediction module uses the latent vector representation H to predict the number of tokens N' in the recognition result corresponding to the first audio block and generates an acoustic feature representation E.

[0101] The second decoding module uses latent vector representation H and acoustic feature representation E to perform decoding processing, obtaining the recognition result of the current first audio block. This recognition result is a text sequence consisting of consecutive tokens, where tokens can be text, characters, or words, etc.

[0102] The second encoding module described above can be implemented based on SAN-M (Self-Attention Network with Memory Units) and FNN (Feedforward Neural Network), or it can be implemented based on a Conformer (a Transformer structure using convolutional enhancement). The second decoding module can be implemented based on SAN-M, FNN, and cross-MHA (Multi-Head Attention).

[0103] In this process, when predicting the number of tokens N' in the recognition result corresponding to the first audio block, the prediction module first uses the latent vector representation H output by the second encoding module to predict the first sequence. This first sequence is actually a weight sequence, which is composed of the weight values ​​corresponding to each frame. The weight value corresponding to each frame represents the amount of acoustic information contained in that frame. Typically, the weight value α can take values ​​between 0 and 1, and the larger the value, the greater the amount of acoustic information contained in that frame.

[0104] The prediction module uses the CIF (Continuous Integrate-and-Fire) mechanism, which continuously accumulates the weight values ​​of each frame. When the accumulated weight values ​​reach a preset threshold, it means an acoustic boundary has been located, and the number of tokens is predicted by the accumulated weight values.

[0105] The following describes in detail step 206, namely "obtaining the latent vectors of each frame obtained by encoding the speech audio stream, and using the latent vectors to predict the first sequence corresponding to the speech audio stream", with reference to the embodiments.

[0106] In this step, the speech audio stream can first be encoded by an encoding module to obtain the latent vector representation of each frame. The encoding module can be implemented based on SAN-M or based on the conformer structure.

[0107] Then, the segmentation module uses the latent vector representation of each frame obtained by the encoding module to predict the first sequence, which contains the weight values ​​of each frame in the speech audio stream. The weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame.

[0108] As one possible implementation, the aforementioned encoding module can be the second encoding module in the first speech recognition model, and the aforementioned segmentation module can be the prediction module in the first speech recognition model. That is, the first sequence obtained by the prediction module in the first speech recognition model for the speech audio stream can be used to segment the speech audio stream of the second speech recognition model.

[0109] The following describes step 208, namely "segmenting the speech audio stream using the first sequence to obtain continuous second audio blocks", in detail with reference to the embodiments.

[0110] Since the first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​represent the amount of acoustic information contained in the corresponding frame, the idea of ​​this step is to use the amount of acoustic information contained in each frame in the first sequence to determine the silence frame or silence segment, and then segment the silence frame or silence segment to obtain the second audio block.

[0111] One feasible approach is to directly utilize the first sequence and segment frames at positions where the weight value is less than or equal to a preset third threshold. This method considers frames with weight values ​​less than or equal to the preset third threshold as silent frames. The third threshold can be a very small value, such as 0.05. However, the silent frames detected by this method might be words in the middle of speech with low acoustic information due to volume or other factors. Segmenting at these words might cause the second speech recognition model to break off at a semantically important word.

[0112] In view of this, this application provides a more preferred implementation method, which can accumulate the weight values ​​of a preset number of frames before and after each frame in the first sequence, and use the accumulated value as the value of that frame in the second sequence. For example, for the i-th frame, the weight values ​​of the two frames before and the two frames after it are accumulated to obtain the value of that frame in the second sequence. Since the first two frames and the last two frames in the first sequence do not have two frames before or after them, zeros can be padded before and after the first sequence, for example, five zeros can be padded before and after each frame. The second sequence corresponding to the speech audio stream can be obtained in this way. If the value of a frame in the second sequence is less than or equal to a preset first threshold, which can be a very small value, such as 0, 0.01, 0.05, etc. Frames with values ​​less than or equal to the preset first threshold obtained in this way can be considered to be in a silent segment, and the speech audio stream can be segmented at the position of that frame to obtain continuous second audio blocks.

[0113] The process of obtaining the second sequence from the first sequence described above can also be achieved by performing a one-dimensional convolution on the first sequence using a full-1 convolution kernel. For example, using a full-1 convolution kernel of length 5 to perform a one-dimensional convolution on the first sequence is equivalent to summing the weight values ​​of the two frames before and the two frames after each frame to obtain the corresponding value of that frame in the second sequence.

[0114] In addition, if there are multiple consecutive frames with values ​​less than or equal to a preset first threshold, the speech audio stream can be segmented only at one of the frames in the multiple consecutive frames. For example, the speech audio stream can be segmented only at the first frame position in the multiple consecutive frames, instead of segmenting at every frame position.

[0115] As a more preferred implementation, the first sequence can be smoothed before this step to set the weight values ​​in the first sequence that are less than or equal to a preset second threshold to 0. The purpose of this smoothing is to eliminate glitches in the silent portions of the first sequence. If each weight value in the first sequence is represented as α, and the smoothed weight value is represented as α', then smoothing can be performed using a formula such as the following:

[0116] α'=RELU(α-A)*B

[0117] Here, A is a preset second threshold, for example, 0.05. B is a preset parameter value, such as 0.6. RELU() is the activation function. After the above smoothing process, weight values ​​less than or equal to 0.05 are reset to 0. The positions of these 0s are the positions of the silent frames. Based on this, in the second sequence, the frames with values ​​of 0 are located in the silent segments. Therefore, the speech audio stream can be segmented at the positions of the frames with values ​​of 0.

[0118] For example Figure 4 As shown, the input speech audio stream consists of speech segments and interspersed audio segments. After obtaining the second sequence (which is a process of continuous generation as the speech audio stream inputs) in the above manner, the frame positions with a value of 0 in the second sequence represent silent segments, and segmentation can be performed at these frame positions. If there are multiple consecutive 0s, segmentation can be performed only at the frame position corresponding to the first 0, that is, this frame position is used as the boundary of the second audio block. As one possible implementation method, if only one consecutive 0 appears, the distinction can be made between the position of that frame and the next frame; if multiple consecutive 0s appear, segmentation can be performed between the frame of the first 0 and the frame of the second 0.

[0119] Furthermore, if no frames with values ​​less than or equal to a preset first threshold appear in the second sequence for more than a preset second duration starting from the latest segmentation position, the speech audio stream is segmented at a position preset second duration away from the latest segmentation position; where the second duration is greater than the first duration. For example, if the duration of each first audio block obtained by the first speech recognition model is 200 milliseconds, the speech audio stream is segmented at the frame position with a value of 0 in the second sequence using the above method to obtain multiple consecutive second audio blocks. However, if no frames with a value of 0 appear for more than 6.5 seconds starting from the previous segmentation position, the speech audio stream is forcibly segmented at a distance of 6.5 seconds from the previous segmentation position, thereby ensuring that the recognition results are not not refreshed for a long time.

[0120] The following describes in detail step 210, namely, "inputting consecutive second audio blocks into the second speech recognition model to obtain the recognition results of each second audio block, and using the recognition results of each second audio block to update the recognition results of the corresponding first audio block that has been displayed," with reference to the embodiments.

[0121] The second speech recognition model in this embodiment is a high-latency model. For the acquired speech audio stream, after segmenting the speech audio stream into continuous second audio blocks using the method provided in the previous steps, the acoustic features of each second audio block are input into the second speech recognition model, and the second speech recognition model performs speech recognition on each second audio block.

[0122] Each time a second audio block is generated, the second speech recognition model immediately performs speech recognition processing on the newly generated second audio block. The recognition result is used to update the recognition result of the already displayed first audio block. In other words, the second audio block used by the second speech recognition model reflects the accuracy of streaming speech recognition.

[0123] One possible approach is to concatenate the acoustic features of consecutive second audio blocks with the vector representations output by the encoding module (i.e., the second encoding module) of the first speech recognition model. The vector representations of each frame output by the second encoding module in the first speech recognition model can assist the second speech recognition model in speech recognition, but this can also be achieved without referencing the vector representations of each frame output by the second encoding module in the first speech recognition model. The concatenated feature representation is then downsampled and input into the second speech recognition model.

[0124] As one possible way, such as Figure 5 As shown, the structure of the second speech recognition model may include a convolution module, a third encoding module, and a third decoding module.

[0125] After segmenting the audio into a second audio block, the acoustic features extracted from the second audio block are concatenated with the vector representation of the corresponding frame output by the second encoding module in the first speech recognition model. The concatenated vector representation is then input into the convolution module for downsampling. Finally, the downsampled vector representation is input into the third encoding module for encoding to obtain the vector representation of the second audio block.

[0126] The third decoding network uses the vector representation of the second audio block for decoding, outputting the recognition result of the second audio block, i.e., the corresponding text. Specifically, when decoding the current second audio block, the second decoding network combines the recognition results of previous second audio blocks (which need to be continuous with the current second audio block) to decode the vector representation of the current second audio block, obtaining the recognition result for the current second audio block. This recognition result is actually a text sequence composed of consecutive elements. These elements can be text, characters, or words.

[0127] As one possible way, such as Figure 6 As shown in the diagram. Each time the second decoding network decodes the recognition result of a second audio block, it updates the recognition result of the corresponding first audio block for display. For example, silence segments typically appear once every second in a speech segment; therefore, the duration of a second audio block is usually in the second range, while the duration of a first audio block is usually in the millisecond range, such as 200ms. Thus, one second audio block is equivalent to several first audio blocks. Each time a recognition result of a first audio block is decoded, it is displayed on the screen; and each time a recognition result of a second audio block is decoded, the recognition results of the corresponding several first audio blocks are updated using that second audio block recognition result.

[0128] In response to the above-mentioned speech recognition methods, this application provides a novel training method for the second speech recognition model. Figure 7 This is a flowchart illustrating a method for training a second speech recognition model provided in an embodiment of this application. This method can be performed by... Figure 1 The model training device in the system architecture shown is executed. For example... Figure 7 As shown, the method may include the following steps:

[0129] Step 702: Obtain training data containing multiple training samples, including speech audio samples and the corresponding recognition result labels of the speech audio samples.

[0130] In this embodiment of the application, some voice audio can be obtained first as voice audio samples, and then the voice audio samples can be labeled with recognition result tags by manual annotation. The recognition result tags are a text sequence, that is, the tokens corresponding to each frame are labeled.

[0131] Step 704: Train the second speech recognition model using training data. The training includes: obtaining the latent vectors of each frame obtained by encoding the speech audio samples; using the latent vectors to predict the first sequence corresponding to the speech audio samples; the first sequence contains the weight values ​​of each frame in the speech audio samples, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame; using the first sequence to segment the speech audio stream to obtain continuous second audio blocks; inputting the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each audio block obtained by the second speech recognition model; the training objective includes: minimizing the difference between the recognition results obtained by the second speech recognition model for the speech audio samples and the corresponding recognition result labels.

[0132] In this step, the speech audio samples can first be encoded by an encoding module to obtain the latent vector representation of each frame. The encoding module can be implemented based on SAN-M or based on the conformer structure.

[0133] Then, the segmentation module uses the latent vector representation of each frame obtained by the encoding module to predict the first sequence. The first sequence contains the weight values ​​of each frame in the speech audio sample. The weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame.

[0134] As one possible implementation, the aforementioned encoding module can be the second encoding module in the first speech recognition model, and the aforementioned segmentation module can be the prediction module in the first speech recognition model. That is, the first sequence obtained by the prediction module in the first speech recognition model for the speech audio samples can be used to segment the speech audio samples for the second speech recognition model.

[0135] One possible approach is to concatenate the acoustic features of consecutive second audio blocks with the vector representations output by the encoding module (i.e., the second encoding module) of the first speech recognition model. The vector representations of each frame output by the second encoding module in the first speech recognition model can assist the second speech recognition model in speech recognition, but this can also be achieved without referencing the vector representations of each frame output by the second encoding module in the first speech recognition model. The concatenated feature representation is then downsampled and input into the second speech recognition model.

[0136] Furthermore, since the second speech recognition model can only utilize the current second audio block and the second audio blocks preceding it during actual prediction (online speech recognition phase), when training the second speech recognition model, if consecutive second audio blocks are input, the subsequent second audio blocks are masked for the currently input second audio block. That is, for the current second audio block, the second speech recognition model can only obtain information from the current second audio block and historical second audio blocks, and cannot obtain information from future second audio blocks. The historical second audio blocks can be set to one or a preset number of multiple blocks, specifically based on experimental or empirical values.

[0137] The training objective of the second speech recognition model includes minimizing the difference between the recognition result obtained by the second speech recognition model for speech audio samples and the corresponding recognition result label. In this embodiment, a loss function can be constructed based on the above training objective. In each iteration, the parameters of the second speech recognition model are updated using the value of the loss function and methods such as gradient descent, until a preset training termination condition is met. The training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset number threshold.

[0138] The methods provided in this application embodiment can be applied to various application scenarios, such as real-time conferencing scenarios.

[0139] More and more users are now using online real-time conferencing for communication. During real-time conferencing, the content spoken by users can be transcribed in real time, similar to subtitles, and displayed on the devices of all participating users.

[0140] In the real-time transcription process, streaming speech recognition can be performed using the method provided in the embodiments of this application. Specifically, this includes: acquiring the audio stream generated by a real-time conference; inputting consecutive first audio blocks obtained by segmenting the audio stream according to a first duration into a first speech recognition model to obtain the recognition results of each first audio block for display on the user terminal of the real-time conference; simultaneously acquiring the latent vectors of each frame obtained by encoding the audio stream; using the latent vectors to predict a first sequence corresponding to the audio stream, the first sequence containing the weight values ​​of each frame in the audio stream, the weight values ​​representing the amount of acoustic information contained in the corresponding frame; then using the first sequence to segment the audio stream into consecutive second audio blocks; inputting the consecutive second audio blocks into a second speech recognition model to obtain the recognition results of each second audio block; and using the recognition results of each second audio block to update the recognition results of the corresponding first audio block already displayed on the user terminal of the real-time conference.

[0141] In this way, during a real-time conference, users can see what each speaker says on their screen. On one hand, the recognition results of each first audio block generated by the first speech recognition model ensure real-time display. On the other hand, the recognition results of each second audio block generated by the second speech recognition model are used to update the recognition results of the first audio blocks, ensuring the accuracy of the recognition results. Furthermore, the method provided in this application advances the refresh from the original method of only updating at the end of a sentence to the silent frames or silent segments within the sentence, thereby improving the display effect of the speech recognition results and enhancing the user experience.

[0142] This method is applied to real-time video live streaming subtitles. It acquires the video stream generated by the live video stream and extracts the audio stream from it. The recognition results of each first audio block generated by a first speech recognition model are displayed on the screen in real time. The recognition results of each second audio block generated by a second speech recognition model are used to update the recognition results of the first audio blocks. By utilizing the method provided in this application embodiment, the original method of refreshing only at the end of a sentence can be moved to silent frames or silent segments within the sentence, thereby improving the display effect of the speech recognition results and enhancing the user experience.

[0143] Besides real-time video subtitles, it can also be applied to scenarios such as real-time video subtitles, real-time court hearing recordings, intelligent voice assistants, and intelligent input. In the real-time video subtitle scenario, the acquired audio stream is the video stream generated by the live video broadcast. In the real-time court hearing recording scenario, the acquired audio stream is the audio stream generated by the real-time court hearing. In the intelligent voice assistant scenario, the acquired audio stream is the audio stream input by the user in the human-computer interaction system. In the intelligent input scenario, the acquired audio stream is the audio stream input by the user in the input method application. These are not listed exhaustively here.

[0144] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0145] According to another embodiment, a streaming speech recognition apparatus is provided. Figure 8 A schematic block diagram of a streaming speech recognition apparatus according to one embodiment is shown. Figure 8 As shown, the device 800 includes: an audio stream acquisition unit 801, a first recognition unit 802, a speech segmentation unit 803, a second recognition unit 804, and a result update unit 805. The main functions of each component are as follows:

[0146] The audio stream acquisition unit 801 is configured to acquire the voice audio stream.

[0147] The first recognition unit 802 is configured to input the continuous first audio blocks obtained by segmenting the speech audio stream according to the first duration into the first speech recognition model, and to obtain the recognition results of each first audio block for display.

[0148] The speech segmentation unit 803 is configured to acquire the latent vectors of each frame obtained by encoding the speech audio stream, use the latent vectors to predict the first sequence corresponding to the speech audio stream, the first sequence containing the weight values ​​of each frame in the speech audio stream, the weight values ​​being used to characterize the amount of acoustic information contained in the corresponding frame; and use the first sequence to segment the speech audio stream to obtain continuous second audio blocks.

[0149] The second recognition unit 804 is configured to input consecutive second audio blocks into the second speech recognition model to obtain the recognition results of each second audio block.

[0150] The result update unit 805 is configured to update the recognition result of the corresponding first audio block that has been displayed using the recognition result of each second audio block.

[0151] As one possible implementation, the aforementioned speech segmentation unit 803 can be the encoding module and prediction module in the first speech recognition model.

[0152] As one possible implementation method, when the speech segmentation unit 803 uses the first sequence to segment the speech audio stream to obtain continuous second audio blocks, it can be specifically configured as follows: using a full-1 convolution kernel to perform one-dimensional convolution processing on the first sequence to obtain the second sequence; segmenting the speech audio stream at the frame positions in the second sequence where the value is less than or equal to a preset first threshold to obtain continuous second audio blocks.

[0153] Furthermore, before performing convolution processing on the first sequence, the speech segmentation unit 803 can first smooth the first sequence to set the weight values ​​in the first sequence that are less than or equal to a preset second threshold to 0. Accordingly, the preset first threshold is 0.

[0154] As one possible implementation, if no frame with a value less than or equal to a preset first threshold appears in the second sequence for more than a preset second duration starting from the latest segmentation position, the speech segmentation unit 803 segments the speech audio stream from a position of preset second duration from the latest segmentation position; wherein the second duration is greater than the first duration.

[0155] As one possible approach, when the second recognition unit 804 inputs continuous second audio blocks into the second speech recognition model, it can concatenate the acoustic features of the continuous second audio blocks with the latent vector representation output by the encoding module in the first speech recognition model, and then input the concatenated feature representation into the second speech recognition model after downsampling.

[0156] According to another embodiment, an apparatus for training a speech recognition model is provided. Figure 9 A schematic block diagram of an apparatus for training a speech recognition model according to one embodiment is shown. Figure 9 As shown, the device 900 includes a sample acquisition unit 901 and a model training unit 902. The main functions of each component are as follows:

[0157] The sample acquisition unit 901 is configured to acquire training data containing multiple training samples, including speech audio samples and the recognition result labels corresponding to the speech audio samples.

[0158] The model training unit 902 is configured to train a second speech recognition model using training data. The training includes: acquiring the latent vectors of each frame obtained by encoding the speech audio samples; using the latent vectors to predict the first sequence corresponding to the speech audio samples, the first sequence containing the weight values ​​of each frame in the speech audio samples, the weight values ​​being used to characterize the amount of acoustic information contained in the corresponding frame; using the first sequence to segment the speech audio stream to obtain continuous second audio blocks; inputting the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each audio block obtained by the second speech recognition model; the training objective includes: minimizing the difference between the recognition results obtained by the second speech recognition model for the speech audio samples and the corresponding recognition result labels.

[0159] One possible approach is for the model training unit 902 to concatenate the acoustic features of consecutive second audio blocks with the vector representations output by the encoding module (i.e., the second encoding module) of the first speech recognition model. The vector representations of each frame output by the second encoding module in the first speech recognition model can assist the second speech recognition model in speech recognition, but this can also be achieved without referencing the vector representations of each frame output by the second encoding module in the first speech recognition model. The concatenated feature representation is then downsampled and input into the second speech recognition model.

[0160] Furthermore, since the second speech recognition model can only utilize the current second audio block and the second audio blocks preceding it during actual prediction (online speech recognition phase), the model training unit 902, when training the second speech recognition model, masks subsequent second audio blocks for the currently input second audio block when inputting consecutive second audio blocks. That is, for the current second audio block, the second speech recognition model can only obtain information from the current and historical second audio blocks, and cannot obtain information from future second audio blocks. The historical second audio blocks can be set to one or a preset number of blocks, specifically based on experimental or empirical values.

[0161] In this embodiment, a loss function can be constructed based on the aforementioned training objective. In each iteration, the model training unit 902 uses the value of the loss function to update the parameters of the second speech recognition model using methods such as gradient descent, until a preset training termination condition is met. This training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0162] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0164] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0165] And an electronic device, comprising:

[0166] One or more processors; and

[0167] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0168] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0169] in, Figure 10 An exemplary architecture of an electronic device is shown, which may include a processor 1010, a video display adapter 1011, a disk drive 1012, an input / output interface 1013, a network interface 1014, and a memory 1020. The processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020 can communicate with each other via a communication bus 1030.

[0170] The processor 1010 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0171] The memory 1020 can be implemented in the form of ROM (Read-Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system 1021 for controlling the operation of the electronic device 1000, and the basic input / output system (BIOS) 1022 for controlling the low-level operations of the electronic device 1000. Additionally, it can store a web browser 1023, a data storage management system 1024, and a streaming speech recognition device / model training device 1025, etc. The aforementioned streaming speech recognition device / model training device 1025 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0172] Input / output interface 1013 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0173] The network interface 1014 is used to connect the communication module (not shown in the figure) to enable communication and interaction between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0174] Bus 1030 includes a pathway for transmitting information between various components of the device (e.g., processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, and memory 1020).

[0175] It should be noted that although the above-described device only shows the processor 1010, video display adapter 1011, disk drive 1012, input / output interface 1013, network interface 1014, memory 1020, bus 1030, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0176] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0177] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for streaming speech recognition, characterized in that, The method includes: Get the audio stream; The continuous first audio blocks obtained by segmenting the speech audio stream according to the first duration are input into the first speech recognition model to obtain the recognition results of each first audio block for display. Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame. The first sequence is used to segment the speech audio stream to obtain continuous second audio blocks, wherein the continuous second audio blocks are obtained by segmenting the speech audio stream at the position of a silence frame or silence segment, the position of the silence frame or silence segment is determined based on the second sequence, which is generated based on the weight value of each frame in the speech audio stream; The continuous second audio blocks are input into the second speech recognition model to obtain the recognition results of each second audio block. The recognition results of the corresponding first audio block that has been displayed are then updated using the recognition results of each second audio block.

2. The method according to claim 1, characterized in that, The operation of encoding the speech audio stream is performed by the encoding module in the first speech recognition model, and the operation of predicting the first sequence corresponding to the speech audio stream using the latent vector is performed by the prediction module in the first speech recognition model.

3. The method according to claim 1, characterized in that, Segmenting the speech audio stream using the first sequence to obtain consecutive second audio blocks includes: The first sequence is convolved using a full-1 convolution kernel to obtain the second sequence. The speech audio stream is segmented at frame positions in the second sequence where the value is less than or equal to a preset first threshold, to obtain the continuous second audio blocks.

4. The method according to claim 3, characterized in that, Before performing convolution processing on the first sequence, the method further includes: smoothing the first sequence to set the weight values ​​in the first sequence that are less than or equal to a preset second threshold to 0. The preset first threshold is 0.

5. The method according to claim 3, characterized in that, Segmenting the speech audio stream at frame positions in the second sequence where the value is less than or equal to a preset first threshold includes: If there are multiple consecutive frames in the second sequence whose values ​​are less than or equal to a preset first threshold, then the speech audio stream is segmented at one of the multiple consecutive frames.

6. The method according to claim 3, characterized in that, If no frame with a value less than or equal to a preset first threshold appears in the second sequence for more than a preset second duration starting from the latest segmentation position, then the speech audio stream is segmented from a position of preset second duration from the latest segmentation position; The second duration is longer than the first duration.

7. The method according to claim 1, characterized in that, Inputting the continuous second audio blocks into the second speech recognition model includes: The acoustic features of the continuous second audio blocks and the latent vector representation output by the encoding module in the first speech recognition model are concatenated, and the concatenated feature representation is downsampled and then input into the second speech recognition model.

8. A method for streaming speech recognition, characterized in that, The method includes: Acquire the audio stream generated during a real-time meeting; The first audio blocks obtained by segmenting the audio stream according to the first duration are input into the first speech recognition model to obtain the recognition results of each first audio block, which are then displayed on the user terminal of the real-time conference. Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame. The first sequence is used to segment the speech audio stream to obtain continuous second audio blocks, wherein the continuous second audio blocks are obtained by segmenting the speech audio stream at the position of a silence frame or silence segment, the position of the silence frame or silence segment is determined based on the second sequence, which is generated based on the weight value of each frame in the speech audio stream; The continuous second audio blocks are input into the second speech recognition model to obtain the recognition results of each second audio block. The recognition results of each second audio block are then used to update the recognition results of the corresponding first audio block that have been displayed on the user terminal of the real-time conference.

9. A method for streaming speech recognition, executed by a cloud server, characterized in that, The method includes: Acquire the voice audio stream from the terminal device; The first audio block obtained by segmenting the speech audio stream according to the first duration is input into the first speech recognition model to obtain the recognition result of each first audio block, and the recognition result of each first audio block is sent to the terminal device for display. Obtain the latent vectors of each frame obtained by encoding the speech audio stream, and use the latent vectors to predict the first sequence corresponding to the speech audio stream. The first sequence contains the weight values ​​of each frame in the speech audio stream, and the weight values ​​are used to characterize the amount of acoustic information contained in the corresponding frame. The first sequence is used to segment the speech audio stream to obtain continuous second audio blocks, wherein the continuous second audio blocks are obtained by segmenting the speech audio stream at the position of a silence frame or silence segment, the position of the silence frame or silence segment is determined based on the second sequence, which is generated based on the weight value of each frame in the speech audio stream; The continuous second audio blocks are input into the second speech recognition model to obtain the recognition results of each second audio block. The recognition results of the corresponding first audio block already displayed by the terminal device are then updated using the recognition results of each second audio block.

10. A method for training a speech recognition model, characterized in that, The method includes: Acquire training data containing multiple training samples, wherein the training samples include speech audio samples and the recognition result labels corresponding to the speech audio samples; The training data is used to train a second speech recognition model. The training includes: obtaining latent vectors for each frame of the encoded speech audio sample; using the latent vectors to predict a first sequence corresponding to the speech audio sample, the first sequence containing weight values ​​for each frame in the speech audio sample, the weight values ​​representing the amount of acoustic information contained in the corresponding frame; segmenting the speech audio sample using the first sequence to obtain continuous second audio blocks, wherein the continuous second audio blocks are obtained by segmenting the speech audio sample at the positions of silence frames or silence segments, the positions of the silence frames or silence segments being determined based on a second sequence, the second sequence being generated based on the weight values ​​of each frame in the speech audio sample; inputting the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each audio block obtained by the second speech recognition model; the training objective includes minimizing the difference between the recognition results obtained by the second speech recognition model for the speech audio sample and the corresponding recognition result labels.

11. A streaming speech recognition device, characterized in that, The device includes: The audio stream acquisition unit is configured to acquire the voice audio stream; The first recognition unit is configured to input continuous first audio blocks obtained by segmenting the speech audio stream according to a first duration into the first speech recognition model, and to obtain the recognition results of each first audio block for display. A speech segmentation unit is configured to acquire latent vectors of each frame obtained by encoding the speech audio stream, predict a first sequence corresponding to the speech audio stream using the latent vectors, the first sequence containing weight values ​​of each frame in the speech audio stream, the weight values ​​being used to characterize the amount of acoustic information contained in the corresponding frame; and segment the speech audio stream using the first sequence to obtain continuous second audio blocks, wherein the continuous second audio blocks are obtained by segmenting the speech audio stream at the positions of silence frames or silence segments, the positions of the silence frames or silence segments being determined based on a second sequence, the second sequence being generated based on the weight values ​​of each frame in the speech audio stream; The second recognition unit is configured to input the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each second audio block; The result update unit is configured to update the recognition result of the corresponding first audio block that has been displayed using the recognition result of each second audio block.

12. An apparatus for training a speech recognition model, characterized in that, The device includes: The sample acquisition unit is configured to acquire training data containing multiple training samples, wherein the training samples include speech audio samples and recognition result labels corresponding to the speech audio samples; The model training unit is configured to train a second speech recognition model using the training data. The training includes: acquiring latent vectors for each frame obtained by encoding the speech audio sample; predicting a first sequence corresponding to the speech audio sample using the latent vectors, the first sequence containing weight values ​​for each frame in the speech audio sample, the weight values ​​representing the amount of acoustic information contained in the corresponding frame; segmenting the speech audio sample using the first sequence to obtain continuous second audio blocks, wherein the continuous second audio blocks are obtained by segmenting the speech audio sample at the positions of silence frames or silence segments, the positions of the silence frames or silence segments being determined based on a second sequence, the second sequence being generated based on the weight values ​​of each frame in the speech audio sample; inputting the continuous second audio blocks into the second speech recognition model to obtain the recognition results of each audio block obtained by the second speech recognition model; the training objective includes minimizing the difference between the recognition results obtained by the second speech recognition model for the speech audio sample and the corresponding recognition result labels.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 10.

14. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 10.

Images